v1.28.0

New features

• Support for captures with a headed browser (setting allow_headed). Only possible if lacus is running in a graphical environment. (see below for details)
• Complete rewrite of the indexing system, allows pagination (transparent on upgrade, but may take a while, be careful if you have a big instance)
• Index IPs and TLDs
• Link hostname and IPs indexes, expose it on the UI

• Trigger CIRCL Passive DNS wherever possible (on IPs and on Hostnames)

• Allow to share direct links to IP/Hostnames/Favicon/... modals

• Push capture to another Lookyloo instance (typically, a headed capture done locally to a central repository)

• Major overhaul of the categorization, use MISP Taxonomy by default (dark-web)

• Optionally makes modules admin only (avoid using all the tokens available on 3rd party APIs)
• Enable full text search on indexes when relevant
• Optional auto-submit of onion URLs to an AIL instance

Changes

• MISP export improvements
• Refactoring and improvements in the javascripts, massive speedups
• Render indexes with AJAX
• Render hostnode popup much faster by moving indexes to a subsequent page
• Submit any file to Pandora
• Trigger optional auto-report in background script
• Allow user accessible MISP servers
• Remove RiskIQ module (killed by microsoft)
• Support KV Rocks 2.11 (major speed improvements)
• Many improvements in logging, reduce noise, and use WARNING more appropriately
• Improve favicon rendering
• Update URLHaus module as it now requires an auth key

Fixes

• Re-enable submit to pandora buttons
• Deduplicate notifications
• Many, many other bugfixes

Notes for headed captures

The classical use of Lookyloo is to have it running on a server with no graphical interface (no X/Wayland server). The capture with Playwright uses a headless browser, runs some interactions on the page (see PlaywrightCapture for details), and finishes after a certain amount of time and/or no traffic. This method is good enough most of the time, but all the interactions on the page are predefined and cannot be modified by the user triggering the capture.

If you use a dedicated lacus instance, please refer to the lacus release for details first. And you need the following to trigger a headed capture:

• The configuration setting "allow_headed" = True in config/generic.json
• Lookyloo installed on a machine with a graphical interface (or at least lacus, if you're using a remote instance)
• [Web UI] On the capture page, in Capture configuration -> untick Use headless browser (it it's missing, check config/generic.json and restart the app)
• [Web UI] Optionally, on the capture page, in Capture configuration -> set Max capture time to a time, if needed (90s by default)
• [API] Pass headless set to False in the capture settings
• [API] Optionally general_timeout_in_sec set to the amount of time you want to interact with the page (it is set to 90 by default)

The new headed capture mode opens a full browser configured with the settings passed to the capture, and none of the predefined interactions. It lets the user interact with the page for a set amount of time (general_timeout_in_sec in the API, or Max capture time on the web interface), stops the capture, and store the result as usual. It is mostly helpful to manually bypass captchas and other techniques used by websites to detect bots.

mondial.webm

PRs

What's Changed

• change redis to valkey by @sebdraven in #1028

New Contributors

• @sebdraven made their first contribution in #1028

Full Changelog: v1.27.0...v1.28.0