urlcheck is a tool I made for scanning and analysing suspicious URLs. The tool was publicly available for some time and it’s source code is still available on GitHub. This project is no longer in active development, as I have integrated an improved version into my new project, Vanguard.
Analysis & Privacy
urlcheck collects metadata about the scanned URLs and stores it in a database. The collected data contains information such as the screenshot, list of contacted domains and IPs, the final HTML render, and more. This data can be used to analyze the URLs and detect possible threats. urlcheck also contains some basic checks and alerts for detecting some very common threats.
urlcheck is made with privacy in mind. It only collects the least amount of data from the user as possible. All the scan results are only stored for 48 hours and then they are automatically purged. urlcheck makes sure that all requests to the scanned URLs are made by the server and not the user’s browser. This ensures that the scanned website doesn’t get any information about the user.
Technical details
urlcheck uses a headless Chromium browser to access the pages and to take screenshots. It was quite tricky to get the browser disguised as a real browser, so that the scanned websites wouldn’t detect it as a bot. The tool is coded with TypeScript, frontend uses Next.js and the backend runs on Node.js. The database is a PostgreSQL database.
The future of urlcheck
After starting to work on Vanguard, I decided to integrate urlcheck into it. The new version of urlcheck is much more powerful and capable of detecting phishing sites with a high accuracy, as well as disuising itself to look more legitimate and not a bot. Vanguard URL Scanner has all the features of the original version and more. It also allows integration with SOAR for automated analysis and response and some third-party services to enrich the scan results. The new version is not publicly available.
