Start a Crawl
- Open the website in the Browser tab.
- Choose the speed mode.
- Choose
Start.
The app switches to Crawl Results after the crawl starts.
The address currently open in the Browser tab is used as the starting URL. It receives Crawl Depth 0. Starting a new crawl creates a fresh working crawl database, so export a previous crawl before replacing it.
What Happens During a Crawl
The crawler works through a queue of normalised same-host URLs. For an HTML page it records:
- response status, headers, redirects and timing;
- rendered title, description, headings and visible text metrics;
- robots and canonical directives;
- structural element and image-alt counts;
- qualifying internal and external link occurrences;
- page resources reported by the DOM and Chromium;
- URLs eligible to enter the queue.
Link-derived totals can change while the crawl is running. In particular, an already completed target can gain Inlinks when later source pages are processed.
Crawl Scope
The crawler follows pages and assets on the exact same host as the first successfully loaded page.
Examples:
| Start URL | In scope | Out of scope |
|---|---|---|
https://www.example.com/ |
https://www.example.com/about/ |
https://shop.example.com/ |
https://example.com/ |
https://example.com/contact/ |
https://www.example.com/ |
External links are recorded in the External Links tab, but they are not crawled as part of the internal site crawl.
Scope uses the exact host, including an explicit port. A different subdomain is external. HTTP and HTTPS URLs on the same host remain in the same scope.
URL Normalisation
The app:
- adds
https://when a starting address has no scheme; - accepts only HTTP and HTTPS URLs;
- removes URL fragments;
- removes common tracking parameters such as
utm_*,gclid,fbclidandmsclkid; - sorts query parameters for consistent matching.
The app avoids some obviously destructive or session-changing URLs, including common logout, checkout, delete, remove, unsubscribe and admin paths or query parameters.
Normalisation affects both queue deduplication and unique link calculations. Two links that differ only by a fragment or removed tracking parameter normally identify one target, while retained functional query parameters can identify separate URLs.
Crawl Speeds
| Mode | Availability | Rendered workers | Asset workers | Typical use |
|---|---|---|---|---|
| Gentle | Standard and Pro | 1 | 1 | Small sites, cautious testing, first crawl of a site. |
| Balanced | Pro | 6 | 6 | Faster crawls where you have permission and the site can handle more load. |
| Fast | Pro | 12 | 12 | Larger crawls on sites you control or have explicit permission to test at higher request volume. |
Higher speed modes can use more CPU, memory and bandwidth because each rendered worker loads page subresources.
Speed controls concurrency, not the amount of information extracted from a completed page. Results should use the same measurement rules in every mode, although highly dynamic pages can naturally vary between requests.
Crawl Limit
The current configured crawl ceiling is 100,000,000 same-host URLs. This is a maximum limit, not a recommendation. Most crawls should be much smaller.
Robots.txt
The app fetches robots.txt, chooses the most specific matching User Agent group and records the winning Allow or Disallow rule. It can mark a URL Non-Indexable / Blocked when a disallow rule wins.
Robots evaluation is currently diagnostic. A URL may already have been requested before its result is classified, so do not rely on the app as an enforcement barrier for prohibited paths.
Do not treat a technical report as permission to crawl. You remain responsible for deciding whether crawling is authorised and appropriate.
Pause, Resume and Stop
Pausepauses an active crawl.Resumecontinues a paused crawl.Stopends the crawl and leaves collected results available for review and export.
Pause takes effect between active work. Pages already being loaded may finish before the crawl becomes fully paused. Stop cancels active work where possible and retains rows completed up to that point.
Monitoring a Crawl
| Counter | How to read it |
|---|---|
| Crawled | Unique queued URLs that have produced and stored a result row. |
| Queued | Discovered URLs still waiting to be processed. |
| Errors | Completed rows with a recorded error or an HTTP status of 400 or greater. |
| Elapsed | Time since Start, or final duration after completion. |
The queue can grow quickly when a rendered page reveals many same-host pages and assets. A falling queue means workers are completing URLs faster than new ones are being discovered.
External Header Checks
The External Links tab can check headers for discovered external URLs. Header checks send HEAD, with a GET retry only for 405 or 501, and update:
- content type;
- status code;
- status text.
The app uses 000 where no valid HTTP status code is available. Non-standard values are not displayed as HTTP response codes.
An external redirect is reported as its redirect response. The app records the destination but does not follow the redirect through the external website. A timeout, DNS failure, TLS failure or blocked connection has no real HTTP status, so 000 remains appropriate.
When a Crawl Finishes
A crawl completes when no queued or active URLs remain. It can also end because it was stopped, a fatal crawl error occurred, or the crawl ceiling was reached.
After completion:
- review Internal Links and External Links;
- check external headers if required;
- inspect important rows in URL Details;
- export the visible internal result columns to CSV before starting another crawl.
Responsible Crawling
Crawling originates from your computer and public IP address. Target services may rate-limit, block or blacklist requests. Crawl only websites you own, manage, or have permission to test.
See Responsible Crawling before testing a site you do not directly control.