Crawl Process at a Glance

For each crawl, the app:

1. validates and normalises the starting URL;
2. creates a fresh crawl queue with the starting URL at depth 0;
3. clears the dedicated crawl browser session;
4. loads HTML pages in Chromium and allows JavaScript to render;
5. extracts page content, directives, links and resources from the rendered DOM;
6. records response and redirect information observed during navigation;
7. normalises discovered HTTP and HTTPS URLs;
8. adds eligible same-host URLs to the queue;
9. records qualifying links to other hosts in External Links;
10. stores results and supporting occurrences in the current crawl's SQLite database.

The crawler repeats this process until the queue is empty, the user stops the crawl, or the configured crawl ceiling is reached.

Starting URL

The address entered when Start is selected becomes the crawl starting URL.

• If the address has no scheme, the app adds https://.
• Only http:// and https:// addresses are accepted.
• Addresses containing a username or password are rejected.
• A fragment such as `

contact` is removed.

• The starting URL is assigned Crawl Depth 0.

The starting address establishes the approved crawl host before the first request. A redirect cannot expand that scope. To crawl a different host, start a separate crawl using that host's address.

Exact-Host Scope

The current crawl scope is an exact hostname and port match. A different subdomain is external.

Starting host	Internal example	External example
www.example.com	https://www.example.com/about/	https://shop.example.com/
example.com	https://example.com/contact/	https://www.example.com/
example.com:8443	https://example.com:8443/help/	https://example.com/help/

The scheme stays the same, except that HTTP may upgrade to HTTPS on their standard ports at the same hostname. HTTPS downgrades, port changes and automatic aliases between example.com and www.example.com are refused. Explicit local development addresses remain supported; the external header checker's public-only policy does not apply to them.

External targets are recorded, but they are not added to the internal crawl queue. Check headers is a separate, limited request process described in [External Header Checks](

external-header-checks).

URL Normalisation

Before a discovered URL is used as a crawl identity, the app:

• resolves relative URLs against the source page;
• accepts only HTTP and HTTPS URLs;
• removes the fragment, including everything after `

`;

• removes parameters beginning with utm_;
• removes common tracking parameters including dclid, fbclid, gclid, gbraid, mc_cid, mc_eid, msclkid, twclid and wbraid;
• sorts the remaining query parameters into a consistent order.

This means these links identify the same crawl target:

https://example.com/services/

# seo
https://example.com/services/

These also identify the same target:

https://example.com/page?b=2&a=1&utm_source=newsletter
https://example.com/page?a=1&b=2

Other query parameters are retained in the identity key. Pages that vary by a retained parameter can therefore remain separate crawl identities.

The first accepted exact request URL is persisted separately from that key. Actual transport preserves query order, duplicate values, percent encoding and tracking fields through document/asset admission, promotion and redirects. URLs sharing a normalised key keep deterministic first-admission ownership; an observation of one representation cannot establish another representation's response. Existing databases keep their old identities and relationships.

URLs Avoided for Safety

The crawler rejects some URLs that look likely to change state or end a session. Current checks include common logout, sign-out, unsubscribe, checkout, WordPress administration and login paths, plus query parameters associated with delete, remove, logout, unsubscribe and add-to-cart actions.

Checks run before requests and at redirect destinations. A bounded comparison copy handles encoded path/query characters and separators; it does not rewrite the actual redirect URL or signed query. Request URLs are limited to 16,384 characters and at most five decoding passes. Malformed escapes, unsupported control characters and deeper encoding are refused. The frontier's URL identity normalization remains as described above.

New URL credentials are rejected at input and discovery boundaries. Older stored userinfo is hidden in displayed results, details, history and CSV output without rewriting or merging stored identities.

The same exclusions apply to browser subresources and AJAX, including /wp-admin/admin-ajax.php. An image or XHR GET can also invoke a destructive endpoint, so those requests do not bypass policy. A bounded count, reasons and samples appear in the crawl summary. Blocked resources can change the rendered page and its discovered links.

This is a precaution, not a complete guarantee. A website can use any URL for a destructive action. Crawl only with permission and avoid authenticated sessions that could expose state-changing links.

Rendered HTML Pages

HTML documents are loaded in a dedicated Chromium crawl view. After loading, the app waits briefly for rendering to settle and extracts data from the DOM visible to Chromium.

Rendered extraction means:

• links or text inserted by JavaScript can be included;
• the result can differ from the original server HTML;
• content added after the extraction point may be missed;
• interactions that require clicking, scrolling, consent or login are not automatically performed;
• shadow DOM and content inside a separate frame may not be represented like ordinary document DOM.

The crawl session is separate from the normal app browser and is cleared when a new crawl begins. A page that depends on the browser tab's cookies or local storage may therefore behave differently in the crawl window.

Navigation starts only after the previous worker page, cache and storage have been cleared successfully. If preparation fails, the crawl reports the failure and makes no new crawl requests. Starting again retries the complete preparation.

DOM extraction is limited to 10 seconds after rendering. If extraction does not return within that period, the row records an extraction error and the affected Chromium worker is replaced so one unresponsive page cannot hold the crawl open indefinitely.

Each crawl and manual external-header check owns its requests and the database generation it may update. Stop prevents new work, cancels pending navigation, asset requests, robots requests and waits, and drains every active operation. Another crawl cannot reset the database until that drain finishes, including when one worker fails while another is still pending. A completed document is cleared from its worker before the next navigation; late callbacks cannot update that worker's next page or a later crawl.

Closing a window waits for crawl retirement. Quitting waits for retirement and active CSV exports before closing the database. Exports already requested can finish during retirement. Retirement has a five-second drain deadline; if an operation or cleanup cannot settle, the app reports the failure and refuses to reset or close the database under that work. An uninterruptible session reset must also settle before its session can be reused. The recovery dialog defaults to Keep open, retaining quarantined ownership. Force quit is explicit and may lose unfinished work; it does not report a clean shutdown. Repeated Quit or macOS activation keeps those controls accessible, even without a normal window. The on-disk report is retained for restart recovery. Manual external checks remain available after a completed, stopped or failed crawl and use their own cancellation state.

Text Extraction

For crawl-table text metrics, the app walks text nodes under the rendered document body. It collapses repeated whitespace and excludes text inside:

• script;
• style;
• noscript;
• svg;
• canvas;
• template.

Navigation, footer and other visible interface text are not automatically removed. Word Count therefore measures rendered page text, not only editorial body copy.

Link and Resource Discovery

Qualifying links are extracted from <a href> elements. Resource URLs are collected from common elements and browser performance entries, including:

• images and srcset candidates;
• scripts and stylesheets;
• audio and video sources;
• video posters and tracks;
• frames, embeds and objects;
• icons, manifests, preloads and prefetches;
• other network resources reported by Chromium.

Links and resources are stored as occurrences. An occurrence preserves useful context such as source URL, target URL, element, attribute, text or alt text, relationship values, approximate placement and nearest heading.

The main link columns count qualifying anchor destinations. They do not count resource occurrences. See How Link Metrics Are Calculated.

Queueing and Crawl Depth

The starting URL enters the queue at depth 0. A same-host URL discovered on a page at depth n is normally queued at depth n + 1.

The queue stores one identity and one result row for each normalised URL. Repeated discoveries do not create duplicate jobs. An ordinary link to a page takes precedence when that URL also appears as a prefetch, iframe or browser resource. A queued asset becomes a document job; an active asset check finishes before the document job starts; a completed asset result is replaced after document rendering. The document is extracted once, including its own links. This can still happen at the URL ceiling because it uses an already accepted identity.

For a document promoted from an asset, Crawl Depth comes from the navigational link, rather than the earlier resource sighting. Shallower document links can update that depth while the job is queued or active. Once the document completes, its recorded depth stays fixed. Resource-only assets retain their original discovery depth.

Crawl Depth is a link-distance measure. It is not a PageRank, authority or importance score.

Page and Asset Requests

URLs considered HTML documents are rendered in Chromium. Eligible asset jobs can use response metadata already received by a selected rendered page. Where no usable browser observation exists and another request is permitted, the crawler uses its explicit GET check, stopping after response headers. Neither path downloads a body for asset analysis.

Direct assets include common image, stylesheet, script, font, media, archive, office-document, feed, data and PDF extensions. Asset rows can therefore have response data while HTML-only fields remain empty.

Linking to a recognised direct asset does not make it an HTML document. Likewise, an asset error response containing HTML does not by itself trigger page extraction. For an extensionless navigational URL, Chromium's own response determines whether HTML extraction is appropriate.

The interface reports separate DOM Crawler and Asset Crawler progress values. An asset job uses the same task accounting whether it resolves from browser metadata or an explicit request. It does not occupy a Chromium crawl-window tile.

Selected Page Response Metadata

The first page that admits an asset job selects its evidence source. Reuse is limited to attributable main-frame CSS, JavaScript, image and font requests in that navigation. URL Details identifies the source page/navigation, response source, timing meaning, known cache/load state, coverage and any fallback reason. This answers what that page received; it is not an independent anonymous or fresh origin check. Later pages sharing the asset do not replace that result.

Matching retains exact query order, repeated values, encoded characters and path case and removes only fragments. Evidence must match the asset job's first accepted exact request URL, even when its normalised key differs. Another request representation sharing that key cannot supply its response. Conflicting observations also cannot overwrite each other. XHR/API traffic, prefetch-only sightings and unattributed frames/workers do not establish reusable asset evidence.

Capture ends when the rendered page's result is ready, before that page is cleared from its worker. Sufficient headers can be used even while the transfer is pending; URL Details then says Pending, rather than claiming the body completed. Known HTTP errors remain errors. A security block, transport failure, cancellation or loss of necessary capture information does not trigger another request. Missing events alone can use the explicit path after this boundary, subject to policy. There is no additional wait for a resource body and no parallel fallback against a known pending transfer.

A font can first report ERR_CACHE_MISS and then finish loading successfully. When the selected page's exact font URL has a complete, attributable response after that preliminary cache miss, the crawler uses the completed response and records Browser Recovery in URL Details. Its status, headers and response time come from that completed attempt. No extra check is sent. Unrecovered cache misses, overlapping or conflicting attempts, real transfer failures and policy blocks remain visible; an HTTP error response retains its actual status.

Live capture is capped at 256 requests per navigation, 2,048 requests and 8 MiB of metadata overall, with at most 32 navigation contexts. Another 2,048 retired request IDs distinguish late callbacks from unknown activity. Response or failure events without an attributable start make the affected live capture unavailable. Header blocks retain at most 16 KiB and 64 values, and redirect chains at most 20 hops. Exceeding a capture limit discards that navigation's evidence and reports it unavailable. Queued decisions retain at most 1,024 entries and 8 MiB; oldest entries are evicted. Missing older decisions then remain unavailable, including when a coarse identity cutoff excludes another older job. Eviction never silently becomes a fresh request. Completion, promotion, cancellation and run disposal release retained evidence; whole worker logs are not kept while assets wait or are paused.

Only compact provenance and bounded response details persist with a result. Request credentials, cookies, response cookies and bodies are not captured. Sensitive diagnostic URL values are redacted without changing crawl identities. There is no reusable response cache across pages or crawls. Browser cache state is reported only when Electron supplies it; service-worker origin, wire status and origin revalidation remain unknown unless separately established.

The scheduler alternates document and asset starts when both classes have queued work and an available worker within their limits. An eligible class waits behind at most one start from the other class. If one class is empty or has no available worker, the other can use the shared capacity. This rule applies to Gentle's single shared slot as well as Balanced and Fast; Pause/Resume preserves whose turn comes next. It bounds scheduling turns, rather than seconds spent waiting for a network response. Existing concurrency limits, rendering waits and request timeouts still apply.

Progress totals count URLs discovered so far, not a fixed total known at the start. Completed is calculated independently for each crawler type from its discovered, queued and active work. Discovering additional URLs can increase a total while the crawl is running.

Promotion transfers a URL from Asset Crawler to DOM Crawler when its document job is queued. A previously completed asset can therefore become pending document work while its existing row remains visible. Replacing that row does not increase the Crawled total; Errors reflects the replacement result.

Redirect Handling

For a requested URL that redirects, the row remains associated with the requested address. The app records:

• the first redirect response as the row's Status Code;
• the first redirect destination as Redirect URL;
• a Permanent or Temporary Redirect Type where recognised;
• the observed redirect status sequence and final response status;
• redirect hops and redirect response headers in URL Details.

Eligible same-host redirect destinations can be added to the crawl queue. Refused destinations receive no crawler request and are not requeued from the denied redirect. The original redirect status and policy error remain available. Rendered and asset redirect chains are limited to 20 followed hops.

When an asset uses a selected page's response, its newly discovered redirect destinations remain separate URLs with unavailable independent metadata. They are not probed again simply because that observed request redirected. A target already accepted from another discovery keeps its own selected evidence.

Robots.txt

The app requests /robots.txt for the relevant origin, selects rules matching the configured crawler User Agent and evaluates each URL against the most specific matching rule. When equally specific Allow and Disallow rules match, Allow takes precedence.

The current app records the result and can classify a URL as Non-Indexable / Blocked. Robots information is diagnostic: it does not itself grant permission, and the current crawler may already have requested a URL before recording that classification.

Robots evaluation limits

Wildcard rules are evaluated with bounded literal matching. A large or complicated policy cannot cause unlimited regular-expression backtracking. The limits are:

Input or work	Limit
Decoded robots.txt text	80,000 characters
Allow/Disallow rules across the parsed groups	2,048
One rule pattern	2,048 characters
Combined patterns supplied to one evaluation	80,000 characters
Configured User Agent supplied to the parser	1,024 characters
Full URL supplied to the evaluator	65,536 characters
Serialized path plus query	16,384 characters
Matching work for one URL across all selected rules	1,000,000 character comparisons

Character limits use JavaScript UTF-16 units; a supplementary Unicode character uses two units. Only comparison copies are normalised according to [RFC 9309 section 2.2.2](https://www.rfc-editor.org/rfc/rfc9309.html

section-2.2.2): unreserved percent-encoded octets are decoded, Unicode rule text is compared as UTF-8, and equivalent hex encoding agrees. Reserved separators remain distinct; encoded wildcard/end characters stay literal. Specificity counts octets. Case sensitivity, wildcard/end matching, most-specific User Agent groups and Allow ties remain intact. Actual request URLs are never rewritten for robots matching.

If a limit is reached, the app discards the incomplete decision. URL Details shows Robots.txt Allowed as Unknown, leaves the winning rule empty and displays the reason in Robots.txt Error. Fetch failures and still-loading policies are also unknown. Other observed indexability signals continue to apply; unknown robots information is not permission to crawl or a confirmed Allow result.

The robots request retains its ten-second deadline and 80,000-character text cap. An oversized response is rejected rather than interpreted as a truncated policy. These rules do not introduce robots request enforcement or change redirect scope.

Indexability Classification

Indexability is a practical app classification, evaluated in this order:

1. a 3xx row is Non-Indexable / Redirected;
2. a 4xx row is Non-Indexable / Client Error;
3. a 5xx row is Non-Indexable / Server Error;
4. a crawl error is Non-Indexable / Error;
5. a matching robots.txt disallow rule is Non-Indexable / Blocked;
6. a noindex or none directive in meta robots or X-Robots-Tag is Non-Indexable / Noindex;
7. a canonical pointing to another URL is Non-Indexable / Canonicalised;
8. otherwise the row is Indexable / Indexable.

Applicable duplicate robots meta fields and repeated X-Robots-Tag fields are combined; indexing restrictions win conflicts. User-agent-specific fields apply only to the matching crawler. Directive values remain attached to their names: max-image-preview:none and max-image-preview: none agree and do not independently prohibit page indexing. Standalone none or noindex does. See Google robots metadata.

This classification is not a guarantee that a search engine will index or exclude the URL. Search engines can apply additional rules and make different canonical or duplicate decisions.

External Header Checks

External links initially have Status Code 000, which means no valid HTTP response code is available. 000 is an app placeholder, not an HTTP status code.

Crawl completion does not start external checks. Select Check headers in External Links to explicitly request checks of the discovered targets. The button checks all targets; the table search does not narrow this action. The operation captures its target selection in a SQLite worklist and reads bounded batches, so status updates cannot skip or repeat selected targets. Cancel checks cancels that operation and drains active requests before its store can be replaced.

The checker:

1. permits HTTP/HTTPS public unicast destinations only, rejecting URL credentials and the app's destructive-path/query exclusions (including bounded decoding of encoded exclusions);
2. checks every returned A and AAAA address and pins the connection to a validated address; private, loopback, link-local, reserved, carrier-grade NAT, multicast, mapped IPv6 and mixed public/private answers are refused;
3. sends HEAD without cookies, HTTP/proxy authentication, client certificates or a referrer, using a fresh direct connection and the configured crawler User Agent;
4. records the first response headers and closes the connection immediately, including when a server incorrectly sends a HEAD body;
5. retains the original 3xx status and reported destination without following it, and never retries with GET (including 405 and 501);
6. uses up to six concurrent requests and one ten-second deadline covering DNS, connection, TLS and response headers.

URLs are limited to 16,384 characters, DNS answers to 64, response headers to 16 KiB per header block, and informational responses to four. No response body is accumulated or decompressed. Some bytes can arrive with the headers before the socket is closed. Special-purpose IP ranges are conservatively excluded even where an exception is globally reachable. IPv6 is restricted to ordinary global unicast outside these exclusions.

This checker does not use browser/system or environment HTTP proxies, browser cookies, ambient authentication or browser client identity. Networks requiring a proxy may therefore fail. HTTPS keeps hostname/SNI and certificate verification. DNS uses the system-configured DNS servers, with no hosts-file override. The exclusions are a heuristic, not a guarantee that every remaining website action is harmless.

A rejected destination, failed DNS lookup, TLS failure, timeout or connection error has no genuine HTTP response code, so the row remains 000; open URL Details to see the error. A server may return a different status for this public, unauthenticated header request than for a browser page load.

Rendered pages can still make ordinary third-party requests for scripts, images and other subresources while browsing or crawling. This external-check policy is separate from explicit browsing and local-development crawling.

Storage and Result Updates

The current crawl is stored locally in SQLite. Summary rows, URL identities, the queue, link and resource occurrences, response headers, redirects and external header checks are stored separately so the app can calculate totals and show URL Details without retaining full rendered pages.

If saving a result fails, its transaction is rolled back together with its cached identities, counts and ordering. Failed writes do not publish a completed result. If rollback recovery also fails, storage becomes unavailable and further queries, writes and exports are refused; restart the app before continuing.

Inlinks are recalculated from stored link occurrences as source pages complete. A target's Inlinks values can therefore increase while the crawl is still running.

Starting another crawl replaces the current working crawl database. Export results that need to be retained.

Current Collection Limits

Limits prevent unusually large pages from overwhelming the interface:

• rendered crawl extraction returns at most 10,000 stored link/resource occurrences from a page;
• a URL Details occurrence tab returns at most 2,000 rows;
• Page Analysis returns at most 300 outline entries, 500 links and 300 images;
• the results grids load data in virtualised batches rather than rendering every row at once.

Main row metrics are calculated independently where possible, so a detail-list display limit does not necessarily reduce the corresponding summary count. On extremely link-heavy pages, however, occurrence-derived Inlinks and detail totals can be limited by captured occurrence data.

Interpreting Results Carefully

Results describe what this app observed from this computer, User Agent, network and crawl session at that time. Authentication, geolocation, personalisation, consent tools, anti-bot services, caching, transient server faults and JavaScript timing can all affect the result.

Use the table as a reproducible diagnostic report, then verify important findings in the Browser tab, URL Details and the target website's own server or analytics data.

Response Failures and Retained Runs

An attributed rendered response retains known HTTP status, content type and applicable headers as soon as headers arrive. A later body/transfer failure is recorded separately. A truncated 503 HTML response therefore remains a 503 with its headers and a visible transfer error; it is not treated as a successful transfer or analysed as Chromium's error page. Original redirect responses/chains and terminal-response evidence remain available.

The default Maximum URLs is 10,000 (documents and assets combined), frozen at admission. The supported setting range ends at 1,000,000, but this is not a database performance claim. Virtual grids keep bounded geometry and caches; synchronous substring/aggregate queries can still scan much of a report and delay the interface.

The current report is recovered after restart. Previously active work is marked interrupted with no live workers and no automatic crawl requests. New crawl replacement waits for valid admission, successful preparation and settled store owners. Unsupported/newer/corrupt databases remain preserved for recovery.