Crawl Process at a Glance

For each crawl, the app:

1. validates and normalises the starting URL;
2. creates a fresh crawl queue with the starting URL at depth 0;
3. clears the dedicated crawl browser session;
4. loads HTML pages in Chromium and allows JavaScript to render;
5. extracts page content, directives, links and resources from the rendered DOM;
6. records response and redirect information observed during navigation;
7. normalises discovered HTTP and HTTPS URLs;
8. adds eligible same-host URLs to the queue;
9. records qualifying links to other hosts in External Links;
10. stores results and supporting occurrences in the current crawl's SQLite database.

The crawler repeats this process until the queue is empty, the user stops the crawl, or the configured crawl ceiling is reached.

Starting URL

The address entered when Start is selected becomes the crawl starting URL.

• If the address has no scheme, the app adds https://.
• Only http:// and https:// addresses are accepted.
• A fragment such as `

contact` is removed.

• The starting URL is assigned Crawl Depth 0.

The first successfully processed URL establishes the exact host used as the crawl scope. If the starting address immediately redirects, the redirect destination can establish that host.

Exact-Host Scope

The current crawl scope is an exact hostname and port match. A different subdomain is external.

Starting host	Internal example	External example
www.example.com	https://www.example.com/about/	https://shop.example.com/
example.com	https://example.com/contact/	https://www.example.com/
example.com:8443	https://example.com:8443/help/	https://example.com/help/

The scheme can change during normalisation or redirection without changing the host comparison, provided the hostname and port remain the same.

External targets are recorded, but they are not added to the internal crawl queue. Check headers is a separate, limited request process described in [External Header Checks](

external-header-checks).

URL Normalisation

Before a discovered URL is used as a crawl identity, the app:

• resolves relative URLs against the source page;
• accepts only HTTP and HTTPS URLs;
• removes the fragment, including everything after `

`;

• removes parameters beginning with utm_;
• removes common tracking parameters including dclid, fbclid, gclid, gbraid, mc_cid, mc_eid, msclkid, twclid and wbraid;
• sorts the remaining query parameters into a consistent order.

This means these links identify the same crawl target:

https://example.com/services/

# seo
https://example.com/services/

These also identify the same target:

https://example.com/page?b=2&a=1&utm_source=newsletter
https://example.com/page?a=1&b=2

Other query parameters are retained. Pages that genuinely vary by a retained query parameter can therefore remain separate crawl URLs.

URLs Avoided for Safety

The crawler rejects some URLs that look likely to change state or end a session. Current checks include common logout, sign-out, unsubscribe, checkout, WordPress administration and login paths, plus query parameters associated with delete, remove, logout, unsubscribe and add-to-cart actions.

This is a precaution, not a complete guarantee. A website can use any URL for a destructive action. Crawl only with permission and avoid authenticated sessions that could expose state-changing links.

Rendered HTML Pages

HTML documents are loaded in a dedicated Chromium crawl view. After loading, the app waits briefly for rendering to settle and extracts data from the DOM visible to Chromium.

Rendered extraction means:

• links or text inserted by JavaScript can be included;
• the result can differ from the original server HTML;
• content added after the extraction point may be missed;
• interactions that require clicking, scrolling, consent or login are not automatically performed;
• shadow DOM and content inside a separate frame may not be represented like ordinary document DOM.

The crawl session is separate from the normal app browser and is cleared when a new crawl begins. A page that depends on the browser tab's cookies or local storage may therefore behave differently in the crawl window.

Text Extraction

For crawl-table text metrics, the app walks text nodes under the rendered document body. It collapses repeated whitespace and excludes text inside:

• script;
• style;
• noscript;
• svg;
• canvas;
• template.

Navigation, footer and other visible interface text are not automatically removed. Word Count therefore measures rendered page text, not only editorial body copy.

Link and Resource Discovery

Qualifying links are extracted from <a href> elements. Resource URLs are collected from common elements and browser performance entries, including:

• images and srcset candidates;
• scripts and stylesheets;
• audio and video sources;
• video posters and tracks;
• frames, embeds and objects;
• icons, manifests, preloads and prefetches;
• other network resources reported by Chromium.

Links and resources are stored as occurrences. An occurrence preserves useful context such as source URL, target URL, element, attribute, text or alt text, relationship values, approximate placement and nearest heading.

The main link columns count qualifying anchor destinations. They do not count resource occurrences. See How Link Metrics Are Calculated.

Queueing and Crawl Depth

The starting URL enters the queue at depth 0. A same-host URL discovered on a page at depth n is normally queued at depth n + 1.

The queue stores one identity for each normalised URL. If the same target is discovered repeatedly, it is not queued repeatedly. Crawl Depth is the discovery depth retained when that URL first enters the breadth-first crawl queue. In normal use this represents the shortest discovered path from the starting page.

Crawl Depth is a link-distance measure. It is not a PageRank, authority or importance score.

Page and Asset Requests

URLs considered HTML documents are rendered in Chromium. URLs recognised as direct assets are requested separately with GET, and their response headers are recorded without downloading and analysing the complete body as a rendered page.

Direct assets include common image, stylesheet, script, font, media, archive, office-document, feed, data and PDF extensions. Asset rows can therefore have response data while HTML-only fields remain empty.

Redirect Handling

For a requested URL that redirects, the row remains associated with the requested address. The app records:

• the first redirect response as the row's Status Code;
• the first redirect destination as Redirect URL;
• a Permanent or Temporary Redirect Type where recognised;
• the observed redirect status sequence and final response status;
• redirect hops and redirect response headers in URL Details.

Same-host redirect destinations can be added to the crawl queue. A redirect leaving the crawl host is not followed into the external site as part of the internal crawl.

Robots.txt

The app requests /robots.txt for the relevant origin, selects rules matching the configured crawler User Agent and evaluates each URL against the most specific matching rule. When equally specific Allow and Disallow rules match, Allow takes precedence.

The current app records the result and can classify a URL as Non-Indexable / Blocked. Robots information is diagnostic: it does not itself grant permission, and the current crawler may already have requested a URL before recording that classification.

Indexability Classification

Indexability is a practical app classification, evaluated in this order:

1. a 3xx row is Non-Indexable / Redirected;
2. a 4xx row is Non-Indexable / Client Error;
3. a 5xx row is Non-Indexable / Server Error;
4. a crawl error is Non-Indexable / Error;
5. a matching robots.txt disallow rule is Non-Indexable / Blocked;
6. a noindex or none directive in meta robots or X-Robots-Tag is Non-Indexable / Noindex;
7. a canonical pointing to another URL is Non-Indexable / Canonicalised;
8. otherwise the row is Indexable / Indexable.

This classification is not a guarantee that a search engine will index or exclude the URL. Search engines can apply additional rules and make different canonical or duplicate decisions.

External Header Checks

External links initially have Status Code 000, which means no valid HTTP response code is available. 000 is an app placeholder, not an HTTP status code.

When Check headers is selected, the app:

1. sends a HEAD request to each filtered external target;
2. retries with GET only when the server returns 405 Method Not Allowed or 501 Not Implemented;
3. records response headers without downloading the full response body;
4. records the first redirect response and its destination without crawling through the external site;
5. uses up to six concurrent requests;
6. gives each request up to ten seconds.

A failed DNS lookup, TLS failure, timeout, connection block or other network error has no genuine HTTP response code, so the row remains 000 and the error is available in URL Details. A server may also return a different status to this header request than it returns to a normal browser page load.

Storage and Result Updates

The current crawl is stored locally in SQLite. Summary rows, URL identities, the queue, link and resource occurrences, response headers, redirects and external header checks are stored separately so the app can calculate totals and show URL Details without retaining full rendered pages.

Inlinks are recalculated from stored link occurrences as source pages complete. A target's Inlinks values can therefore increase while the crawl is still running.

Starting another crawl replaces the current working crawl database. Export results that need to be retained.

Current Collection Limits

Limits prevent unusually large pages from overwhelming the interface:

• rendered crawl extraction returns at most 10,000 stored link/resource occurrences from a page;
• a URL Details occurrence tab returns at most 2,000 rows;
• Page Analysis returns at most 300 outline entries, 500 links and 300 images;
• the results grids load data in virtualised batches rather than rendering every row at once.

Main row metrics are calculated independently where possible, so a detail-list display limit does not necessarily reduce the corresponding summary count. On extremely link-heavy pages, however, occurrence-derived Inlinks and detail totals can be limited by captured occurrence data.

Interpreting Results Carefully

Results describe what this app observed from this computer, User Agent, network and crawl session at that time. Authentication, geolocation, personalisation, consent tools, anti-bot services, caching, transient server faults and JavaScript timing can all affect the result.

Use the table as a reproducible diagnostic report, then verify important findings in the Browser tab, URL Details and the target website's own server or analytics data.