Try our SEO Web Spider — a new professional SEO crawler for up to 1-million URLs, completely free for life.
London Web Factory

Blog

Practical notes on web design, performance, SEO, PPC and digital marketing.

How the Crawler Works

This guide explains how LWF SEO Spider discovers, requests and records URLs. It is intended to make the figures in Crawl Results reproducible and easier to interpret. For the calculation rules behind link columns, see How Link Metrics Are Calculated. For a field-by-field reference, see Result Variables and Column Glossary.

Learn more

Responsible Crawling

LWF SEO Spider is designed for SEO, accessibility, content, quality-assurance and technical analysis. Use it only where you have a lawful and reasonable basis to send crawl requests.

Learn more

How the Crawler Works

This guide explains how LWF SEO Spider discovers, requests and records URLs. It is intended to make the figures in Crawl Results reproducible and easier to interpret. For the calculation rules behind link columns, see How Link Metrics Are Calculated. For a field-by-field reference, see Result Variables and Column Glossary.

Learn more

Crawl Results and URL Details

The Crawl Results tab is where crawled data appears while a crawl is running and after it finishes. The table combines raw observations, such as HTTP Status Code, with derived classifications, such as Indexability Status and Canonical Status. The complete calculation reference is Result Variables and Column Glossary.

Learn more

Installation

This guide is for people installing the desktop application from a published London Web Factory download.

Learn more

LWF SEO Spider: Crawling a Website

The crawler loads HTML pages in a visible Chromium-based crawl window and extracts data from the rendered DOM after JavaScript has run. Same-host assets are requested directly so their response data can also appear in results. This page explains how to operate a crawl. For the complete discovery and calculation rules, see How the Crawler Works.

Learn more

Settings, Local Data and Exports

LWF SEO Spider stores settings and crawl data locally on the computer running the app. The files are placed under Electron's per-user application data directory for LWF SEO Spider. The exact parent path varies by operating system and user account.

Learn more

Result Variables and Column Glossary

This reference explains where each value comes from, how derived values are calculated, and what the value does and does not prove. The current Internal Links table contains seven column groups in this order: Page, Content, Structure, Links, Directives, Response and Technical. For the end-to-end collection process, see How the Crawler Works. Link counting has additional examples in How Link Metrics Are Calculated.

Learn more