London Web Factory

LWF SEO Web Spider: Quick Start

This guide walks through a basic page analysis and crawl.

Open a Website

  1. Open LWF SEO Web Spider.
  2. Click the address bar.
  3. Enter a website address, such as https://www.example.com/.
  4. Press Enter.

You can also type a search query. The browser opens Google by default.

The address bar can suggest previously crawled websites. Suggestions show the host, the full URL and a crawl count where the site has been crawled more than once.

Analyse One Page

  1. Open a page in the Browser tab.
  2. Choose Analyse page.
  3. Review the Page Analysis sidebar.

The sidebar shows title, meta description, H1, element counts, word count, path, image information, canonical URL, robots meta, first paragraph, document outline, top phrases, links, images, robots.txt and sitemap.xml. It refreshes automatically when a new browser page finishes loading while the sidebar is open.

Crawl a Website

  1. Open the website you want to crawl.
  2. Choose a crawl speed.
  3. Choose Start.
  4. Review the Crawl Results tab while the crawl runs.

Standard users can use Gentle mode. Pro users can also use Balanced and Fast modes.

The crawler follows pages and assets on the same host as the crawl start URL. For example, www.example.com and shop.example.com are treated as different hosts.

Pause, Resume or Stop

Use:

  • Pause to temporarily stop scheduling more crawl work;
  • Resume to continue a paused crawl;
  • Stop to end the crawl.

The app reports the number of crawled URLs, queued URLs, errors and elapsed time.

Review Results

The Crawl Results section has two tabs:

  • Internal Links shows rendered pages and assets found on the crawled host.
  • External Links shows link targets discovered outside the crawled host.

Click a URL in the results table to open URL Details. The details drawer includes overview data, inlinks, outlinks, resources, redirects and response headers.

Check External Headers

Open the External Links tab and choose Check headers to request headers for discovered external link targets. This updates content type, status code and status without crawling the external pages as part of the site crawl.

Export a CSV

Choose Export CSV after a crawl has produced results. The app saves the internal crawl results as a CSV file in the location you choose.

The default file name includes the crawled host and the current date.

Understand the Numbers

The table contains both raw observations and calculated classifications. For example, Status Code is observed from HTTP, while Indexability Status is derived from response, robots, noindex and canonical information.

Use these detailed references when reviewing a report: