London Web Factory

SEO Spider Crawl Results and URL Details

The Crawl Results tab is where crawled data appears while a crawl is running and after it finishes. The table combines raw observations, such as HTTP Status Code, with derived classifications, such as Indexability Status and Canonical Status. The complete calculation reference is Result Variables and Column Glossary.

Summary Metrics

Metric Meaning
Crawled Number of URLs completed in the current crawl.
Queued Number of URLs waiting to be crawled.
Errors Number of crawl items that ended with an error.
Elapsed Time since the crawl started, or total time after it finishes.

Errors includes rows with a recorded crawl error and rows whose official HTTP status is 400 or greater. A redirect is not counted as an error solely because it is a 3xx response.

Results Tabs

Internal Links

The Internal Links table shows same-host pages and assets collected by the crawler. It uses the full results table column set and supports:

  • filtering;
  • sorting;
  • virtual scrolling;
  • column resizing;
  • column visibility settings;
  • CSV export.

The columns are arranged into seven categories:

Category Contents
Page Address, response type/status and indexability.
Content Resource size, rendered text and principal descriptive elements.
Structure Counts of headings, paragraphs, lists, anchors, tables and images.
Links Incoming, outgoing, unique, external and depth metrics.
Directives Robots, refresh and canonical signals.
Response Timing, modification, redirect and HTTP protocol data.
Technical Content hash, encoded URL and crawl timestamp.

Same-host assets can appear in this table. HTML-only fields are normally empty for an image, stylesheet, script, PDF or other direct asset.

External Links

The External Links table shows link targets discovered outside the crawl host. It includes:

  • row number;
  • address;
  • content type;
  • status code;
  • status;
  • crawl depth;
  • inlinks.

Use Check headers to request external response headers. This updates external status fields without adding those external pages to the internal crawl.

One External Links row represents one normalised external target URL. It does not represent one domain or one link occurrence. Inlinks on that table is the total number of captured occurrences pointing to the target.

Before a header check, Status Code is 000 and Status is <none>. These values mean "not available", not a response sent by the external server.

Link Counts

The app deliberately separates raw elements, occurrences and unique URLs:

Value Unit
A Tags Rendered anchor elements.
Outlinks Qualifying internal anchor occurrences from the row.
Unique Outlinks Distinct internal target URLs from the row.
External Outlinks Qualifying external anchor occurrences from the row.
Unique External Outlinks Distinct external target URLs from the row.
Inlinks Internal link occurrences pointing to the row.
Unique Inlinks Distinct internal source URLs linking to the row.

These numbers are not expected to be equal. How Link Metrics Are Calculated gives exclusions and worked examples.

Filtering

Use Filter results to search the active result set.

For Internal Links, filtering searches the stored values of all current result columns, including hidden columns. It is a case-insensitive text match across the row rather than a field-specific query language.

For External Links, filtering can match address, content type, status, status code, recorded error and final redirect URL.

Filtering changes the displayed result count and row positions. It does not change the crawl database or the link calculations.

Sorting

Click a column header to sort the active table by that column. Numeric columns sort numerically where supported. Text columns sort alphabetically.

Empty values are placed after populated values for an Internal Links column sort. A stable crawl-order value is used as a tie-breaker. Row order therefore remains predictable when several values match.

Column Settings

The Internal Links table has a Columns menu. Use it to:

  • hide or show individual columns;
  • hide or show whole column categories;
  • restore the default column set.

Internal column visibility and column widths are persistent. Older saved settings that refer to removed columns are ignored.

Visibility changes presentation and CSV column selection; they do not delete stored crawl data. At least one internal column always remains visible.

URL Details Drawer

Click a URL in the results table to open the URL Details drawer.

The drawer includes:

Tab What it shows
Overview Main URL fields, indexability, canonical data, redirect data, robots.txt data, response time and crawl timestamp.
Inlinks Internal pages that link to the selected URL.
Outlinks Links from the selected URL to internal and external targets.
Resources Resources used by the selected URL, such as images, scripts and stylesheets.
Redirects Captured redirect chain for the URL.
Headers Response headers and redirect response headers where captured.

The drawer has Open in Browser and Analyse Page actions. Open in Browser opens the selected URL in the app browser. Analyse Page opens the app browser and shows the analysis sidebar for that URL.

Overview

Overview repeats the most useful summary fields and adds diagnostic values that are not all visible by default, including Final URL, Raw Status Code, robots.txt decision, external Checked At time and recorded Error.

Inlinks

Each row is one captured incoming link occurrence. It can show source URL, source title, link text, element, approximate placement, nearest detected heading and rel values.

For an internal result, only internal incoming occurrences are listed. For an External Links result, the tab lists occurrences from crawled internal pages to that external target. The same source URL can appear several times.

Outlinks

Each row is one qualifying outgoing link occurrence from the selected internal page. Internal and external targets are shown together and labelled by Scope.

The total normally corresponds to Internal Outlinks plus External Outlinks. Resources such as images and scripts are not included.

Resources

Each row is one captured resource occurrence. The table can show resource URL, element, source attribute, alt/title text, placement, browser initiator, transfer size and internal/external scope.

DOM references and browser performance entries can both contribute, so the same URL may legitimately appear more than once.

Redirects

Each row is one observed HTTP redirect hop with status, source URL and destination URL. The first hop supplies the main row's redirect status and Redirect URL.

Headers

Response Headers shows captured name/value pairs for the selected response. Redirect Response Headers are grouped by hop. A network failure can leave these sections empty because no response headers arrived.

Detail Limits

Inlinks, Outlinks and Resources return at most 2,000 occurrence rows per selected URL. When more were captured, the drawer shows a "rows shown" note. This is a display/query limit and does not automatically mean the main summary count is 2,000.

CSV Export

Choose Export CSV after a crawl has produced internal results. The app streams the export to disk, which avoids building the entire CSV in memory.

The export uses the internal result column definitions. Hidden columns are excluded when the export is called with the current visible column selection.

CSV export contains Internal Links rows only. External Links and the detailed occurrence, redirect and header tabs are not currently included in that export.

Rows are exported in original crawl order. Values are quoted as CSV fields, and the file includes a UTF-8 byte-order mark for compatibility with common spreadsheet applications.

Reading Empty Values

An empty field can mean the source did not provide the value, the resource type does not support it, the page did not render successfully, or Chromium did not expose the measurement. Examples:

  • an image row normally has no Title or H1;
  • a server without Last-Modified produces an empty Last Modified value;
  • HTTP Version can be unavailable for direct assets;
  • a connection error can have no Status Code;
  • an unchecked external link has 000 because no response has been requested.

Use the row's Content Type, Status, Error and URL Details together before treating an empty cell as an SEO fault.