Get Permission First
Only crawl a website if:
- you own it;
- you manage it for a client;
- the owner has given you permission;
- you otherwise have a lawful and reasonable basis to crawl it.
Technical ability to crawl a website is not permission.
Check the Site's Rules
Before crawling, review:
- website terms;
- robots.txt directives;
- published crawling or API policies;
- access restrictions;
- rate-limit instructions.
The app can report robots.txt information, but you remain responsible for deciding whether a crawl is authorised and appropriate.
Start Gently
Use Gentle mode for a first crawl. Increase speed only when you know the target site can handle the request volume and you have permission to crawl at that rate.
Rendered crawling can generate many HTTP requests because each page may load scripts, stylesheets, images, fonts and other resources.
Understand IP Risk
Crawling originates from your device and public IP address. Target websites, hosting providers, network operators and security services may:
- rate-limit requests;
- block requests;
- blacklist an IP address;
- block or blacklist a user agent;
- suspend an account;
- contact the site owner or network operator;
- make a complaint.
London Web Factory is not responsible for those actions or consequences.
Avoid Harmful Activity
Do not use the app to:
- access non-public content without authorisation;
- bypass authentication, CAPTCHAs, paywalls or rate limits;
- overload a service;
- probe for vulnerabilities without explicit authority;
- collect personal, confidential or special-category information unlawfully;
- misrepresent your authority to crawl.
Watch the Crawl
While a crawl runs:
- monitor the Crawled, Queued and Errors counts;
- stop or pause the crawl if the site slows down;
- reduce speed if you see unusual errors;
- respond quickly to complaints or warning messages.
Protect Sensitive Data
Crawl results can contain sensitive URLs, query strings, page titles, anchor text and response headers. Review exports before sharing them.
Scope, Downloads and Local Sites
Local-network browsing and same-origin page-analysis site-file reads are supported intentionally. User-opened HTTP(S) links may open local destinations in the system browser. The separate external HEAD checker remains public-only and DNS-pinned.
The interactive browser uses a download Save prompt; crawl and splash contexts deny downloads. Destructive URL exclusions remain active for crawl subresources and AJAX as well as top-level pages, which can reduce rendering fidelity. Use the bounded policy diagnostics to understand excluded requests.
Maximum URLs defaults to 10,000 documents and assets combined. A limit notice means the site may not have been fully covered. Background completion does not bring the window to the foreground.