Get Permission First
Only crawl a website if:
- you own it
- you manage it for a client
- the owner has given you permission
- you otherwise have a lawful and reasonable basis to crawl it.
Technical ability to crawl a website is not permission.
Check the Site's Rules
Before crawling, review:
- website terms
- robots.txt directives
- published crawling or API policies
- access restrictions
- rate-limit instructions
The app can report robots.txt information, but you remain responsible for deciding whether a crawl is authorised and appropriate.
Start Gently
Use Gentle mode for a first crawl. Increase speed only when you know the target site can handle the request volume and you have permission to crawl at that rate.
Rendered crawling can generate many HTTP requests because each page may load scripts, stylesheets, images, fonts and other resources.
Understand IP Risk
Crawling originates from your device and public IP address. Target websites, hosting providers, network operators and security services may:
- rate-limit requests
- block requests
- blacklist an IP address
- block or blacklist a user agent
- suspend an account
- contact the site owner or network operator
- make a complaint
London Web Factory is not responsible for those actions or consequences.
Avoid Harmful Activity
Do not use the app to:
- access non-public content without authorisation
- bypass authentication, CAPTCHAs, paywalls or rate limits
- overload a service
- probe for vulnerabilities without explicit authority
- collect personal, confidential or special-category information unlawfully
- misrepresent your authority to crawl
Watch the Crawl
While a crawl runs:
- monitor the Crawled, Queued and Errors counts
- stop or pause the crawl if the site slows down
- reduce speed if you see unusual errors
- respond quickly to complaints or warning messages
Protect Sensitive Data
Crawl results can contain sensitive URLs, query strings, page titles, anchor text and response headers. Review exports before sharing them.