Website Crawling Rules
Website crawling is fundamental to search engines and SEO. Google crawls billions of webpages to build its search index, while SEO professionals use website crawlers to identify broken links, redirects, missing titles, duplicate content and technical problems.
But what happens when a website owner doesn't want their site crawled? Can they legally stop you, and are you breaking the law if you ignore their robots.txt file?
The answer is more nuanced than you might expect.
Is it legal to crawl a public website?
In many countries, simply using automated software to request publicly accessible webpages isn't inherently unlawful. Website crawling is, after all, how search engines discover and analyse much of the public web.
A website crawler essentially does what a web browser does: it requests a URL from a web server and receives a response. The difference is that a crawler can automate thousands of these requests and follow links to discover additional pages.
However, laws vary between countries and the fact that information is publicly accessible doesn't necessarily give you unrestricted rights to collect, store or reuse it. Depending on where you and the website are located, copyright, database rights, privacy and data-protection laws, contractual website terms and computer-access legislation may all be relevant.
For example, the European Union and UK provide legal protection for certain databases, while privacy legislation such as the EU GDPR and UK GDPR can apply when crawled information contains personal data. The United States has different laws governing computer access, copyright and contractual restrictions, and other countries have their own rules.
There is also an important distinction between crawling information that a website makes publicly available and attempting to circumvent a genuine access restriction. Bypassing passwords, authentication systems, paywalls or other technical security measures can create much more serious legal issues than simply requesting an unrestricted public webpage.
For an SEO crawler, the sensible international principle is therefore straightforward: crawl public websites responsibly, respect applicable restrictions and don't attempt to circumvent technical measures designed to prevent access.
What is robots.txt?
Most websites can publish a file called robots.txt, normally found at:
https://example.com/robots.txt
It allows the website owner to provide instructions to automated crawlers.
For example:
User-agent: *
Disallow: /private/
This tells crawlers that the website owner doesn't want them crawling URLs within /private/.
Importantly, robots.txt isn't a security system. It doesn't normally prevent the server from delivering those pages.
If you enter a disallowed URL into your browser, the page may still appear perfectly normally. A crawler that ignores robots.txt may also receive a normal 200 OK response.
The Robots Exclusion Protocol is therefore better thought of as a mechanism through which website owners communicate their crawling preferences.
Is ignoring robots.txt illegal?
Ignoring robots.txt doesn't automatically make a crawl illegal.
However, that doesn't mean crawlers should simply disregard it.
A website may have terms and conditions governing automated access, while the purpose and scale of the crawl can also matter. Copyright, database rights and data-protection rules may become relevant when large quantities of information are extracted, stored or reused.
There is also a significant difference between ignoring a robots.txt instruction and deliberately circumventing an actual technical access restriction.
Professional crawlers should therefore respect robots.txt by default.
What happens if you ignore robots.txt?
From a purely technical perspective, potentially nothing.
If the server accepts your request, it may simply return the requested page.
Website owners do, however, have numerous ways of controlling automated traffic. They can block an IP address or User-Agent, impose rate limits, use a web application firewall, return 403 Forbidden responses or respond with 429 Too Many Requests when a crawler makes excessive requests.
Continuing after a website has deliberately blocked access is considerably different from encountering a simple robots.txt instruction.
Trying to circumvent authentication, passwords, paywalls, bot protection or other technical access controls can also raise much more serious legal and ethical issues.
Why would you ever ignore robots.txt?
There are perfectly legitimate reasons.
An SEO consultant may have been specifically instructed by a client to audit their website. A developer may need to crawl a staging website. A website owner might deliberately block search engines while still wanting to perform a comprehensive technical audit.
In these circumstances, respecting robots.txt could actually prevent the SEO crawler from doing the job its user has authorised it to perform.
This is why professional crawling software may provide the ability to override robots.txt rather than permanently preventing access to disallowed URLs.
Responsible website crawling
A well-designed crawler shouldn't just attempt to download as many pages as possible as quickly as possible.
Responsible crawling means controlling the number of simultaneous requests, responding appropriately to server errors and rate limiting, identifying the crawler through its User-Agent where appropriate, and avoiding unnecessary pressure on somebody else's infrastructure.
Most importantly, a crawler shouldn't be designed to defeat genuine security or access controls.
There's a considerable difference between auditing a public website and attempting to gain access to something its owner has actually secured.
How SEO Web Spider handles robots.txt
SEO Web Spider respects robots.txt by default.
When a website indicates that a URL shouldn't be crawled, SEO Web Spider can identify that restriction rather than simply proceeding with the request.
For legitimate SEO and development work, however, there can be circumstances where authorised users need to perform a complete crawl regardless of robots.txt instructions.
The important principle is simple: crawl responsibly, respect website owners and only override restrictions when you have a legitimate reason and appropriate authority to do so.
Website crawling is an extraordinarily useful part of technical SEO. Used responsibly, it helps website owners understand exactly what search engines and users encounter — and provides the information needed to build faster, healthier and more accessible websites.
This article provides general information about website crawling in the UK and should not be regarded as legal advice.