Does robots.txt actually stop AI crawlers?

Only the ones that choose to obey it. robots.txt is a voluntary protocol with no enforcement: it is a sign on a door, not a lock. The major operators publish named user-agents and state that they honour it, which makes a disallow rule effective against them. It does nothing about a crawler that ignores the file, and nothing at all about content already collected.

Can two robots.txt blocks for the same crawler conflict?

They do not conflict — they combine. Under RFC 9309, the rules that apply to a user-agent are the union of every group that names it. A file that disallows a path in one block and allows a different path in another block for the same agent is applying both, which is not what most people expect when they write a second block, and is a common source of files that behave differently from how they read.

What is X-Robots-Tag and how is it different?

It is an HTTP response header carrying the same kind of directive per response rather than per site. It matters because it can say things robots.txt cannot express, it applies to non-HTML files, and it is invisible in the page source — so a site can be sending instructions no one reading the markup would ever see.

A sign on a door, not a lock

Q: If robots.txt says nothing about a crawler, is it allowed?

Yes. Silence is permission. robots.txt is an opt-out protocol, so a crawler with no matching rule is permitted by default — and most sites that believe they have blocked AI training have simply never named the relevant user-agent. The absence of a rule is the most common outcome and the least visible.

robots.txt is the file everyone reaches for when they want AI companies to stop reading their site. It is worth having, it works better than people assume, and it does none of the things most site owners believe it does.

2 September 2026 · from the parser inside MurmCrawl

Silence is permission. robots.txt is an opt-out protocol. A crawler with no rule naming it is allowed — so most sites that believe they have blocked AI training have simply never named the user-agent. The absence of a rule is the commonest outcome and the hardest to notice.

What it can do

More than the cynics say. The major AI operators publish named user-agents and state that they honour robots.txt, which means a correctly written disallow rule genuinely stops them. It is a voluntary protocol observed by the parties who have the most to lose from being caught ignoring it. That is not nothing.

What it cannot do

Stop a crawler that ignores it. There is no enforcement mechanism. A disallow rule is a request, and a request can be declined silently.
Retrieve anything already taken. A rule added today says nothing about content collected last year. It is a fence, not a recall.
Distinguish training from search. Some operators run separate agents for indexing and for model training; some do not. Blocking one may cost you the other, and the file cannot express “index me but do not train on me” unless the operator has chosen to offer two agents.
Hide anything. robots.txt is public, and it is a list of the paths you would rather people did not visit. Disallowing /admin-secret publishes its existence to everyone who reads the file.

The rule most hand-written files get wrong

Under RFC 9309, the rules applying to a user-agent are the union of every group that names it. Not the first match — all of them, combined.

So a file like this:

User-agent: SomeBot
Disallow: /private/

# ...forty lines later...

User-agent: SomeBot
Allow: /private/public-bit/

is applying both blocks, which is usually what the author wanted. But the same mechanism means a second block written to replace the first does not replace it — it adds to it. Reading only the first matching group, which is the intuitive thing to do, reports the opposite of what the file actually says. That is a genuinely easy bug to write into a parser, and we wrote it once before catching it.

The header nobody reads

X-Robots-Tag carries the same kind of directive in an HTTP response header rather than a site-wide file. It matters for three reasons: it is per-response rather than per-site, it applies to files that have no HTML to put a meta tag in, and it is invisible in the page source. A site can be issuing instructions that nobody reading the markup would ever see, which is why any honest check has to make a request and read the headers rather than only fetching one file.

How to actually check a site

Fetch /robots.txt and read every group, not the first match.
For each AI user-agent you care about, take the union of all groups naming it.
Check the response headers for X-Robots-Tag.
Treat any agent with no matching rule as allowed, and say so — that is the answer people most need and least expect.

MurmCrawl does this on the site you are looking at. It reads the file, merges every group per agent, checks the header, and reports the agents that are permitted by silence — because those are the ones nobody knew about.

How MurmCrawl works →→

Questions

Does robots.txt stop AI crawlers?

The ones that choose to obey it, which includes the major named operators. There is no enforcement.

What if it says nothing about a crawler?

Then that crawler is allowed. Silence is permission.

Can two blocks for one crawler conflict?

They combine rather than conflict — the union of every group naming the agent.

What is X-Robots-Tag?

The same kind of directive in an HTTP header, invisible in the page source.