Which AI crawlers does this site actually allow?
Every site has a robots.txt and almost nobody reads it correctly. MurmCrawl fetches the file from the site’s own origin, resolves it against each named AI crawler the way the standard says a crawler must, and tells you which are allowed, which are blocked, and which were never mentioned at all.
Silence is permission. robots.txt is an opt-out protocol: anything not disallowed is allowed. A crawler that goes unmentioned is permitted, and MurmCrawl says so rather than reporting it as unknown — because “unknown” would let a site look more protected than it is.
What it resolves
GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot, Applebot-Extended and the rest, each against the rules that actually apply to it. The verdict names the line it rests on, so a disputed answer can be checked against the file in one step.
The rule everyone implements wrong
Under RFC 9309, the rules for one user-agent are the union of every group that names it. A file that splits its rules across two blocks — entirely ordinary in hand-written robots.txt — is misread by any parser that takes only the first match, and the answer it gives can be the exact opposite of what the file says. MurmCrawl merges the groups.
Headers count too
A site can also refuse crawlers with an X-Robots-Tag response header, which never
appears in the file and is invisible to anyone reading robots.txt alone. MurmCrawl reads both,
and says which source each verdict came from.
What it refuses to tell you
- Whether a crawler obeyed. robots.txt is a request, not a fence. This reports what the site asked for, never what any bot did.
- Whether your content was used for training. Not observable from a browser, not inferable from a file, not claimed.
- A verdict for a crawler the file does not name and no wildcard covers. That is reported as allowed, with the reason.
Why it needs no host permissions
The request is made from inside the page you already have open — the site asking its own server for its own file, which is same-origin and needs nothing extra. Fetching robots.txt from a popup would be cross-origin and would require access to every site you visit. That difference is the whole reason this extension asks for two permissions instead of the world.
Free, with no paid tier. Two permissions — activeTab and
scripting — and no host permissions at all.
Questions
How do I check if a site blocks ChatGPT?
Open the site and click the icon. MurmCrawl resolves its robots.txt against GPTBot and names the rule the verdict rests on.
Does robots.txt actually stop AI crawlers?
It is honoured voluntarily. MurmCrawl reports what the file asks for, not what any bot did.
What if a crawler is not mentioned?
Then it is allowed. Anything not disallowed is permitted.
Is it free?
Yes, with no paid tier, no account and no server.