AI Access Check
See which AI crawlers and agents a site's robots.txt allows or blocks, and what it may have missed.
Reads the domain's robots.txt and evaluates it against
RFC 9309
for every AI crawler whose user-agent its operator publishes. robots.txt is a
request, not a lock — this reports what a site asks crawlers to do, which
is not the same as what they will do.
Posture
Enforcement
Crawler matrix
Color describes access, not approval: green means a crawler can reach the site, red means it is asked not to. Which of those is right for you is the thing this table is here to help you decide.
| Bot | Operator | Status | Rules |
|---|
Findings
Nothing to flag beyond what the matrix above already shows.
Discovery files
Conventions rather than standards. None is an adopted specification, and no major crawler is known to fetch them at scale — a missing file here is not a fault.
| File | Present | Status | Size | Notes |
|---|
robots.txt
Show the file
Lines that name an AI user-agent, and the rules under them, are highlighted.
What this tool checks
robots.txt is a plain text file at the root of a site listing which crawlers may fetch which
paths. AI companies publish their own user-agent names for it, so a site can allow one and refuse another.
Those crawlers do three different jobs, and the difference matters. Training crawlers collect pages in bulk to build a model, and send no visitors back. Search crawlers build the index an AI answer engine cites from, so blocking one usually costs you the citation and its link. User-initiated fetches happen because a person asked an assistant about a specific page; operators generally treat those as visits rather than crawls, and say robots.txt may not apply to them.
llms.txt and the other discovery files are community conventions, not standards. They are worth
publishing if you want one, but no major crawler is known to fetch them at scale, so treat them as
documentation of your intent rather than as a control.
Above all, robots.txt is honor-based. It is a request that well-behaved crawlers choose to respect, and nothing in it stops a crawler that does not — only something in front of your site, such as a CDN rule or a WAF, can turn a request into enforcement.