We use cookies
Some are needed to run the site and keep you signed in. Others help us understand what’s useful through analytics. We don’t run analytics until you agree. Cookie details
Reads your robots.txt for the AI crawlers specifically: which ones can train on your writing, which can actually open a page to answer or cite it, and which rule is quietly blocking each one.
This check is part of the membership
Members run it on their own pages, with every other check, the decision tools and the courses.
robots.txt was written for search engines, and almost nobody has reread it since ChatGPT, Claude and Perplexity started fetching pages too. A rule copied from an old template can block an AI answer engine nobody meant to name, while a training crawler nobody named either keeps reading everything by default.
It reads the rules as written. Some AI crawlers have a documented history of crawling past the limits set for them, so a block here is a request, not a lock.
