We use cookies
Some are needed to run the site and keep you signed in. Others help us understand what’s useful through analytics. We don’t run analytics until you agree. Cookie details
Reads the one file that can remove an entire site from search and tells you what it is really doing: what it blocks, what it lets through, and what it forgot to say.
This check is part of the membership
Members run it on their own pages, with every other check, the decision tools and the courses.
One line in this file can remove an entire site from search, and the mistake is invisible: the site keeps working perfectly for everybody except the crawler. Blocking the CSS and JavaScript is the subtler version, because the page still gets crawled and then gets judged on how it looks without them.
It reads the rules as written. Whether a crawler obeys them is its own decision, and the ones that ignore robots.txt were never going to read it.
