robots.txt tester

Is robots.txt blocking Google?

Type an address. We read the site's robots.txt and tell you whether a rule shuts every crawler out of the whole site. Free, no signup, nothing is stored.

One page per check. We fetch it the way a crawler does, without a browser, and keep nothing.

What it checks

robots.txt is a plain text file at the root of your site that tells search engine crawlers which paths not to crawl. The tool reads the "User-agent: *" group, which applies to every crawler. If that group says "Disallow: /", the entire site is closed to crawling.

Why it matters

That line is usually put on a staging site on purpose and forgotten when the site goes live. Google stops crawling: pages may linger in the index, but their content is no longer refreshed and new pages are never added. Traffic falls quietly over weeks rather than overnight, which is why it is noticed late.

How to fix it

  1. Open the file in a browser: yoursite.com/robots.txt.
  2. Remove the "Disallow: /" line under "User-agent: *", or list only the paths you mean to close (for example "Disallow: /wp-admin/").
  3. Check Search Console's robots.txt report to see which version Google fetched, and request a recrawl.
Questions

What people ask next.

Is "Disallow: /" always a mistake?

No. On a test or staging site it is exactly right. The problem is the same file reaching the live site.

My site has no robots.txt. Is that a problem?

No. Without the file, crawlers treat everything as allowed, and this tool reports it as fine.

What about a block on a single bot, such as AI crawlers?

The tool only reads the "User-agent: *" group that applies to every crawler. A block on one named bot is a decision, not an accident, so it is not flagged.

Not once. Every day.

This check is today's snapshot. Crawlens runs the same checks on your clients' sites on a schedule and tells you, with the evidence, when something breaks.

Start monitoring free ↗