Why is Googlebot blocked when there is no Disallow?

You did not write a single Disallow, and Googlebot still will not touch the page. This is almost never a robots.txt problem. There are five places a block hides, and only one of them is the file you are staring at.

First, tell the two symptoms apart

“No crawl” and “no index” look similar in a report and have different causes. Fixing a crawl block will not help an index block.

Symptom	Meaning
Googlebot never requests the URL	Something blocked the fetch (CDN, WAF, robots.txt, 5xx)
Googlebot requests it, page still absent	Something blocked indexing (X-Robots-Tag, meta noindex)

Get the raw picture before guessing. Fetch the URL exactly as Google would:

# 用 Googlebot UA 请求，观察状态码与响应头
curl -sIL -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  https://example.com/page \
  | grep -iE '^(HTTP|location|x-robots-tag|content-type|server|cf-)'

If the status is 403, 429 or a challenge page, the block is upstream. If it is 200 with an x-robots-tag, the block is in the headers.

1. CDN and WAF rules

A firewall rule, bot-management feature or rate limit can refuse or challenge the Googlebot user agent before your origin ever replies. Because it lives at the edge, robots.txt stays clean and the origin logs show nothing. Look for:

• A “block known bots” or “bot fight” toggle that treats Googlebot as hostile.
• Geo-blocking that excludes the countries Google crawls from.
• A rate limit tripped by a crawl burst, returning 429.

Verify against the published Googlebot IP ranges rather than trusting the user-agent, since the user-agent is trivially spoofed.

2. X-Robots-Tag response headers

This header carries the same directives as a meta robots tag, but it is delivered as HTTP and works on any file type — including PDFs and images, where a meta tag cannot exist:

# nginx —— 给整个 /private/ 目录打上 noindex
location /private/ {
    add_header X-Robots-Tag "noindex, nofollow";
}

Because it is set once in server or CDN config, it commonly survives long after the reason for it is gone. One header on a broad location block silently deindexes everything underneath it. Check the response headers of the exact URL, not the site root.

3. Meta robots tags

A noindex in the <head> keeps a page out of the index even when crawling works perfectly. It is easy to miss because it can be injected by a CMS, a theme, or a page-level toggle rather than hand-written:

<meta name="robots" content="noindex, nofollow">

The crawler still fetches the page — it has to, in order to read the tag — so the fetch appears in logs while the page never indexes. If that is the pattern you see, look at the HTML, not the network.

4. A robots.txt served as HTML

If your site uses a single-page-app fallback, unknown paths may return index.html with a 200. That includes /robots.txt. The crawler gets HTML where it expected plain text and cannot parse any group.

# 期望 text/plain；如果是 text/html，说明被兜底路由接管了
curl -sI https://example.com/robots.txt | grep -i content-type

Serve a real text/plain file, or return a proper 404. Note the related trap: a robots.txt that returns a 5xx can make Google pause crawling the site entirely, while a 404 is treated as “no rules”.

5. Inherited reverse-proxy config

A staging deploy, a legacy vhost or a leftover proxy block can carry its own robots.txt or headers onto production. When a block appears without a matching commit, diff what the live server returns against what your repository says it should. The two are frequently different files.

Common mistakes

Debugging robots.txt first. When there is no Disallow in the file, the block is elsewhere. Start with the response headers and status code.

Trusting the user-agent alone. CDNs validate Googlebot by reverse DNS or published IP ranges. A spoofed UA can behave differently from the real crawler, so test with the documented ranges.

Checking the homepage instead of the URL. X-Robots-Tag and meta tags are often scoped to one path. Inspect the exact page.

Leaving a staging Disallow: / or noindex in production. A config that ships with the build is the usual culprit.

Assuming a 404 on robots.txt is fine to ignore. It is fine — it means allow-all — but a 5xx is not, because it can stop crawling site-wide.

Forgetting that a CDN caches headers. After fixing a header, purge the edge cache or you will keep serving the old one.

Where this tool fits

The robots.txt tester resolves whether a given user-agent is allowed for a specific URL and shows the rule that decided it, so you can rule robots.txt out quickly. When it comes back allowed and the page is still refused, the block is in the headers or the network — check the X-Robots-Tag and the CDN firewall next.