AI AI Tools Guidebook
Troubleshooting

robots.txt Blocks CSS/JS and Indexing Quality Drops

You disallowed `/assets/` or `/_next/static/` to "save crawl budget," so Googlebot can't render the page. Fastest fix: stop blocking render resources and re-test in URL Inspection.

Published: Updated: Author: AI Productivity Guide Team 🌐 查看中文版本

TL;DR: A Disallow: rule in robots.txt is hiding your CSS, JS, or JSON data endpoints from Googlebot. It renders the page with a broken layout, can’t see JS-injected content, and may flag it as not mobile-friendly. Fastest fix: open Search Console → URL Inspection → Test Live URL → View tested page → More info → Page resources, find anything marked Blocked by robots.txt, then add Allow: rules (or delete the Disallow:) so Googlebot can fetch those paths. Then Request Indexing. Remember Google caches robots.txt for up to 24 hours, so the live test reflects your change faster than the indexed report does.

You inherited a robots.txt that disallows /assets/, /_next/static/, or /wp-content/plugins/. The argument was “Googlebot doesn’t need to crawl static assets — save crawl budget.” But Googlebot has to FETCH your CSS and JS to render the page. Without them it sees a layout-less, half-broken DOM, can’t separate main content from boilerplate, can’t run the JS that injects content, and may mark the page mobile-unfriendly. URL Inspection will show Page resources couldn't be loaded.

Google has been explicit since 2014: don’t block CSS, JS, or images that affect rendering. The “crawl budget” intuition is wrong here. Googlebot now renders with an evergreen Chromium engine, and blocked render resources also make a page weaker for AI Overviews, which extract signals from the rendered page, not the raw HTML.

Which bucket are you in?

Symptom in URL InspectionLikely causeJump to
CSS/JS listed under “Other resources” as Blocked by robots.txtDisallow rule covers your asset pathCause 1, 2, 7
Resources on cdn.example.com blocked, main host cleanCDN host has its own restrictive robots.txtCause 3
Resource returns 403/401, not “blocked by robots.txt”WAF/firewall blocking the bot, not robots.txtCause 4
Rendered HTML is blank, API calls failingDisallow: /api/ blocks the client data fetchCause 6
Versioned asset main.css?v=abc blockedDisallow: /*? wildcard catches query stringsCause 7

Common causes

1. Legacy “block /assets/” rule from a 2010-era SEO guide

The directive looked clean: stop wasting crawl on bundles. It made sense in 2010 when Google didn’t render JS. It has been outdated since 2015.

How to spot it: curl https://yoursite.com/robots.txt. If you see Disallow: /assets/, Disallow: /static/, Disallow: /_next/, or Disallow: /wp-content/plugins/, you found it.

2. Wildcard rules accidentally block CSS

A rule like Disallow: /*.json$ meant to block API responses also blocks manifest.json, webpack-runtime.json, or build-critical config files.

How to spot it: List any Disallow: /*.<extension> rules. Test each against your actual static file paths.

3. CDN subdomain disallowed in its own robots.txt

Your assets live on cdn.example.com. Per Google’s spec, robots.txt rules apply only to the exact host, protocol, and port where the file is served, so Googlebot reads cdn.example.com/robots.txt when fetching CSS from there. If no one configured it and it returns a blanket disallow, the assets are blocked even though your main robots.txt is clean.

How to spot it: curl https://cdn.example.com/robots.txt. If it is User-agent: * followed by Disallow: /, your assets are blocked there.

4. Fastly / Cloudflare WAF rule blocking Googlebot from /static/

Not robots.txt, but functionally equivalent. A WAF rule blocks bot user-agents from asset paths to prevent hotlinking, and Googlebot gets a 403.

How to spot it: curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://yoursite.com/static/main.css. If the status is 403 / 401, the WAF is blocking, not robots.txt. URL Inspection reports these as failed fetches, not “blocked by robots.txt” — that label distinction is your fastest tell.

5. Multiple robots.txt files in the repo

A robots.txt is only honored when served from the host root (https://yoursite.com/robots.txt). A second file at https://yoursite.com/app/robots.txt is ignored by crawlers, but a misconfigured server that serves different root content per path or per environment can still ship the wrong rules to production.

How to spot it: Search the repo for every robots.txt. There should be exactly one served at each host root. Confirm the live one with curl -s https://yoursite.com/robots.txt.

6. Disallow /api/ blocks JS data-fetch endpoints

Common for SPAs: the page renders only after fetching /api/page/foo. Blocking /api/ means Googlebot can fetch the JS but the client-side render still produces an empty page because the data call is blocked.

How to spot it: The page is mostly client-rendered AND robots.txt has Disallow: /api/. In URL Inspection, the rendered screenshot is blank and the XHR endpoints show as blocked.

7. Disallow: /*? blocks query strings on resources

Tries to deduplicate parameter URLs. Side effect: cache-busted assets like main.css?v=abc123 get blocked.

How to spot it: Check version-tagged asset URLs. If they include ?, the wildcard query rule blocks them.

Shortest path to fix

Step 1: Audit current robots.txt

curl -s https://yoursite.com/robots.txt

Identify every Disallow: line. For each, ask: “Does Googlebot need to fetch this to render or understand the page?” Note that Google enforces a 500 KiB limit on robots.txt; anything after that is ignored, so a bloated file can have rules that never apply.

Step 2: Allow rendering-critical resources explicitly

Whitelist the paths Googlebot needs to render. Google resolves conflicts by the longest (most specific) path match, and on a tie picks the least restrictive rule, so a specific Allow: beats a broad Disallow::

User-agent: *
Disallow: /admin/
Disallow: /private/

Allow: /assets/
Allow: /static/
Allow: /_next/static/
Allow: /api/articles/
Allow: /wp-content/themes/
Allow: /wp-content/plugins/

Put render-critical Allow: rules in the User-agent: * group so every crawler (including Bingbot and AI crawlers) gets them, not just Googlebot.

Step 3: Test rendering with URL Inspection

The old standalone robots.txt Tester was removed in late 2023; use these two tools instead.

  1. Search Console → URL Inspection → enter a key page → Test Live URL → View tested page → More info → Page resources. Any resource tagged Blocked by robots.txt is hurting rendering. Compare the Screenshot tab against how the page should look.
  2. Search Console → Settings → robots.txt report to see the file Google last fetched per host (top 20 hosts), its fetch status, and any parse warnings. Use its Request a recrawl button after editing the file.

Step 4: Verify the CDN’s robots.txt

curl https://cdn.example.com/robots.txt

If it returns a blanket disallow, fix the CDN-side robots.txt, or remove it so the path returns 404 — Google treats a 4xx response (other than 429) as “no restrictions, crawl allowed.”

Step 5: Whitelist Googlebot in your WAF

In Cloudflare: Security → Bots and allow the Verified Bots category (or add a WAF skip rule for cf.client.bot). In Fastly: a VCL rule allowing requests whose User-Agent matches Googlebot. Always confirm a bot is real with a reverse-DNS lookup (the IP must resolve back into googlebot.com or google.com), because the user-agent string alone is spoofable.

Step 6: Replace broad wildcards with targeted rules

Swap an overly broad Disallow: /*.json$ for specific paths like Disallow: /api/admin.json. Test the wildcard against your actual file inventory before shipping, since /* wildcards silently catch hashed bundle names and ?v= query strings.

Step 7: Resubmit and watch

After fixing robots.txt, click Request Indexing in URL Inspection on a few key pages. Because Google caches robots.txt for up to 24 hours, the indexed report lags; the live test updates immediately. Within roughly 1-2 weeks the Page resources blocked warnings should clear and pages should re-render correctly.

How to confirm it’s fixed

  1. Live test passes: in URL Inspection → Test Live URL → Page resources, no rendering-critical CSS/JS shows Blocked by robots.txt.
  2. Screenshot looks right: the rendered screenshot in URL Inspection matches your real layout, not a stripped-down page.
  3. Direct fetch as Googlebot returns 200: curl -s -o /dev/null -w "%{http_code}" -A "Googlebot/2.1 (+http://www.google.com/bot.html)" https://yoursite.com/_next/static/css/main.css should print 200.
  4. robots.txt report is clean: Settings → robots.txt report shows status Fetched with no errors for every host that serves assets.

When this is not on you

Crawl-budget concerns are real for sites with millions of URLs and tight server capacity, but you handle them through internal linking, sitemaps, and pruning low-value parameter URLs — never by blocking rendering resources. Don’t apply enterprise-scale tactics to a 5k-page site.

Easy to misdiagnose as

A “JS not rendered” issue. The symptoms overlap (blank rendered HTML), but the root cause differs: here Googlebot couldn’t fetch the JS or its data at all, not that the JS ran and produced nothing. The tell is the Blocked by robots.txt label in Page resources. Check robots.txt first when JS rendering issues appear.

Prevention

  • Default robots.txt should allow everything except /admin/, /private/, and search-results pages.
  • Never disallow /assets/, /static/, /_next/, /wp-content/themes/, or /wp-content/plugins/.
  • Add a CI check that fetches robots.txt and asserts known asset paths are not disallowed.
  • Verify the CDN host’s robots.txt during launches and migrations.
  • Re-audit robots.txt yearly as asset paths evolve.

FAQ

  • Will Google penalize me for excessive crawling if I allow everything? No. Google scales its own crawl rate automatically, and assets are scheduled separately from HTML. Allowing CSS/JS does not meaningfully increase crawl load.
  • Should I disallow /wp-admin/? Yes, that’s fine — it isn’t needed for rendering. WordPress already allows /wp-admin/admin-ajax.php by default, which some themes need, so don’t blanket-block the whole admin path without checking.
  • I fixed robots.txt but URL Inspection still shows blocked. Why? Google caches robots.txt for up to 24 hours. Use Test Live URL (it refetches), and request a recrawl from the robots.txt report to speed it up.
  • Does blocking CSS/JS cause a manual penalty? No, it’s not a penalty — it degrades rendering quality, which lowers how well Google understands and ranks the page, and can drop it from AI Overviews.
  • Is noindex in robots.txt the same as Disallow? No. Disallow blocks crawling; Google ignores noindex directives placed inside robots.txt. To keep a crawlable page out of the index, use a noindex meta tag or X-Robots-Tag header instead.

Tags: #SEO #Troubleshooting #Indexing #Search Console #robots-txt #rendering

Related Articles