AI AI Tools Guidebook
Troubleshooting

ads.txt Redirect Breaks AdSense Verification

AdSense reports ads.txt as missing even though browsers load it fine. The crawler refuses 3xx redirects; host the file at the canonical root with a direct 200.

Published: Author: AI Productivity Guide Team 🌐 查看中文版本

You added ads.txt to your repo, deployed, and confirmed in a browser that visiting https://your-domain.com/ads.txt shows the expected google.com, pub-XXX, DIRECT, ... line. Hours later, AdSense → Sites still flags the domain with “ads.txt status: Not found” or “Earnings at risk.” You re-deploy. You wait another day. Same warning. The culprit is almost always a redirect: your hosting platform 301s http:// to https://, or strips the www, or normalizes the path, and AdSense’s crawler refuses to follow any of those for ads.txt. The IAB spec is strict — ads.txt must return HTTP 200 directly at the canonical root, no 3xx hops allowed.

Common causes

Ordered by how often each pattern triggers the failure.

1. Apex-to-www (or vice versa) redirect

You serve from https://www.example.com but AdSense crawled https://example.com/ads.txt, which 301s to the www host. The crawler does not follow the redirect for ads.txt and records “missing.”

How to spot it: Run curl -I https://example.com/ads.txt and curl -I https://www.example.com/ads.txt. If one returns 301/302 to the other, you have this problem.

2. HTTP-to-HTTPS redirect on the ads.txt path

Your site forces TLS via a 301 from http://example.com/ads.txt → https://example.com/ads.txt. AdSense may probe HTTP first and refuse the redirect.

How to spot it: curl -I http://example.com/ads.txt returns 301 Location: https://.... Most modern hosts do this by default and it is usually fine, but some AdSense crawler paths fail on it.

3. Trailing-slash or case normalization redirect

Hosts like Netlify, Vercel, or Cloudflare Pages sometimes 301 /ads.txt/ → /ads.txt or normalize uppercase paths. If AdSense ever happens to probe with an odd path, it gets the redirect and fails.

How to spot it: curl -I https://example.com/Ads.txt (capital A) and curl -I https://example.com/ads.txt/ (trailing slash). If either returns 301/302, the platform normalizes.

4. CDN-level redirect for “bot” user agents

A bot-protection rule (Cloudflare, AWS WAF, Akamai) sends crawlers to a captcha or interstitial. The AdSense crawler hits the rule, gets a 302 to /captcha, and reports the file as missing.

How to spot it: curl -I -A "Mediapartners-Google" https://example.com/ads.txt. If the response includes cf-mitigated, x-firewall-action, or a 302 to a challenge page, the WAF is the cause.

5. SPA framework intercepting unknown paths

Single-page apps configure a catch-all that serves index.html for any unknown path. If ads.txt is not in your build output, the SPA returns 200 but with HTML content — AdSense parses it, sees no valid record, and flags the file as malformed.

How to spot it: curl https://example.com/ads.txt | head -3. If you see <!DOCTYPE html> instead of google.com, pub-XXX, ..., the SPA is swallowing the path.

6. Caching layer serving stale 404 from before you added the file

Your CDN cached a 404 from before ads.txt existed. Even after deployment, the edge keeps returning the stale 404 for hours.

How to spot it: curl -I https://example.com/ads.txt shows cf-cache-status: HIT and a 404, while the origin direct fetch returns 200.

Before you start

  • Confirm your canonical domain — apex (example.com) or www (www.example.com). AdSense should be configured for the canonical one.
  • Know which DNS provider, host, and CDN are in the chain.
  • Have the publisher ID handy: pub-XXXXXXXXXXXXXXXX (visible at AdSense → Account).

Information to collect

  • Output of curl -IL https://example.com/ads.txt (the -L follows redirects so you see every hop).
  • Output of the same against the www variant.
  • Output of curl -I -A "Mediapartners-Google" https://example.com/ads.txt (simulate AdSense’s UA).
  • Hosting platform name (Vercel, Netlify, Cloudflare Pages, S3, custom Nginx, etc.).
  • Your current ads.txt content (first 5 lines).
  • The exact warning text from AdSense → Sites → your domain.

Step-by-step fix

Ordered from cheapest to most invasive.

Step 1: Verify the canonical URL returns 200 directly

The AdSense crawler probes https://<your-domain>/ads.txt at your registered domain. Confirm both apex and www return 200:

curl -IL https://example.com/ads.txt
curl -IL https://www.example.com/ads.txt

If either shows a 301 or 302 line in the trace, that variant is broken. The fix path depends on which side is canonical.

Step 2: If you redirect apex→www (or vice versa), host ads.txt on BOTH

The crawler may probe either side. The simplest fix is to serve a real ads.txt at both example.com/ads.txt and www.example.com/ads.txt, bypassing the redirect for that specific path.

On Nginx:

server {
  server_name example.com;

  # Normal redirect everything to www...
  location / {
    return 301 https://www.example.com$request_uri;
  }

  # ...EXCEPT ads.txt
  location = /ads.txt {
    alias /var/www/ads.txt;
  }
}

On Vercel, add to vercel.json:

{
  "redirects": [
    { "source": "/(.*)", "destination": "https://www.example.com/$1", "permanent": true, "missing": [{ "type": "header", "key": "x-skip-redirect" }] }
  ],
  "rewrites": [
    { "source": "/ads.txt", "destination": "/ads.txt" }
  ]
}

On Cloudflare Workers, add an exception rule for the /ads.txt path that serves the file directly.

Step 3: Disable any path normalization redirects

If your host 301s case or trailing-slash variants, exempt ads.txt. On Netlify, _redirects:

# Force trailing slash on everything EXCEPT ads.txt
/ads.txt   /ads.txt   200
/* /:splat/ 301

The 200 status keeps it as a direct serve, not a redirect.

Step 4: Whitelist the AdSense crawler in your WAF

If a bot-protection rule is firing, allow the AdSense crawler UAs:

  • Mediapartners-Google
  • AdsBot-Google
  • Googlebot (sometimes used for ads.txt verification)

On Cloudflare → Security → WAF → Tools → Allowlist, add a custom rule:

(http.request.uri.path eq "/ads.txt") or (http.user_agent contains "Mediapartners-Google")

Set action to “Skip” → check “Bot Fight Mode,” “Managed Rules,” “Rate Limiting.”

Step 5: Make ads.txt a static file your SPA does not intercept

For Next.js, place ads.txt in public/:

public/
  ads.txt

Next serves files in public/ directly without invoking the React app. Verify in production:

curl https://example.com/ads.txt
# expected: google.com, pub-XXX, DIRECT, f08c47fec0942fa0

For Astro, place in public/ as well. For Gatsby, same. For pure SPAs (Vite, CRA) the file must also live in public/.

For a server-rendered framework, ensure your router does not match /ads.txt:

// Express example
app.get("/ads.txt", (req, res) => {
  res.type("text/plain");
  res.sendFile(path.join(__dirname, "ads.txt"));
});

Step 6: Purge the CDN cache for the path

Even after a fix, stale cache can keep the warning alive:

# Cloudflare
curl -X POST "https://api.cloudflare.com/client/v4/zones/ZONE_ID/purge_cache" \
  -H "Authorization: Bearer TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"files":["https://example.com/ads.txt","https://www.example.com/ads.txt"]}'

For Vercel / Netlify, redeploying triggers a purge automatically. For S3 + CloudFront, invalidate /ads.txt from the CloudFront console.

Step 7: Wait for AdSense recrawl

After all the above, AdSense recrawls ads.txt within 24-48 hours. Don’t keep redeploying — give it a day. Check status at AdSense → Sites → your domain.

Verify

  • curl -IL https://your-canonical-domain/ads.txt shows a single 200 OK with no 301 or 302 in the trace.
  • The body of the file is plain text starting with google.com, pub-..., not HTML.
  • AdSense → Sites → your domain shows “ads.txt status: Authorized” within 48 hours.
  • curl -A "Mediapartners-Google" https://your-domain/ads.txt returns the same 200 + plain text, confirming the crawler UA is not blocked.

Long-term prevention

  • Use a single canonical domain (pick apex OR www) and configure AdSense for that exact host from day one.
  • Keep ads.txt in your repo and version-controlled — never edit through the host’s web UI.
  • After any platform / CDN migration, re-run the curl checks above as part of the cutover checklist.
  • Add a synthetic monitor (UptimeRobot, Pingdom) that hits /ads.txt daily and alerts on non-200 or wrong content-type.
  • Document the canonical host in your team’s runbook — many redirect mistakes happen during DNS edits.
  • If you use a CMP that injects HTTP headers, confirm it does not touch responses to /ads.txt.

Common pitfalls

  • Editing ads.txt in a CMS-managed file editor that adds a UTF-8 BOM at the start — the BOM byte makes AdSense see the line as malformed.
  • Adding Content-Type: text/html to the response — AdSense expects text/plain (or any text type; HTML works in browsers but signals “wrong file”).
  • Putting ads.txt in a subdirectory like /static/ads.txt and rewriting /ads.txt to it — the rewrite is invisible to crawlers but the rewrite logs sometimes register as a redirect.
  • Listing only one of apex vs www in AdSense Sites while your canonical redirect runs the other way — fixable by registering both.
  • Forgetting to invalidate the CDN after fixing the redirect, then assuming “the fix didn’t work” when it is just stale cache.

FAQ

Q: I see a single 200 from curl but AdSense still says “Not found.” What now?

Try curl -A "Mediapartners-Google" -I <url> — if you get a different response (302, 403, etc.), a WAF is treating the crawler differently. Whitelist the UA as in Step 4. See adsense ads not showing for the downstream symptom this causes.

Q: Do I need an ads.txt for every subdomain?

No — only for the host you registered in AdSense Sites. If you registered www.example.com, the file must be at www.example.com/ads.txt. Subdomains like blog.example.com need their own ads.txt only if they are also AdSense-registered.

Q: Can I use a sellers.json file instead?

sellers.json is for SSPs / exchanges, not publishers. For AdSense, you need ads.txt. Both can coexist but ads.txt is the required one for publisher verification. See ads.txt not found for the simpler “file just isn’t there” variant.

Q: How long until AdSense rechecks after I fix it?

24-72 hours typically. The status field updates asynchronously. Don’t disable / re-enable ads units in the panel during this window — it just creates noise without speeding up the recheck.

Q: My redirect is needed for SEO — am I really stuck hosting at both hosts?

Yes for ads.txt specifically. The two-line exception (one for apex, one for www) costs nothing and unblocks verification. SEO canonicalization works through <link rel="canonical"> and Search Console settings; it does not require redirecting the ads.txt path.

Tags: #AdSense #ads.txt #verification #redirects #Troubleshooting

Related Articles