Noindex vs Nofollow vs Disallow: When to Use Each
Three SEO controls, three different jobs. Pick the wrong one and you leak pages into the index, waste crawl budget, or hide content so thoroughly Google can never drop it.
#robots.txt
Articles tagged with #robots.txt
Meta Robots vs X-Robots-Tag — Which One Wins
When `<meta name="robots">` and the `X-Robots-Tag` HTTP header conflict, Google merges them and takes the most restrictive value. How to diagnose it with curl and keep both in sync.
Sitemap Lists URLs That Carry noindex (How to Fix the Conflict)
Your sitemap.xml lists URLs that render `<meta name="robots" content="noindex">`. Google indexes some and not others. Why it happens and how to make both signals agree.
Next.js Sitemap and robots.txt: The App Router Way (2026)
A correct sitemap and robots.txt decide whether Google indexes your Next.js site. Here are the App Router idioms for both, verified against Next.js 16.
robots.txt — What to Put, What to Never Put (2026)
A surgical guide to robots.txt for indie sites: the working default, the rules that quietly deindex you, robots.txt vs noindex, and the AI-crawler tokens (GPTBot, ClaudeBot, Google-Extended) that matter in 2026.
Should Tag Pages Be Noindex? (For Most Sites, Yes)
When tag archives help SEO and when they hurt — plus the exact noindex pattern, the robots.txt mistake to avoid, and copy-paste code for WordPress, Astro, and Next.js.