Noindex vs Nofollow vs Disallow: When to Use Each
Three controls, three different jobs. Pick the wrong one and you either leak pages into the index, waste crawl budget, or hide content from yourself by accident.
#robots.txt
Articles tagged with #robots.txt
Meta Robots vs X-Robots-Tag — Which One Wins
When `<meta name="robots">` and `X-Robots-Tag` HTTP header conflict, results are unpredictable. How to decide which to use and how to keep them in sync.
Robots Meta and Sitemap Are Saying Different Things
Sitemap.xml lists URLs that have `<meta name="robots" content="noindex">`. Google reports them, but inconsistently. Why this happens and how to fix.
Sitemap and robots.txt Basics in Next.js
A correctly served sitemap and robots.txt are non-negotiable for SEO. Here is the App Router idiom for both in 2026.
robots.txt — What to Put, What to Never Put
A surgical guide to robots.txt for indie sites — the two-line default that works, the rules that quietly deindex you, and the difference vs noindex.
Should Tag Pages Be Noindex? (Hint: Probably Yes)
A clear answer to the WordPress-era question — when tag archives help SEO, when they hurt, and the exact noindex pattern to ship.