AI AI Tools Guidebook
Indie Dev & Website Building

When Google Won't Crawl Your New Pages — a Real Checklist

A real diagnostic checklist for when you publish content and Google ignores it. The order matters — start at the top.

Published: Author: AI Productivity Guide Team 🌐 查看中文版本

You published a new article three weeks ago. It is not in Google. You typed site:yourdomain.com/the-slug/ and got nothing. Search Console shows the URL as either unknown or “Discovered — not indexed”. Here is the actual diagnostic order, from “5-minute fixes” down to “this might be a months-long problem”.

Background

When Google does not crawl new pages, the cause is almost always one of: blocked by configuration (5% of cases — fast fix), missing from sitemap or internal links (25% — medium fix), or low site authority / crawl priority (70% — slow, content-driven fix). Most indie devs spend hours on the first two and miss the third.

How to tell

  • New pages published weeks ago do not appear in Google.
  • Search Console says “URL is unknown to Google” or “Discovered — currently not indexed”.
  • Old pages on the same site are indexed fine.
  • You see normal crawl activity in the Crawl Stats report — Google is crawling you, just not these URLs.

Quick verdict

Walk through the checklist in order. Stop at the first issue you find — fixing one thing then waiting is faster than changing everything at once and not knowing what worked.

Step by step

  1. Run URL Inspection on the missing URL. The Crawl allowed? and Indexing allowed? rows in the live test answer 90% of cases instantly. If either is No, you have a config bug — fix and move on.
  2. Verify the URL is in your sitemap. Open /sitemap.xml, search for the slug. Not there? Fix your generator. Then go to Sitemaps in Search Console and resubmit.
  3. Verify the URL is internally linked. Grep your codebase for the slug — count occurrences. If zero or one, link from your homepage and 2 related articles. Google rarely indexes orphan pages.
  4. Check rendering. View the page source (not the rendered DOM) — does the body text actually appear in the HTML? Single-page apps where content only appears after JS often have crawl issues. If your framework supports SSR or static export, use it.
  5. Confirm canonical. View source, search for <link rel="canonical". The value must match the URL you want indexed. Many “page won’t index” cases are actually “canonical points elsewhere and Google indexed the other URL”.
  6. Check the Crawl Stats report (Settings -> Crawl stats). Are crawl requests trending up or flat? A flat or falling curve is a sign your site is being deprioritized — usually content quality.
  7. If steps 1-6 all pass and you have waited 30 days, the issue is site authority. Solutions: more inbound links, more internal links to the affected URL, and overall content quality lift. There is no faster fix.

Common pitfalls

  • Skipping the live test in URL Inspection. It catches the most common config bugs in 10 seconds and people skip it because it feels too basic.
  • Submitting sitemaps over and over. Sitemap influences discovery, not crawl priority. After the first submission, more sitemap submissions do nothing.
  • Adding priority and changefreq to sitemap entries. Google has publicly stated they ignore both. Do not waste time setting them.
  • Building thin programmatic pages and expecting them to be crawled. Google has gotten very good at recognizing pages-for-the-sake-of-pages.
  • Trying to “trick” Google with redirect chains or hidden links. It does not work and risks the whole site getting penalized.

Who this is for

Anyone with > 30 published pages who has a specific set of new URLs not getting crawled.

When to skip this

A brand-new site with no pages yet indexed at all — that is a different problem (new site submission guide).

FAQ

  • Could it be a robots.txt issue I missed?: Yes — that is exactly what step 1 (URL Inspection live test) catches. The live test explicitly reports whether the URL is blocked. Always run it before assuming anything else.
  • My site uses Next.js / Astro / Hugo — should I worry about JS rendering?: Astro and Hugo are static — no issue. Next.js with SSG / SSR is fine. Pure client-side React (CRA, Vite SPA) with no SSR is risky for SEO; if that is you, fix that first.
  • How long should I wait before declaring a page “stuck”?: After step 6, wait 30 days. Then accept it might be site-authority and pivot to publishing more linked / better content rather than fighting one URL.
  • Does requesting indexing 5 times help?: No. The system de-duplicates. Submit once, wait, look at the result. Repeat requests waste your daily quota.

Tags: #Indie dev #SEO #Google #Search Console #Indexing #Troubleshooting #Content ops

Related Articles