You published an article three weeks ago. It is not in Google. You typed site:yourdomain.com/the-slug/ and got nothing. Search Console shows the URL as “URL is unknown to Google” or “Discovered - currently not indexed”. This is the diagnostic order I actually use, from 5-minute config fixes down to “this is a months-long content problem”. Run the steps top to bottom and stop at the first one that fails.
TL;DR
- Run URL Inspection first. The live test’s “Crawl allowed?” and “Indexing allowed?” rows resolve most config bugs in 10 seconds.
- Then check, in order: sitemap presence, internal links, server-rendered HTML, the canonical tag, and the Crawl Stats trend.
- “Discovered - currently not indexed” almost always means crawl priority, not a bug: Google knows the URL exists but has not judged it worth crawling yet.
- Requesting indexing more than once does nothing. The manual quota is roughly 10-12 URLs per day per property (as of June 2026).
- If steps 1-6 pass and you have waited 30 days, the fix is authority and content quality, not a setting. There is no faster path.
What causes it
When Google will not crawl new pages, the cause falls into three buckets:
| Cause | Share of cases | Fix speed | Where you find it |
|---|---|---|---|
| Blocked by config (robots, noindex, bad canonical) | ~5% | Minutes | URL Inspection live test |
| Missing from sitemap or no internal links | ~25% | Hours | Sitemap file + codebase grep |
| Low crawl priority / site authority | ~70% | Weeks to months | Crawl Stats trend + page quality |
Most indie devs spend hours on the first two buckets and never confront the third. The split matters because the fix is completely different: a config bug is a one-line change, while low crawl priority is a content and linking problem that no button in Search Console can shortcut.
It also helps to know the two statuses apart. Discovered - currently not indexed means Google found the URL (usually via your sitemap) but has not crawled it yet, which is a crawl-budget signal. Crawled - currently not indexed is more serious: Google fetched the page, read it, and chose not to index it, which points at content quality or duplication. This checklist covers the first case; for the second, see the linked guide at the end.
How to confirm it is this problem
- New pages published weeks ago do not appear for
site:queries. - Search Console reports “URL is unknown to Google” or “Discovered - currently not indexed”.
- Older pages on the same site are indexed normally.
- The Crawl Stats report shows steady crawl activity overall, just not on these URLs.
If old pages are also dropping out, that is a different (sitewide) problem and this checklist will not fix it.
The checklist
Walk these in order. Fixing one thing and waiting is faster than changing five things at once and not knowing which one worked.
1. Run the URL Inspection live test
In Search Console, paste the URL into the top inspection bar, then click Test Live URL. Read two rows: Crawl allowed? and Indexing allowed?. If either says No, you have a config bug. The usual culprits:
- A
Disallowrule in/robots.txtmatching the path. - A
<meta name="robots" content="noindex">tag, or anX-Robots-Tag: noindexHTTP header. - A canonical that points away from this URL (covered in step 5).
Fix the bug, click Request Indexing once, and move on. Do not skip this step because it feels basic. It catches the most common failure in 10 seconds, and people routinely waste an afternoon before running it.
2. Confirm the URL is in your sitemap
Open /sitemap.xml in a browser and search for the slug. If it is not there, your sitemap generator is the bug. Keep the sitemap to only 200 OK, canonical, indexable URLs: a sitemap full of redirects, 404s, or noindex pages wastes crawl budget and dilutes the signal. Once fixed, go to Sitemaps in Search Console and resubmit.
Two attributes are worth getting right (as of June 2026):
| Attribute | Does Google use it? | What to do |
|---|---|---|
<lastmod> | Yes, if it is consistently and verifiably accurate | Set it to the real last-edit date; never bump it daily for unchanged pages |
<priority> | No, ignored | Remove it; it does nothing |
<changefreq> | No, ignored | Remove it; it does nothing |
A fake or daily-churning lastmod is worse than none. Google compares it against the page’s actual change history and stops trusting the value when it does not match.
3. Confirm the URL is internally linked
grep -r "the-slug" . your codebase or built site and count occurrences. If the result is 0 or 1, the page is effectively an orphan, and Google rarely crawls orphans. Add links from your homepage or a hub page plus two related articles. Internal links are the strongest crawl-priority lever you control directly, far more than resubmitting a sitemap.
4. Check that the HTML is server-rendered
View the page source (not the rendered DOM in DevTools). Does the body copy actually appear in the raw HTML? Googlebot renders JavaScript, but client-side-only content is crawled on a slower, lower-priority queue, which is the last thing a new page needs.
| Framework | Default rendering | SEO risk for new pages |
|---|---|---|
| Astro, Hugo, Eleventy | Static HTML | None |
| Next.js (SSG/SSR), Nuxt, SvelteKit | Server-rendered | None when SSG/SSR is used |
| Create React App, Vite SPA, plain CRA | Client-side only | High; body text is missing from source |
If your framework can do static export or SSR, use it for content pages.
5. Confirm the canonical tag
View source and find <link rel="canonical" ... >. The href must equal the exact URL you want indexed, including trailing slash and protocol. A large share of “page won’t index” reports are really “the canonical points elsewhere, so Google indexed the other URL and dropped this one”. Self-referencing canonicals are correct for standalone pages.
6. Read the Crawl Stats trend
Go to Settings -> Crawl stats. Look at total crawl requests over the last 90 days and the average response time. Two signals matter:
- A flat or falling crawl-request curve means Google is deprioritizing the site, which is almost always a content-quality signal.
- Average response time should sit near 100 ms. If it climbs toward 1,000 ms, Googlebot throttles crawling to protect your server, and new pages wait longer. Check Host status in the same report for availability errors.
Slow server response is one of the few crawl problems you can fix with engineering rather than content. Cache, use a CDN, and cut the slowest database queries.
7. If everything passes, it is authority
If steps 1-6 are clean and you have waited a full 30 days, the page is stuck on crawl priority. There is no setting that fixes this. The levers, in order of impact:
- More internal links to the affected URL from high-traffic pages.
- Inbound links from other sites.
- A real lift in overall content depth and uniqueness so Google raises the site’s crawl budget.
This is slow and there is no trick. Plan to keep publishing genuinely useful, well-linked pages rather than fighting one URL.
Common mistakes
- Skipping the URL Inspection live test. It is the single highest-yield 10 seconds in this whole process, and it is the step people most often skip.
- Resubmitting the sitemap repeatedly. A sitemap influences discovery, not crawl priority. After the first submission, more submissions change nothing.
- Setting
priorityandchangefreq. Google has publicly confirmed it ignores both. Time spent tuning them is wasted. - Requesting indexing five times. The system de-duplicates. You get roughly 10-12 manual requests per property per day; spamming the same URL burns the quota and earns a “Quota exceeded” message without crawling anything faster.
- Shipping thin programmatic pages and expecting them to be crawled. Google is very good at recognizing pages-for-the-sake-of-pages and parks them in “Discovered - currently not indexed”.
- Trying to trick Google with redirect chains or hidden links. It does not work and risks a sitewide penalty.
Who this is for
Anyone running a site with 30+ published pages where a specific set of new URLs is not getting crawled, while the rest of the site indexes fine.
When to skip this
A brand-new site with nothing indexed at all is a different problem; see the new site submission guide. If Google has crawled the page but declined to index it (“Crawled - currently not indexed”), the fix is content quality, not crawling; see Crawled - currently not indexed.
FAQ
- Could it be a robots.txt issue I missed?: Yes, and step 1 catches it. The URL Inspection live test explicitly reports whether the URL is blocked from crawling or indexing. Always run it before assuming anything else.
- My site uses Next.js / Astro / Hugo. Should I worry about JS rendering?: Astro and Hugo output static HTML, so no. Next.js with SSG or SSR is fine. Pure client-side React (CRA or a Vite SPA with no SSR) is risky because the body text is absent from the raw source; fix that first.
- How long should I wait before declaring a page stuck?: After step 6 passes, wait 30 days. Then treat it as a site-authority problem and pivot to publishing more well-linked, higher-quality content instead of fighting the one URL.
- Does requesting indexing five times help?: No. The manual quota is roughly 10-12 URLs per day per property (as of June 2026), and the system de-duplicates repeat requests. Submit once, wait, read the result.
- Should I use the Indexing API to force it?: Google officially supports the Indexing API only for
JobPostingandBroadcastEventpages. Using it for ordinary articles is against the terms and does not reliably help; the URL Inspection API (2,000 calls/day per property) is for inspection, not bulk submission.
Related
- Discovered - currently not indexed, explained
- Crawled - currently not indexed, how to fix
- URL Inspection: the practical guide
- How to submit a sitemap in Search Console
Tags: #Indie dev #SEO #Google #Search Console #Indexing #Troubleshooting #Content ops