Submit a New Website to Google: robots.txt + Manual Indexing Playbook

The full 2026 submission playbook: Search Console verification, sitemap, robots.txt Sitemap directive, the ~10/day Request Indexing quota, IndexNow for Bing, and realistic indexing timelines.

The biggest misconception with a new site is “I launched it, Google will find me.” It won’t, not on any schedule you’d like. You have to tell Google you exist, then give its crawler a clean path through your pages.

This is the comprehensive playbook: Search Console verification, sitemap, robots.txt, manual indexing requests, and the link signals that actually move the needle. If you only want the minimal four-step order without the deep robots.txt section, see Submit a new site to Google — 2026 four-step flow.

TL;DR

  1. Verify the site in Google Search Console (DNS TXT record is the most durable method).
  2. Submit your sitemap.xml (or sitemap-index.xml) in the Sitemaps report.
  3. Add a Sitemap: line to robots.txt and make sure nothing important is blocked.
  4. Use URL Inspection → Request Indexing for your 5–10 most important pages only. The quota is roughly 10 URLs per property per day, and re-submitting the same URL does nothing.
  5. Repeat in Bing Webmaster Tools and turn on IndexNow. Then wait: 24–72 hours to first crawl, 1–3 months for the bulk of pages.

Step 1: make sure the site is actually crawlable

Before you ask Google to look, remove the reasons it would give up. Check three things:

  1. Visit https://yourdomain.com/robots.txt and confirm it does not contain Disallow: / (that single line blocks your whole site).
  2. Open a few article URLs and view source — each should have a real <title> and a <meta name="description">, and the canonical tag should point to the URL you actually want indexed.
  3. In DevTools → Network, tick Disable JavaScript, then reload. If the main content disappears, Googlebot will struggle too. Server-rendered or statically generated HTML (Astro, Next.js SSG, Hugo) sidesteps this entirely.

One subtle trap: Search Console can flag a page Discovered – currently not indexed purely because the page is thin or duplicative, not because of a technical block. Fix crawlability first, then content quality.

Step 2: set up Google Search Console

Go to Search Console and add your property. You have two property types:

  • Domain property (recommended): covers every subdomain and both http/https. Verified once via a DNS TXT record.
  • URL-prefix property: covers only the exact prefix you enter (e.g. https://www.example.com/).

Verification methods, in order of durability:

MethodWhere it livesSurvives a redeploy?
DNS TXT recordYour registrar/DNS hostYes — most durable
HTML file uploadSite root /google….htmlOnly if you keep the file in the build
HTML meta tag<head> of homepageYes, if the tag stays in the template
Google Analytics / Tag ManagerExisting GA4/GTM snippetYes, while the snippet stays

Pick the DNS TXT record if you control the domain. File and meta-tag methods silently break the day you change frameworks or hosts.

Step 3: submit your sitemap

Most modern frameworks generate one of these automatically:

https://yourdomain.com/sitemap-index.xml
https://yourdomain.com/sitemap.xml

In Search Console → Sitemaps, paste that URL and submit. Google processes it within hours to a few days.

A sitemap is the highest-leverage step for a new site because it is the only realistic way to get all your pages into the queue. The manual Request Indexing button (Step 5) tops out at roughly 10 URLs a day; a sitemap has no such cap. Per Google’s own guidance, for large numbers of new or updated pages you mark each entry with an accurate <lastmod> and let the sitemap do the work.

Keep the sitemap clean: only 200-status canonical URLs you actually want indexed. A noindex page or a redirect in your sitemap sends Google a contradictory signal and wastes crawl budget.

Step 4: declare the sitemap in robots.txt

The Sitemap: directive in robots.txt is the standard, authoritative way to tell every search engine where your sitemap lives — independent of Search Console.

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap-index.xml

Replace the domain with yours, and put the full absolute URL on the Sitemap: line. The one rule that catches people: robots.txt and the sitemap must agree. If you block a path with Disallow: but still list those URLs in the sitemap, Google sees the conflict, can’t index them, and you’ve burned crawl budget on “phantom coverage.”

Step 5: manually request indexing for your key pages

In Search Console’s top URL bar, paste a single URL. Google runs a live check, then shows Request Indexing. Click it for your most important pages: homepage, main category/hub pages, and your two or three flagship articles.

Three things to know before you start clicking:

  • The quota is small. It is not officially published, but in practice it’s about 10 URLs per property per day. Spend it on pages that matter.
  • Re-submitting does nothing. Google logs the first request and ignores repeats for the same URL within a crawl cycle. Asking again does not jump the queue.
  • It’s a request, not a guarantee. Google’s docs are explicit: requesting a crawl “does not guarantee that inclusion in search results will happen instantly or even at all.”

Skip the Google Indexing API unless you publish job postings or live-stream events. As of June 2026 it officially supports only JobPosting and BroadcastEvent (in a VideoObject) structured data. Google ignores — and may deprioritize — general blog or product pages pushed through it, so it is the wrong tool for a normal content site.

Step 6: do the same for Bing (and turn on IndexNow)

Bing still holds a meaningful slice of search (roughly 3–7% of global desktop+mobile share as of 2026, higher in some regions), and ChatGPT Search and Microsoft Copilot lean partly on Bing’s index. Bing Webmaster Tools is free — add your site and import the property straight from Search Console in one click.

While you’re there, enable IndexNow. It is a free protocol that pings Bing, Yandex, and others the moment a URL changes, so you don’t wait for a crawl. Google does not support IndexNow as of June 2026, so it complements (never replaces) your Search Console workflow. Many static hosts and CMSs have a one-line IndexNow plugin or postbuild hook.

  • Internal links: every article should link to a few related ones. A page with zero internal links (an “orphan”) is hard for Google to discover and tends to sit unindexed. Make sure your homepage and hub pages link down to deep content.
  • External links: pull at least one link from a channel you already own — social bio, personal site, a public Notion or GitHub README.

External links aren’t mandatory, but even one real inbound link in the first week noticeably speeds up first crawl, because it gives Googlebot a second, independent path to your URLs.

Step 8: be patient, and read the report

Realistic timeline for a brand-new domain:

WindowWhat normally happens
24–72 hoursGooglebot starts crawling; sitemap shows “Success”
1–2 weeksA handful of pages move to “Indexed”
1–3 monthsCrawl pattern stabilizes; most quality pages get indexed

If after a week nothing is indexed, open Search Console → Page indexing and read the reasons. The usual culprits are Blocked by robots.txt, Excluded by 'noindex' tag, Page with redirect, Duplicate without user-selected canonical, and Discovered – currently not indexed (often a content-quality signal). Each one points at a specific fix.

FAQ

How long until a new site shows up on Google? Expect 24–72 hours for the first crawl and 1–3 months for the bulk of pages to be indexed, assuming the site is crawlable and the content is original. A single page you Request-Index can appear within a day or two, but most do not — Google indexes on its own schedule.

How many URLs can I submit with Request Indexing per day? Roughly 10 per property per day. Google doesn’t publish the exact figure and it can flex with site history. For anything larger, rely on your sitemap with accurate <lastmod> values rather than clicking through URLs.

Does re-submitting the same URL get it indexed faster? No. Google records the first request and ignores duplicates within the same crawl cycle. Re-clicking wastes your daily quota with zero benefit.

Do I need the Google Indexing API? Only if you publish job postings or live broadcast events. For a normal blog, docs site, or store, the Indexing API is the wrong tool — Google supports it solely for JobPosting and BroadcastEvent content. Use sitemaps and Request Indexing instead.

Should I bother with Bing if I’m focused on Google? Yes. Bing Webmaster Tools is free, takes five minutes to import from Search Console, and feeds ChatGPT Search and Copilot. Enabling IndexNow there gets your new URLs in front of Bing almost instantly.

My sitemap was submitted but nothing is indexed — what now? Check that the sitemap returns 200, lists only canonical indexable URLs, and isn’t contradicted by robots.txt or noindex. Then read Page indexing for the specific exclusion reason. See Sitemap submitted but no indexing.

Summary

Being discoverable is not passive. The minimum kit is Search Console verification + a clean sitemap + a matching robots.txt + a handful of manual Request Indexing clicks, then Bing/IndexNow and a few real links. Set those up correctly once and the rest is mostly waiting and reading the Page indexing report.

Tags: #SEO #Google #Search Console #Indexing