AI AI Tools Guidebook
Indie Dev & Website Building

Submit a New Website to Google: robots.txt + Manual Indexing Playbook

The full submission playbook — Search Console, sitemap, robots.txt configuration, manual indexing requests, and internal-link signals. For the shorter 2026 four-step order, see the linked guide.

Published: Author: AI Productivity Guide Team 🌐 查看中文版本

The biggest misconception with a new site is “I launched it, Google will eventually find me.” It won’t. You have to tell Google, “Hey, I’m here.”

This page is the comprehensive playbook covering Search Console verification, sitemap, robots.txt, manual indexing, and internal-link strategy. If you only need the minimal 2026 four-step order without the deep robots.txt section, see Submit a new site to Google — 2026 four-step flow.

Here’s the fastest path to getting indexed, current for 2026.

Step 1: make sure the site is crawlable

Open the site and check three things:

  1. Visit https://yourdomain.com/robots.txt — make sure it doesn’t say Disallow: /.
  2. Open a few articles — each should have <title> and <meta name="description">.
  3. In DevTools → Network → “Disable JavaScript”, reload a page. Can you still see the content? If not, crawlers will struggle.

Step 2: set up Google Search Console

Go to Search Console and add your site.

Two recommended verification methods:

  • DNS record: add a TXT record at your registrar (preferred).
  • HTML file: upload the verification file to your site root.

Once verified, you have Google’s view of your site. Essential.

Step 3: submit your sitemap

Modern frameworks (Astro, Next.js, etc.) typically generate:

https://yourdomain.com/sitemap-index.xml

or:

https://yourdomain.com/sitemap.xml

In Search Console → Sitemaps, submit that URL.

This is the single biggest accelerator for a new-site indexing.

Step 4: reference the sitemap in robots.txt

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap-index.xml

Replace the domain with yours.

Step 5: manually request indexing for key pages

In Search Console’s top URL bar, paste a URL — you’ll see a “Request indexing” button.

Manually submit only the important ones: homepage, category pages, flagship articles. Don’t spam-submit — Google notices.

Step 6: do the same for Bing

Bing still holds roughly 5–10% of search share (region dependent), and ChatGPT/Copilot search partly leans on Bing’s index. Bing Webmaster Tools is free; submit there too.

  • Internal links: articles linking to related articles.
  • External links: pull at least one link from a channel you already own (social bio, personal site, public Notion).

External links aren’t mandatory, but even one in the first week noticeably speeds up indexing.

Step 8: be patient

  • 24–72 hours: Google begins crawling.
  • 1–2 weeks: a handful of pages indexed.
  • 1–3 months: structure stabilizes; most pages indexed.

If after a week nothing is indexed at all, check Search Console → Page indexing for blocked pages or redirect issues.

Summary

Being discoverable isn’t passive. Search Console + sitemap + robots + manual requests is the minimum kit.

Cost and risk check

  • Know which part is free, which part becomes paid after traffic, storage, function calls, build minutes, or team seats grow.
  • Keep secrets out of static bundles, screenshots, public repos, and client-side config.
  • Do not combine domain, SEO, ads, analytics, payment, and hosting changes in one uncontrolled release.
  • Write down the rollback path before changing DNS, canonical URLs, redirects, billing plans, or App Store settings.

What to document

Keep a small launch note for each meaningful change: date, reason, affected URLs, commands run, platform settings changed, DNS records, env vars touched, and the rollback method. Indie sites look simple until something breaks three months later. Documentation is how future-you avoids re-solving the same deployment, SEO, or monetization problem from zero.

Detailed implementation path

  1. Define the business reason for the change: launch faster, improve indexing, reduce hosting risk, prepare monetization, improve conversion, or support maintenance.
  2. Write the current state down before changing anything: domain, hosting target, build command, analytics, Search Console property, billing plan, and key URLs.
  3. Make the smallest version that proves the change works. For a site, that usually means homepage, one article page, sitemap, robots.txt, canonical, and 404.
  4. Test locally or in preview first. Production should be the confirmation step, not the first time you discover the behavior.
  5. Launch during a window where you can watch logs, analytics, Search Console, and user-facing pages for at least 30 minutes.
  6. Record the final settings and the rollback path in the project README, launch notes, or an internal checklist.

Launch template

Change:
- [what changed and why]

Affected URLs:
- /
- /articles/example/
- /sitemap.xml
- /robots.txt

Commands run:
- npm run build
- [deploy command]

Platform settings changed:
- Hosting:
- DNS:
- Search Console:
- Analytics:
- Ads / payments / app store:

Verification:
- Homepage:
- Deep page:
- Mobile:
- 404:
- Sitemap:
- Canonical:

Rollback:
- Previous release:
- Old DNS/config:
- Person responsible:

Common launch mistakes

  • Treating “deploy succeeded” as “site is ready”. A successful deploy can still have broken routes, wrong canonicals, stale sitemap, missing analytics, or blocked indexing.
  • Changing DNS and app code together. If traffic drops, you will not know which layer caused it.
  • Forgetting that static sites bake environment variables at build time.
  • Publishing AI-generated content without checking accuracy, intent match, internal links, and duplicate coverage.
  • Waiting until AdSense, Search Console, or App Store review to discover missing privacy, contact, policy, or ownership details.

Tags: #SEO #Google #Search Console #Indexing

Related Articles