AI AI Tools Guidebook
Indie Dev & Website Building

Find Content Gaps With AI: A Repeatable Workflow

A June 2026 workflow for using AI to spot the articles your site should have but doesn't, built from your own sitemap, Search Console data, and a topic pillar map.

Published: Updated: Author: AI Productivity Guide Team 🌐 查看中文版本

Content gaps are where a site stalls. You know there is more to write about, but you cannot see what is missing. A long-context model is good at this, but only if you feed it the right map: your own structure plus the queries you already half-rank for. Competitor scraping alone gives you someone else’s plan, not your own holes.

TL;DR

  • Export your articles (URL + title + pillar) and your Search Console queries into two CSVs, then paste both into a 1M-token model (Claude Opus 4.7 / Sonnet 4.6, Gemini 3.1 Pro, or GPT-5.5) for a structured gap analysis.
  • The signal that matters: queries with impressions > 100 and position > 10. You have reach but no ranking page, which is the clearest coverage gap.
  • The Search Console UI export caps at 1,000 rows. For more, use the Search Analytics API (25,000 rows/request, 50,000/day) or the free BigQuery Bulk Data Export.
  • Filter every AI suggestion against your own edge and intent, queue 8 to 15 topics at a time, and rerun the analysis quarterly because the query data shifts.

Why your own data beats competitor scraping

Most “find content gaps” advice tells you to scrape competitors. That works once and then you are chasing their roadmap. The durable approach is to feed AI your own pillar/cluster structure plus the queries you already partly rank for, then ask it where the holes are. This works because a model is genuinely good at completing a structured pattern when you give it enough of one. Competitor data is still useful, but as validation after the fact, not as the seed.

When this workflow is worth running

  • You publish regularly but feel like you are running out of obvious topics.
  • You can describe your site’s pillar topics in three to five sentences.
  • You have Search Console access and at least three months of query data.
  • You have not done a formal gap analysis in the past 60 days.

Step by step

  1. Export sitemap to a flat CSV. Run in your terminal:

    # Pull sitemap, extract URL + last-modified
curl -s https://yoursite.com/sitemap.xml \
  | grep -E "<(loc|lastmod)>" \
  | sed -E 's/<\/?(loc|lastmod)>//g; s/^[[:space:]]+//' \
  | paste - - \
  > sitemap_urls.tsv

# Pull <title> for each URL via Python (~5 minutes)
python3 -c "
import csv, requests
from bs4 import BeautifulSoup
with open('sitemap_urls.tsv') as f, open('articles.csv','w') as out:
    w = csv.writer(out)
    w.writerow(['url','slug','title','lastmod'])
    for line in f:
        url, lastmod = line.strip().split('\t')
        html = requests.get(url, timeout=10).text
        title = BeautifulSoup(html,'html.parser').title.string.strip()
        slug = url.rstrip('/').split('/')[-1]
        w.writerow([url, slug, title, lastmod])
"

    Output articles.csv has 4 columns: url, slug, title, lastmod.

  2. Tag pillar / cluster. Add pillar and cluster columns in Sheets / Excel, fill manually (500 articles < 1 hour). If your URL already encodes category:

    # If URL is /en/articles/indie-dev/foo/
awk -F'/' 'NR>1\{print $0","$5\}' articles.csv > articles_with_pillar.csv

  3. Pull your top GSC queries. Search Console -> Performance -> Queries -> Export -> “Download CSV”. The UI export is capped at 1,000 rows as of June 2026, which is plenty for a single-site gap pass. If you need more (or want to script this), the Search Analytics API returns up to 25,000 rows per request and 50,000 per day per site per search type via startRow pagination, and the free BigQuery Bulk Data Export has no row limit at all. The lightweight Python wrapper (pip install searchconsole) is enough here:

    import searchconsole
account = searchconsole.authenticate(client_config='client_secret.json')
webproperty = account['https://yoursite.com/']
report = webproperty.query.range('today', days=-90).dimension('query').limit(1000).get()
report.to_dataframe().to_csv('gsc_top_queries.csv', index=False)

    The CSV contains query, clicks, impressions, ctr, position. Focus on rows where impressions > 100 AND position > 10. That combination means reach without a ranking page, which is the clearest coverage-gap signal you have. Google’s own performance-data deep dive explains how the row limits and filtering work if you need to go beyond the UI.

  4. Feed it to AI for gap analysis. Open a long-context model and paste all three files plus the prompt below. Any of the current 1M-token models work: Claude Opus 4.7 or Sonnet 4.6, Gemini 3.1 Pro, or GPT-5.5. One caveat as of June 2026: ChatGPT Plus only exposes about 320 pages of context in-app (the full 1M window is on the $200 Pro tier), so if your two CSVs are large, split them or use Claude/Gemini, which carry the full 1M at their $20 and $19.99 tiers.

    Attachment 1: articles_with_pillar.csv (all current articles + pillar)
Attachment 2: gsc_top_queries.csv (last 90 days of high-impression queries)

Run a gap analysis:

1. Per pillar, list the 5 queries in GSC Top 200 that have impressions but no current article directly answers them
   - "Directly answers" = article title or URL slug contains the query's core terms
   - Output: | pillar | query | current impressions | current position | existing coverage (none/partial/full) |

2. Per pillar, list 5 sub-topics NOT in GSC but that should be there
   - Reason: a competitor ranks for it, OR your existing article mentions it but doesn't have a dedicated page
   - Output: | pillar | missing sub-topic | why it should exist | 1 seed title |

3. List 5 cross-pillar bridge topics — content that connects 2 pillars
   - Example: indie-dev pillar + ai-tools pillar → "Use AI to audit App Store screenshots"
   - These typically have the strongest differentiation

Do NOT hallucinate articles / queries — only use what I gave you.

  5. Filter with domain knowledge. For every AI suggestion, ask:

    - Is this actually searched (not AI-fabricated semantics)?
- Can I write this better / more concretely than the top 3?
- Does it have commercial value (affiliate / conversion / brand)?
- Will 1 page do, or do I need a 3-5 article cluster?

    Any “no” → drop.

  6. Cross-cluster bridges, second pass:

    Based on the previous gap analysis, give me 8 cross-pillar bridge topics:

- Each topic must naturally belong to 2 pillars (not forced)
- For each:
  - Title (verb-led or number-led)
  - Core question (≤10 words)
  - Primary audience (pillar A folks + pillar B folks)
  - Internal-link opportunities (which existing X articles can it link to)

Bridge content captures the "cares about both" audience — competitors rarely sit on both sides.

  7. Write an “angle card” per gap topic:

    For this gap topic: <title>

1. Search the current Google top 10 — note their titles, opening lines, structure
2. List 3 common blind spots (all dodge X / all answer Y shallowly / all are stale)
3. Give me 1 "counter-conventional + first-person + number" opening angle
4. List 5 pieces of specific evidence I must collect before publishing (numbers / screenshots / tool versions)

  8. Write back to a content queue: content_pipeline.csv:

    slug,title,pillar,cluster,target_keyword,intent,angle_note,evidence_needed,priority
ai-app-store-screenshot-review,Use AI to audit App Store screenshots,indie-dev,app-store-launch,app store screenshot review ai,how-to,counter-conventional + measured,3 before/after sets,P1
...

    Set priority to P1 (high-impression GSC query plus your strongest angle), P2 (bridge topics), or P3 (exploratory). Pull three P1 items per week to actually write.

Tools and limits at a glance (June 2026)

ToolFree tierPaidWhat it does here
Search Console UI exportFreeFreeTop queries CSV, capped at 1,000 rows
Search Analytics APIFreeFree25,000 rows/request, 50,000/day per site per search type
BigQuery Bulk Data ExportFree to set up (Blaze billing for storage)Pay-as-you-goNo row limit; daily query dumps
Claude (Opus 4.7 / Sonnet 4.6)Limited Sonnet 4.6Pro $20Full 1M context for the gap prompt
Gemini (3.1 Pro)LimitedAI Pro $19.99Full 1M context for the gap prompt
GPT-5.5 (ChatGPT)Free, tight limitsPlus $20 / Pro $200In-app ~320 pages on Plus; full 1M on Pro
Screaming Frog SEO Spider500-URL crawl~$259/year (about £199)Optional: crawl your own site or a competitor

If your site is under 500 pages, the free Screaming Frog crawl is an alternative to the sitemap script in step 1 for pulling URLs and titles.

Common pitfalls

  • Letting the AI suggest topics without your data. It produces generic “things people search in your niche” lists that miss what your site is actually positioned for.
  • Skipping the manual pillar tagging. Without structure, the model cannot return structured gaps.
  • Chasing every gap. Most “gaps” are gaps because nobody searches for them. Cross-check intent before you commit.
  • Running it only once. Search Console data shifts every quarter, so the analysis should too.
  • Trusting AI search-volume estimates. Models invent traffic numbers freely, so pull real volume from GSC impressions or a keyword tool instead.

Who this is for

Established sites (50+ articles) with a real pillar structure and Search Console history. Brand-new sites with no query data should focus on writing the first 30 articles before optimizing gaps; there is nothing to analyze yet.

FAQ

  • Should I scrape competitors too?: Yes, as a supplement. Your own data is a stronger signal than competitor mimicry, so use a competitor crawl to validate a gap, not to seed the list.
  • How many gap topics should I queue at once?: Eight to 15 is a healthy backlog. More than that and you start picking weaker topics just to clear the list.
  • Can the AI prioritize the gaps for me?: It can offer an opinion, but final prioritization should reflect your edge and your effort cost, neither of which the model can see.
  • What if AI suggests gaps already on my list?: That is a good signal. The same gap surfacing twice means it is real, so move it up.
  • Which model should I use for the gap prompt?: Any current 1M-token model is fine. Claude Sonnet 4.6 and Gemini 3.1 Pro both carry the full window at the $20 tier; on ChatGPT Plus, watch the ~320-page in-app limit and split your CSVs if needed.
  • How often should I rerun this?: Quarterly. New impressions accumulate, your published set changes, and last quarter’s gaps may now be filled.

Tags: #Indie dev #AI-assisted build #SEO #Content ops #Workflow

Related Articles