TL;DR
Paste your cleaned open-ends into a long-context model with the prompt below, force it to return a theme table with verbatim quotes and response_id citations, then manually verify 10-15 assignments before you trust any count. As of June 2026, Claude (Opus 4.7 / Sonnet 4.6, 1M-token context) and Gemini 3.1 Pro (1M) can hold a few thousand short answers in a single pass; ChatGPT Plus tops out around 320 pages in-app. The hard part is not clustering, it is proving the clusters are real and stable across re-runs.
The task
You ran a survey and got hundreds, sometimes thousands, of free-text answers to questions like “what should we improve?” or “describe a recent frustration.” Counting by hand is impossible, but a single chart of themes will steer the next quarter of product or marketing decisions. You need clusters that survive scrutiny from a skeptical stakeholder.
When AI is the right tool
- You have 200-5,000 open-ends and need structure within a day.
- Responses are short (5-100 words) and largely in one or two languages.
- You want a first-pass clustering that a human will refine, not a final verdict.
- You re-run the same analysis after every wave and need consistent labels.
A practical scale check: short open-ends run roughly 7-130 tokens each, so 2,000 responses land near 60,000-120,000 tokens. That fits comfortably inside the 1M-token context of Claude Opus 4.7, Sonnet 4.6, or Gemini 3.1 Pro in one pass. Past ~3,000 responses, batch in chunks of 500-800 and merge codebooks rather than risking a “lost-in-the-middle” miss on a single giant paste.
When not to rely on AI alone
- Very small samples (under 50). Read them yourself; you will be faster and more accurate.
- Sensitive topics (mental health, harassment, layoffs) where a misread does real harm.
- Strategic decisions you must defend to a board or regulator, where every theme needs an audit trail.
- Multilingual datasets where the model’s coverage is uneven across languages.
Pick the right tool
For a one-off analysis, a chat model with a long context is the cheapest path. For a recurring program with dashboards and trend tracking, a purpose-built platform earns its keep. As of June 2026:
| Tool | Best for | Notable point |
|---|---|---|
| Claude (Pro $20/mo) | One-off deep clustering, long datasets | Opus 4.7 / Sonnet 4.6 at 1M-token context; strong at verbatim citation discipline |
| ChatGPT (Plus $20/mo) | Quick analysis, mixed teams | GPT-5.5 default; in-app context ~320 pages, full 1M only on $200 Pro |
| Gemini 3.1 Pro (Google AI Pro $19.99/mo) | Sheets-native data, 1M context | Lives next to your Google Sheets export |
| Thematic / Kapiche | Recurring CX feedback programs | NLP-driven theme discovery, trend dashboards, no prompt engineering |
| Dovetail / ATLAS.ti | Mixed survey + interview repositories | Auditable coding trail for academic or regulated work |
If this is your first or only survey wave, start with Claude or ChatGPT and the prompt below. Move to a dedicated platform only when you are re-running this monthly.
What to feed the AI
- The cleaned responses, deduped, with empty and spam entries removed.
- A
response_idon every row so every quote stays traceable. - The survey question text. Clustering is question-dependent.
- The number of themes you expect (5-12 is typical).
- A stop-list of generic themes to forbid (“other feedback”, “general comments”).
- The respondent segment (job role, region, plan tier) if you want segmented clusters.
Copy-ready prompt
Replace each [bracketed] placeholder with your own value. Use response_id labels (R001, R002, …) so the model can cite each quote.
You are a research analyst clustering free-text survey responses.
Survey question: [survey_question]
Number of responses: [n]
Expected number of themes: [expected_themes]
Stop-list (themes to never use): [stop_list]
Segment metadata (if any): [segments]
Responses:
"""
[responses, one per line, each prefixed with its response_id]
"""
Output:
1. Theme table:
- Theme name (3-6 words, specific)
- 1-sentence definition
- 3-5 representative verbatim quotes, each with its response_id
- Count and percentage of total
2. Segment breakdown (if segments provided): theme frequency per segment.
3. "Long tail" section: 5-10 responses that fit no theme, with your notes.
4. Confidence flags: mark any theme with fewer than 5 supporting quotes as [weak].
Rules:
- Quotes must appear verbatim in the source. Never paraphrase a quote.
- Cite the response_id for every quote.
- Sum of theme counts must equal total responses minus the long tail.
- Do not invent themes that have fewer than 3 supporting quotes.
- Themes must be mutually exclusive: assign each response to exactly one theme.For reproducibility, set the model to its lowest-variability mode. In the API, use a low temperature (0-0.2). In ChatGPT, use a Thinking model; in Claude, Opus 4.7. Higher creativity settings increase run-to-run drift, which is the enemy here.
How to check the output
This is the step most people skip, and it is the only thing standing between you and a confidently wrong slide.
- Manually verify 10-15 random quote-to-theme assignments against the source rows.
- Confirm theme counts plus the long tail equal the total. If they do not, the model dropped or double-counted responses.
- Re-run the same prompt and compare. Peer-reviewed studies of LLM thematic coding (arXiv 2506.14634) report that categories and individual assignments shift between runs, so significant drift is a signal the clusters are weak, not just noise.
- Hand the theme labels to a colleague who has not seen the data. If they cannot predict what kind of comment fits each theme, the label is too vague.
Where stakes are high, treat the model as one coder and have a human code an independent sample, then compare agreement (Cohen’s Kappa). Published work finds well-prompted LLMs can reach human-crowdworker agreement levels, but only on clear, well-defined codebooks.
Common mistakes
- Forcing the model toward your preferred themes instead of letting clusters emerge.
- Trusting cluster counts without verifying a single assignment.
- Reporting percentages from a self-selected survey as if they were representative.
- Running once and shipping. One run is a draft, not a finding.
- Ignoring the long tail, which is sometimes where the next product idea lives.
Keep the analysis comparable across waves
After each wave, save the theme list as a “codebook.” Feed it as predefined themes for the next survey so trend lines stay comparable across quarters, then allow one or two new themes to emerge. Track theme volume over time. A rising theme deserves attention before it ever becomes a majority.
FAQ
- What survey size justifies AI analysis? 200+ open-ends. Below that, manual reading is faster and more accurate.
- Which model handles the most responses at once? As of June 2026, Claude Opus 4.7, Sonnet 4.6, and Gemini 3.1 Pro all hold 1M tokens, roughly a few thousand short answers in one pass. ChatGPT Plus is limited to about 320 pages in-app; the full 1M context needs the $200 Pro tier.
- Should I anonymize responses first? Yes. Strip names, employer details, and anything personally identifying before you paste.
- Why do I get different themes each time I run it? LLM clustering is non-deterministic. Lower the temperature, use a Thinking or Opus model, and re-run twice. Stable themes survive; unstable ones do not.
- How do I report findings honestly? Lead with sample size, response rate, and segment caveats, every time.
- Can I combine survey themes with interview themes? Yes, but build separate codebooks first, then reconcile them.
Related
Pair this with user feedback clustering prompts, the user feedback clustering AI workflow, and survey result interpretation AI.
Tags: #Data analysis #Workflow #Research