TL;DR
- For 100-5,000 short comments, paste them into Claude (Sonnet 4.6, 1M-token context as of June 2026) or ChatGPT Plus and ask for 5-10 named themes with counts, severity, and verbatim quotes. A single pass takes about 30 minutes and gets you ~80% of a clean theme tree.
- One paid seat ($20/mo Claude Pro or ChatGPT Plus) covers this. You only need a dedicated platform (Dovetail from ~$30/user/mo, Thematic enterprise from a reported ~$25K/yr) when feedback streams in continuously and several people query it.
- The output is only trustworthy if you verify it: spot-check 20 comments, hand-count one theme’s frequency, and merge any two themes that share more than 30% of their quotes.
- Skip pure-AI clustering for board-deck statistics, regulated content (medical, financial), or feedback that is mostly star ratings with no text. The model will hallucinate themes from noise.
The task
You have 300-2,000 raw comments (App Store reviews, NPS responses, Intercom tickets, survey replies) and a deadline. By Friday the product team needs themes: what users actually complain about, what they love, and which clusters justify a roadmap change.
Reading every comment by hand is slow and biased toward the loudest reviews. The real job is synthesis: dedupe similar feedback, name themes, count them, attach severity, and produce a table a PM can paste straight into a roadmap doc.
When AI is the right tool
Use AI when your corpus is roughly 100 to 5,000 short comments and you need directional themes, not statistical significance. Modern models are strong at semantic grouping, naming, and pulling representative quotes.
The first pass is where it earns its keep. Thirty minutes with a chat model usually produces a theme tree a human can polish in another hour, versus a half-day of manual tagging.
When not to rely on AI alone
- Board-deck statistics. If a number is going to drive a budget decision, count it in a spreadsheet, not in a chat window. Models approximate frequencies and routinely miscount.
- Regulated content. Medical or financial-advice feedback needs a human in the loop for compliance.
- Mostly-empty feedback. If comments are just star ratings or emoji, the model invents themes from noise. Filter to text-bearing rows first.
- Low-resource languages. Test a small sample before trusting whole-batch results in a language the model handles weakly.
And never let AI decide what ships. It has no business context, no revenue weighting, and no view of your engineering cost.
What to feed the model
- Raw feedback as one comment per line, with exact duplicates removed
- Optional metadata: star rating, plan tier, date, app version
- One sentence of product context (what the app does)
- The decision you need to make (prioritize bugs vs. growth features, for example)
Strip personally identifiable information first. Anonymize names, emails, and order IDs before pasting. If you cannot remove PII, use an enterprise or team plan where your data is excluded from training rather than a free consumer account.
How much fits in one pass
As of June 2026 the practical ceiling is data, not the model’s raw context window:
| Tool | Comfortable per pass | Note |
|---|---|---|
| Claude Pro ($20/mo) | 500-1,500 short comments | Sonnet 4.6 ships a 1M-token context; quality holds well to ~1,500 short comments |
| ChatGPT Plus ($20/mo) | 500-1,200 short comments | In-app context is ~320 pages on Plus; the full 1M window is $200 Pro |
| File upload (ChatGPT) | Larger batches as a CSV | Up to 20 files per message, 512MB / 2M tokens each (as of Feb 2026), but it then samples rather than reads every row |
Beyond ~1,500 comments, split by date or product area, cluster each slice, then merge the resulting theme lists in a final pass.
The clustering prompt
This is the workhorse. Keep the placeholders in square brackets and swap in your own text. Paste your feedback where shown.
You are a senior product researcher. Cluster the feedback below into 5-10 themes a PM can act on this sprint.
Product context: [one sentence about the product]
Decision needed: [what we will decide from this]
For each theme output:
- Theme name (action-oriented, max 6 words)
- Plain-language description (1-2 sentences)
- 3 representative verbatim quotes, copied exactly
- Frequency: approximate count and percentage of total
- Severity: blocker / pain / nice-to-have, with a one-line justification
- Suggested next step: research, fix, or ignore
Rules:
- Do not invent quotes. Only quote text that appears in the feedback.
- If a theme has fewer than 3 supporting comments, label it "watch" instead of a full theme.
- Flag any comment you could not confidently place.
End with a 3-bullet "what surprised me" section.
Feedback:
[paste one comment per line]The “do not invent quotes” and “flag what you could not place” lines matter. Without them, models will paraphrase quotes (so stakeholders cannot gut-check) and silently drop ambiguous comments.
Output structure that PMs actually use
Ask for a short executive summary (3 bullets), then the theme table, then the surprises. The table is the work product, so keep it scannable. Map severity to your existing triage labels (P0/P1/P2, or blocker/major/minor) so an engineer can pick it up without translation.
How to check the output
The verification step is what separates a usable analysis from a confident-sounding guess.
- Spot-check placement. Pull 20 random comments and confirm the model filed each in the right theme. More than a couple of misfiles means the theme definitions are too fuzzy; re-prompt with tighter names.
- Hand-count one frequency. Pick the theme that will drive a decision and count it yourself with a spreadsheet filter or
grep -ci "keyword" feedback.txt. Models approximate counts, so treat their percentages as estimates until you confirm one. - Merge overlaps. If two themes share more than 30% of their quotes, they are one theme. Merge and re-run.
- Demote thin clusters. A theme resting on 1-2 quotes is noise. Move it to a “watch” list rather than a roadmap line.
When to graduate to embeddings or a platform
The chat-window method has a ceiling. If you are clustering the same feedback stream every week, or you need reproducible clusters across thousands of items, move to one of these:
- Embedding-based clustering (a small script): embed each comment with OpenAI
text-embedding-3-small(about $0.02 per 1M tokens as of June 2026) ortext-embedding-3-large($0.13 per 1M tokens), run k-means or HDBSCAN, then ask a chat model to name each cluster. Gemini Embedding 001 runs about $0.15 per 1M tokens. This makes frequencies exact and clusters stable run to run. - A dedicated platform when several people need to query feedback continuously. Dovetail starts around $30/user/mo (its automated-ingestion Channels add-on is a separate ~$50/mo line item), and Thematic is enterprise-priced with a reported floor near $25,000/year. These earn their cost when feedback is a daily input, not a quarterly project.
For a one-off “themes by Friday” task, the $20 chat plan wins on speed and cost. For a standing feedback loop, the platform pays for itself in saved analyst time.
Common mistakes
- Over-clustering into 20+ tiny themes no one can act on
- Dropping the severity dimension, so everything looks equally urgent
- Trusting the model’s frequency numbers without hand-counting at least one
- Mixing languages without telling the model which themes belong to which language
- Losing the original verbatims, so stakeholders cannot gut-check a cluster
Keep the loop closed
Save the prompt, the input, and the output together in one folder. Re-run the same prompt each month so themes are comparable over time. When a top theme reaches the roadmap, tag the originating feedback IDs so you can notify those users when the fix ships. Closing that loop is what turns a clustering exercise into retention.
FAQ
How many comments is too many for one pass? Most chat models handle 500-1,500 short comments well as of June 2026. Past that, split by date or product area and merge the theme lists. For tens of thousands of items, switch to embedding-based clustering.
Should I deduplicate first? Yes, remove exact duplicates because they skew frequency counts. Near-duplicates (paraphrases of the same complaint) are fine to leave in; the model groups them correctly and they reflect real volume.
Why do my themes come out generic? Almost always because the product context and decision statement are vague. “Cluster these reviews” yields filler like “user experience.” Add a sharp decision (“should we fix onboarding bugs or build the export feature this quarter?”) and themes snap into focus.
Is it safe to paste customer feedback into ChatGPT or Claude? Strip PII first. On consumer free tiers your inputs may be used to improve models, so use a team or enterprise plan (which excludes business data from training) for anything sensitive, and anonymize names, emails, and IDs regardless.
Will the model count frequencies accurately? No, treat its counts as estimates. Models approximate when tallying. Hand-count any theme whose number will drive a real decision.
Related
For prompt variations and edge cases, see user feedback clustering prompts. When you need fresh signal to cluster, run customer discovery questions with AI.
Tags: #Data analysis #Workflow #Research