TL;DR
Run the statistics yourself (in your experimentation platform), then hand the model the already-computed lift, 95% CI, and p-value plus the segment and guardrail breakdowns. The AI’s job is structure and plain-English translation, not math. Use the copy-ready prompt below to produce a one-pager with a forced ship/hold/kill decision, an explicit segment view, a novelty check, and a sample-ratio-mismatch line. A model like GPT-5.5 or Claude Sonnet 4.6 (both default-available in their respective free tiers as of June 2026) handles this in one pass once you feed it real numbers.
The task
Your A/B test concluded. The dashboard has the primary metric, two or three secondary metrics, a sample-size split, and segment breakdowns by device or persona. Friday at 3pm you need to drop a one-pager in Slack: was it a win, was it a loss, what did we learn, what ships, and what experiment comes next. The summary has to survive a skeptical reader, the kind who spots a missing CI or a hidden mobile loss in 30 seconds.
Where AI helps, and where it does not
AI is good at three things here:
- Structure. It enforces a consistent shape (headline → primary → secondary → segment → caveat → next) so nothing gets dropped under deadline.
- Translation. It turns stats jargon into one sentence: “the true lift is somewhere between 7% and 17% with 95% confidence.”
- The caveat checklist. It surfaces the standard risks you forget when you are excited about a win: novelty, segment heterogeneity, sample-ratio mismatch.
Where AI fails: it cannot compute statistics. Feed it raw conversion counts and ask for a p-value and it will return a plausible, wrong number. Compute the CI and p-value in your experimentation platform (Statsig, GrowthBook, Optimizely, VWO, or your in-house pipeline) and feed the model the results. Modern models are better at arithmetic than they were, but a confidence interval is not arithmetic you want a language model improvising under a Friday deadline.
What to feed the AI
- Test name, one-line hypothesis, start/end dates, total days run
- Sample sizes per variant, and any imbalance (this is your SRM check, see below)
- Primary metric: control value, variant value, absolute lift, relative lift, 95% CI, p-value
- Secondary metrics that moved meaningfully, positive AND negative
- Segment breakdown for at least 2 cuts (device, new vs returning, plan tier)
- Guardrail metrics (latency, error rate, refund rate), even if they did not move
- Known seasonality or external events during the window (sale, outage, holiday)
- The team’s current ship bar (e.g., “we ship if primary lifts and no guardrail breaks”)
Copy-ready prompt
Write a 1-page A/B test summary for our team meeting.
Test: [name + 1-line hypothesis + dates + days run]
Sample: [n_control / n_variant, note any imbalance]
Primary: [metric, control, variant, lift abs, lift rel, 95% CI, p]
Secondary: [list with deltas, mark + or -]
Segments: [device / cohort / plan splits]
Guardrails: [latency, error rate, refund rate]
External: [any holidays, outages, campaigns during window]
Ship bar: [our shipping criteria]
Return:
1) Headline - one sentence with the decision (ship / hold / kill) and the single most-important caveat
2) Primary result - translate the CI into plain English, no jargon
3) Secondary effects - explicitly call out anything negative
4) Segment view - was the lift concentrated in one segment while hiding a loss elsewhere?
5) Caveats - at minimum novelty effect, seasonality, sample-size adequacy, and sample-ratio mismatch
6) Decision and rollout plan
7) Next experiment to follow up on the loose endVariant prompt: exec-only TL;DR
Same inputs as above. But write a 5-line exec summary, not a 1-pager.
Line 1: ship/hold/kill in 6 words.
Line 2: lift + CI in plain English.
Line 3: the one thing that worried you most.
Line 4: rollout scope (100%, segment-only, or staged).
Line 5: the next test.
No headers, no bullets, no jargon.Sample output
Good headline: “Ship to desktop only. Variant B lifted activation 12% (true lift 7-17%, p=0.001), but the entire gain came from desktop; mobile users showed a flat 0.4% change inside noise. Mobile gets its own test next sprint.”
Exec TL;DR: “Ship variant B to desktop. Activation went up 12%, real lift between 7% and 17%. Worry: mobile flat, so the win isn’t universal. Roll out to desktop traffic only this week. Next test: mobile-specific version with a shorter form.”
The four numbers a skeptic checks first
Before you write the headline, sanity-check these. The model will repeat whatever you give it, so the gatekeeping is on you.
| Check | What “good” looks like | Why it matters |
|---|---|---|
| Sample-ratio mismatch (SRM) | Chi-square p ≥ 0.01 on the allocation split | A failed SRM (p < 0.01) means randomization or tracking broke; the whole result is untrustworthy, not just imprecise |
| Statistical power | Designed for ~80% power at your MDE | At 80% power you still miss a true winner 1 time in 5; a null result with low power is “we don’t know,” not “no effect” |
| Confidence interval | 95% CI excludes zero and is narrow enough to act on | A lift of +12% with CI [+0.5%, +23%] is technically significant but too wide to size the rollout |
| Test duration | At least 2 full business cycles (typically 2 weeks) | Shorter windows over-weight novelty and weekday/weekend skew |
If SRM fails, stop. Do not write a winner summary on top of a broken experiment. Tell the model: “SRM failed at p < 0.01; write this up as invalid and list the likely causes (broken event tracking, redirect bug, bot filtering, mid-test reallocation).”
Why “peeking” ruins the p-value
The single most common way teams ship a fake win: they watch the dashboard daily and call it the moment it crosses p = 0.05. Checking an experiment ~20 times instead of once can inflate the false-positive rate from the intended 5% to roughly 30-40%. The fix is either (1) pre-commit to a sample size and end date and only read the result once, or (2) turn on sequential testing, which most modern platforms (Statsig, GrowthBook frequentist mode) support and which widens the confidence bounds so you can monitor continuously without inflating error. If your platform offers CUPED variance reduction, it cuts the required sample by roughly 30-50% by regressing out pre-experiment behavior, so you reach a decision faster without peeking.
State which regime you used in the summary. A reader who knows you peeked daily on a fixed-horizon test will discount the p-value entirely.
How to refine the output
- If the model glosses over caveats: “Every A/B test has at least four caveats: novelty effect, segment heterogeneity, sample-size adequacy, and sample-ratio mismatch. Name each one with a one-line risk assessment.”
- If it worships the p-value: “Translate the lift into a user-meaningful unit: extra signups per week, extra dollars per cohort. A p-value alone is not a decision.”
- If the segment view is generic: “Pick the segment whose lift deviates most from the average and write one sentence on whether it should roll out separately.”
- If the headline hedges: “Force a decision: ship, hold, or kill. If you cannot pick, write ‘hold pending X’ and name what X is.”
- If the next-experiment idea is vague: “Propose one concrete follow-up with a hypothesis, the metric, and the segment to target.”
Common mistakes
- Reporting only the primary metric. The segment cut or a flipped secondary metric is usually the thing that changes the decision. Miss it and you ship a “win” that is a loss on mobile.
- Ignoring segment heterogeneity. A 5% average lift made of +15% desktop and -5% mobile is not +5% everywhere, and the rollout should not be the same either.
- P-value worship. A 0.3% lift at p = 0.04 on a million users is statistically significant and operationally meaningless.
- Hiding novelty effects. Day 1-3 lift of 18% and day 12-14 lift of 4% needs to be called out, not averaged away.
- No guardrail line. If the variant lifted conversion but doubled refund rate or latency, the summary should kill it, not ship it.
- Skipping the SRM check. A variant that “won” but has 52/48 traffic when you allocated 50/50 may have won because tracking dropped users, not because the change worked.
- Skipping the next-test slot. Every conclusive test opens at least one new question; the summary that does not propose the follow-up wastes the result.
FAQ
- When is the sample size big enough? Pre-compute power before launch using your baseline rate, minimum detectable effect (MDE), and 80% power, then commit to that sample and a fixed end date. A common MDE target is around 5%, dropping to 1-2% for very high-traffic sites and rising to ~10% for low-traffic ones. After launch, do not peek and do not chase significance; only the pre-registered analysis counts.
- What if results are mixed: primary wins but a secondary breaks? Default to “hold” and redesign. A variant that lifts revenue but increases churn is not a win; it is a deferred loss.
- How do we handle the novelty effect? Split the test window into thirds and report the lift for each. A variant that looks like a winner on day 3 can revert toward baseline by day 14 as the novelty fades. If the lift decays sharply by week 2, treat the test as inconclusive and re-run for longer.
- Can I trust a p = 0.06 result if the lift is huge? Pre-register the threshold (usually 0.05 at 95% confidence, the industry-standard default) and stick to it. A near-miss is a re-test signal, not a ship signal.
- What is SRM and why does the summary need a line for it? Sample-ratio mismatch is a statistically significant gap between your intended split (say 50/50) and the actual one, detected with a chi-square test at p < 0.01. It usually means tracking or randomization broke, which invalidates the experiment regardless of how good the lift looks.
- Should the summary include the raw numbers? Yes, in a small appendix or footnote. The body is for the decision; the numbers are for the skeptic who will ask.
Related
- AI experiment interpretation
- AI funnel analysis readout
- Decision Summary Prompts: 15 Templates for Memos People Actually Read
- AI Interprets Survey Results
Tags: #AI writing #Data analysis #Workflow #Experiment #A/B test