You asked “what’s the best database for this project?” and got back a confident-sounding pick. Run the same prompt three times — you get three different “best” databases. Each pick has plausible reasoning. None of the reasoning anchors to your actual project. The model is not failing. You asked “best” without a decision rule, so it picked whichever option had the most positive token-level associations in training data for “best database”. That tends toward whatever was trendy when the model was trained. Without explicit weights, criteria, and a tie-breaker, “best” is whatever sounds confident — and confidence is not correctness.
This page walks through how to convert “best” into an explicit decision rule that produces a consistent, defensible answer.
Common causes
1. No axis specified
“Best” along what axis? Cost, latency, scalability, team familiarity? You did not say. The model picks an axis it likes — usually the one with the strongest training-data sentiment.
How to spot it: your prompt has no axis word (“cheapest”, “fastest”, “most reliable”).
2. Multiple criteria, no weights
You named criteria — “should be fast and cheap and easy to maintain” — but did not say which matters most. The model picks the criterion with the strongest sentiment in its prior.
How to spot it: 2+ criteria listed with no order or weight.
3. No constraints on what to ignore
If you do not say “ignore vendor brand reputation” or “ignore vibes”, those factors quietly dominate. Brand recognition is a huge implicit weight.
How to spot it: the model picked AWS / OpenAI / Stripe / similar default-popular options without your input data supporting it.
4. No tie-breaker
When two options are close, the model picks the one mentioned first or the one with more presence in training data. Random.
How to spot it: re-running the prompt picks different “winners” within the top 2-3.
5. Implicit criteria from prior turns
In a long conversation, earlier turns established a framing (e.g., “we care about cost”) that you forgot. The model still anchors on it for “best”.
How to spot it: same prompt in a new chat picks differently.
Before you change anything
- Write down the axis you actually care about.
- If multiple, rank them and assign rough weights.
- Identify what factors you want the model to ignore.
- Decide your tie-breaker rule.
- For decisions you make often, store these as a reusable rubric.
Information to collect
- Current prompt with “best” in it.
- Outputs from 3 separate runs.
- Your actual project context (workload, constraints, team).
- Past decisions you would consider “right” and why.
- Model and any system prompt.
Shortest path to fix
Step 1: Replace “best” with “best on [axis]”
Bad: "What is the best database for this?"
Good: "What is the cheapest database that handles 10k writes/min
with sub-100ms p99 read latency? Workload: 90% reads, 10% writes,
3kb average row, 100GB total."The axis (“cheapest”), the threshold (“10k writes/min”, “<100ms p99”), and the workload spec eliminate ambiguity.
Step 2: Use a weighted rubric
For multi-criteria decisions:
Criteria with weights:
- Cost: 40% (lower is better; cap $300/mo)
- p99 read latency: 30% (lower is better; max 100ms)
- Team familiarity: 20% (1=never used, 5=daily)
- Maintenance burden: 10% (1=managed service, 5=self-hosted)
For each option:
1. Score 1-5 on each criterion.
2. Multiply by weight.
3. Sum for total score.
4. Pick highest.
5. Show the scoring table.This forces explicit reasoning the model cannot hide.
Step 3: Declare what to ignore
Ignore in your evaluation:
- Vendor brand reputation.
- Whether the option is mentioned in HN comments.
- Training-data-recency bias.
- Marketing material's claims; use only documented limits and pricing.Naming ignored factors prevents them from sneaking in as implicit weights.
Step 4: Declare a tie-breaker
Tie-breaker (use when top 2 scores are within 0.3):
1. Prefer the option whose team has the most senior contributor.
2. If still tied, prefer the option with longer track record.
3. If still tied, return both with a note.Step 5: Ask for trade-offs as part of the answer
For the chosen option, list:
- 2 things this choice is worse at than the runner-up
- 1 thing you would lose if you were forced to switch later
- 1 unknown that could change the answerTrade-offs separate “best” from “best-given-our-constraints”.
Step 6: Build a reusable rubric file
For decisions you make often (vendor choice, library selection, architecture pattern), store the rubric in a file. Each new decision uses the same rubric with new options.
How to confirm the fix
- Running the same prompt 3 times produces the same “best” pick.
- The scoring table is reproducible; you can recompute it by hand.
- The trade-off list names concrete weaknesses, not vibes.
- A teammate following the same rubric reaches the same conclusion.
- The decision is defensible in writing.
If it still fails
- Criteria may be too vague — make each one quantitative if possible.
- Add 1-2 worked examples of past decisions to anchor.
- For decisions with sensitive context, the model may not have the data — provide it.
- Some decisions genuinely lack a “best” — the right answer is “pick any of these top 3; here are the trade-offs”.
Prevention
- Default: convert every “best” prompt into a weighted rubric.
- Keep rubric files per decision type (vendor, architecture, library, tool).
- For team decisions, agree on the rubric before running the prompt.
- Audit “best” decisions monthly — were they reproducible? If not, your rubric is missing something.
- Replace “best” / “most” / “top” with axis-specific language as a writing reflex.
- When in doubt, ask the model to propose a rubric first; review and adopt before asking for the pick.
Related reading
- Ambiguous evaluation criteria
- No success criteria
- AI answer too vague
- Output polished not actionable
- Conflicting instructions weaken output
Tags: #Troubleshooting #Prompt #Prompt quality #Prompt engineering