You asked for help debugging a deploy. The model returned 4 polished paragraphs about “considering your environment configuration”, “investigating the deployment pipeline”, and “reviewing relevant logs”. No file paths. No exact commands. No specific config values to check. It reads like a consultant’s deck, not a fix. Polish without artifacts is decoration — it satisfies the form of “useful answer” without supplying any concrete handles a human can act on. The model is being polite, not unhelpful: training nudges it toward “consultative” register for open-ended asks, and consultative register has very few artifacts in it.
This page walks through why polished output stays polished even when you ask for specifics, and how to force artifacts via the schema rather than via more adjectives.
Common causes
1. Prompt asked for advice, not artifacts
“How should I think about X” gets you a thinking frame. “Produce X” gets you X. The verb determines whether you get a deliverable or a discussion.
How to spot it: your prompt verb is “advise”, “explain how”, “discuss”, “consider”.
2. No artifact requirement
If you do not say “include at least 1 file path, 1 command, 1 number”, the model omits them in favor of prose. Prose is cheaper to generate and reads as thoughtful.
How to spot it: your prompt has no list of artifacts required.
3. Soft verbs in the prompt leak into the output
If your prompt says “consider whether…”, the model echoes “you might want to consider…”. Soft verbs are contagious.
How to spot it: prompt contains “consider”, “think about”, “explore”, “look into”.
4. No example of the actionable output you want
You described actionable but did not show it. The model defaults to its training average of “actionable”, which is mostly still prose.
How to spot it: prompt describes form without showing one.
5. RLHF politeness on open asks
Modern chat models hedge on open questions to avoid being wrong. Hedging hides artifacts behind “may”, “might”, “depending on”.
How to spot it: every other sentence has a hedge word.
Before you change anything
- Identify what artifacts a useful answer would contain: file paths, commands, numbers, version pins, code snippets, schema fragments, named tools.
- Save the polished output for diffing against the actionable version.
- Decide who acts on the output and what they need to act.
- Plan a schema or template that requires artifacts as fields.
- Identify soft verbs in your current prompt to replace.
Information to collect
- Current prompt.
- The polished output that was not actionable.
- A list of artifacts an actionable version would contain.
- The downstream consumer of the output.
- Model and any system prompt.
Shortest path to fix
Step 1: Replace “advise” with “produce”
Bad: "Advise on how to fix the deploy."
Good: "Produce a 5-step fix as a numbered list. Each step must include:
- one command to run (in a code block),
- one file path to inspect or edit,
- the expected result (1 line)."The verb “produce” + the artifact list forces concreteness.
Step 2: Mandate artifacts in a schema
Output schema:
[
{
"step": <int>,
"command": "<exact shell command>",
"file": "<absolute or relative path>",
"expected_output": "<one-line string>"
},
...
]Schema fields cannot be filled with prose. If the model has nothing concrete, it must say UNKNOWN.
Step 3: Forbid soft verbs
Forbidden: "consider", "might want to", "you could", "perhaps", "explore".
Banned in your output. If you cannot give a concrete next step, write
"INSUFFICIENT_INFO" and ask for the specific data you need.Banning hedges forces the model to either commit or escalate.
Step 4: Provide a concrete example
Like this:
1. Run `vercel logs --since=10m` to see recent errors.
File: vercel.json (check buildCommand)
Expected: log shows "Error: env STRIPE_KEY missing"
Not like this:
1. Consider reviewing your deployment logs to look for any
anomalies that might be relevant to the issue.The contrast makes the form unambiguous.
Step 5: Have the model self-audit
Append:
After writing, count artifacts per step:
- Does each step have exactly 1 command, 1 file, 1 expected_output? Yes/No
- Total artifacts across the output: <count>
- If artifacts < 15 across a 5-step output, rewrite for more concreteness.Artifact-counting is mechanical and the model is good at it.
Step 6: Use real input data
If your prompt is generic (“help me debug”), the output will be generic. Paste the actual error message, the actual config, the actual logs. Concrete input pulls concrete output.
How to confirm the fix
- Output contains 3+ artifacts per logical unit (per step, per bullet, per section).
- A teammate reading the output can execute the fix without asking follow-up questions.
- Soft verbs (consider, might, perhaps) appear 0 times.
- Running the same prompt with the same input produces outputs with similar artifact counts.
- The output reads more like a runbook than a memo.
If it still fails
- The task may genuinely lack concrete handles given your input — paste more input data.
- Try a more capable model — some hedge less than others.
- Switch to a multi-step workflow: first prompt extracts facts, second prompt produces the action plan from facts.
- For repeated tasks, build a JSON-schema-enforced action-plan template at the API level.
Prevention
- Default: every “advise” prompt names the artifacts it must produce.
- Maintain an actionable-output checklist per task type (deploy fixes, code reviews, PRDs).
- Treat any output without artifacts as a draft, not a deliverable.
- Audit accepted outputs: count artifacts; if low, your prompts need tightening.
- Replace “advise / discuss / consider” with “produce / list / write” as default verbs.
- For team workflows, agree on a minimum artifact density (e.g., “every step has a command”).
Related reading
- AI answer too vague
- AI gives lists when execution is needed
- No success criteria
- Prompt asks for “best” without defining
- Negative constraints vague
Tags: #Troubleshooting #Prompt #Prompt quality #Vague answer