You run three instances of a LangGraph code-review agent. Production is on prompt v1.2, staging on v1.3, a canary on v1.4. Each version’s system prompt differs slightly: v1.2 checks for security issues, v1.3 added performance checks but dropped the formatting rules, v1.4 restored formatting but renamed an output-schema field. Reviewers see wildly inconsistent quality depending on which instance picked up their PR, and nobody can say what changed between versions because the prompts live as triple-quoted Python strings scattered across the codebase.
Fastest fix: stop loading prompts by latest. Pin an explicit, immutable version (a git tag, a Prompt Hub commit hash, or a Langfuse production label), and log the exact prompt version plus its SHA-256 hash with every run. Once you can read prompt_version and prompt_hash off each request, drift becomes a one-line diff instead of a mystery. The rest of this page is how to get there and how to keep it from recurring.
Which bucket are you in?
Match the symptom to the cause before you change anything.
| Symptom you observe | Most likely cause | Jump to |
|---|---|---|
| Two instances give different output for identical input | Loading latest/unpinned prompt; instances started at different times | Cause 3, Step 3 |
| Output format randomly reverts to free text | Renamed template placeholder, stale call sites render blank | Cause 4, Step 4 |
| Quality regressed for one PR type after a “small” prompt edit | No golden-output regression test | Cause 6, Step 5 |
| Cannot tell which prompt produced a bad run | No prompt version/hash in run logs | Cause 1/5, Step 2 |
| Prompt changed but no PR or diff exists | Inline string or remote config edited out of band | Cause 1/2, Step 1 |
Common causes
1. Prompts stored as inline strings with no version tracking
The system prompt is a triple-quoted string in agents/reviewer.py. When a developer edits it, the change rides along with a code commit. There is no separate history, no changelog, and the diff is invisible in the PR unless the reviewer opens the agent file.
How to spot it: search for multi-line strings in agent files. grep -rnE 'You are|SYSTEM_PROMPT|system_prompt\s*=\s*("""|f""")' agents/ surfaces prompts living without version tracking. Five or more independent definitions means there is no single source of truth.
2. Prompts assembled dynamically from multiple sources without a manifest
The final prompt is built from a base template, a task-specific snippet, and a format-instruction block, each from a different file or function. No manifest records which combination was used for a given run, so a one-piece difference is invisible until you compare all three sources.
How to spot it: grep for prompt concatenation or f-string assembly (system_prompt = base + task_prompt + format_block). If the fully assembled prompt is not logged as one string, you cannot know the exact combination a run used.
3. Prompt is fetched by latest with no pinning
The agent fetches its prompt from a config service or registry on startup, e.g. GET /prompts/reviewer/latest, client.pull_prompt("org/reviewer"), or langfuse.get_prompt("reviewer") with no label. “Latest”/default changes the moment someone updates the config, so two instances started minutes apart load different prompts. There is no version in the URL and no record of which version each instance loaded. This is the single most common cause of instance-to-instance drift.
How to spot it: audit every prompt fetch. Flag any call where the identifier is latest, has no version/commit/label suffix, or relies on the default (Langfuse serves the production label when none is given, so an unset label is still implicit pinning you should make explicit).
4. Prompt interpolation changes field names between versions
v1.3 renamed the placeholder {output_format} to {response_schema} but missed two call sites. Those sites still pass output_format; with str.format, an unused kwarg is silently ignored and the new placeholder renders blank, so the agent gets a prompt with an empty format section and emits free-form output.
How to spot it: compare the placeholders in each template against the kwargs passed to template.format(...) or Template.substitute(...). With str.format, a missing key raises KeyError, but an extra/renamed key is dropped silently — that asymmetry is the trap. Any mismatch is a drift indicator.
5. A/B test framework swaps prompts without recording the active variant per run
The pipeline A/B-tests prompt variants, switching variant per request via a runtime flag, but run logs do not record which variant was active. When debugging a bad output you cannot tell whether the agent was on variant A or B.
How to spot it: check whether variant assignment is written to run metadata. No prompt_variant field means you cannot correlate output quality with the active variant.
6. Prompt evolution is not covered by regression tests
A developer tweaks wording to help one scenario and quietly regresses another. With no golden-output tests, the regression surfaces in production via a user report rather than in CI.
How to spot it: search the test suite for any test that runs the agent on fixed inputs and asserts on output format or content. Zero such tests means prompt regressions ship undetected.
Shortest path to fix
Step 1: Move prompts to versioned files and track them in git
agents/
prompts/
reviewer/
v1.2.0.txt
v1.3.0.txt
v1.4.0.txt
CHANGELOG.md # what changed in each versionLoad by explicit version:
from pathlib import Path
def load_prompt(agent: str, version: str) -> str:
path = Path(f"agents/prompts/{agent}/{version}.txt")
if not path.exists():
raise FileNotFoundError(f"Prompt not found: {path}")
return path.read_text(encoding="utf-8")
REVIEWER_PROMPT = load_prompt("reviewer", "v1.4.0")Git history on the prompt files is now explicit, searchable, and attributable with git blame. Prefer plain .txt/.md so diffs stay readable; avoid burying prompts inside YAML or JSON where whitespace changes are hard to review.
Step 2: Log the prompt version and hash with every run
import hashlib
def get_prompt_with_metadata(agent: str, version: str) -> dict:
content = load_prompt(agent, version)
return {
"content": content,
"version": version,
"sha256": hashlib.sha256(content.encode("utf-8")).hexdigest()[:16],
}
# In every agent invocation:
meta = get_prompt_with_metadata("reviewer", REVIEWER_VERSION)
logger.info(
"agent=reviewer prompt_version=%s prompt_hash=%s run_id=%s",
meta["version"], meta["sha256"], run_id,
)If you emit OpenTelemetry spans, attach these as span attributes alongside the standard GenAI keys (gen_ai.request.model, gen_ai.operation.name). The OTel GenAI semantic conventions are still experimental as of June 2026 and have no dedicated prompt-version key, so use a stable custom attribute such as prompt.version and prompt.sha256. When two instances disagree, comparing prompt_hash is the fastest way to confirm drift.
Step 3: Pin prompt versions explicitly — never latest
Pin in deployment config so a change goes through review and deploy like any other:
# deployment/config.yaml
agents:
reviewer:
prompt_version: "v1.4.0" # explicit pin, never "latest"
coder:
prompt_version: "v2.1.0"config = load_deployment_config("deployment/config.yaml")
REVIEWER_PROMPT = load_prompt("reviewer", config["agents"]["reviewer"]["prompt_version"])If you use a managed prompt store, the same rule applies — pin to an immutable reference, not a moving pointer:
- LangSmith Prompt Hub (as of June 2026): every
push_promptcreates an immutable commit. Pin to a commit hash for reproducibility, e.g.client.pull_prompt("my-org/reviewer:a1b2c3d4"). Tags like:productionare mutable pointers; promoting staging to prod just moves theprodtag to a commitstagingalready points at — no redeploy. Use a tag for “currently active” and a commit hash when you need a run to be exactly reproducible. - Langfuse (as of June 2026): versions are immutable integers (1, 2, 3…); labels (
production,staging) are mutable pointers.langfuse.get_prompt("reviewer")serves theproductionlabel by default — make it explicit withget_prompt("reviewer", label="production"), or pin hard withget_prompt("reviewer", version=7). Setcache_ttl_secondsand afallbackprompt so a registry outage never silently changes behavior.
The anti-pattern to delete everywhere: GET /prompts/reviewer/latest or a bare pull_prompt("org/reviewer") in production code.
Step 4: Validate all prompt placeholders at load time
import string
class PromptConfigError(Exception):
pass
def validate_and_render(template: str, **kwargs) -> str:
formatter = string.Formatter()
template_fields = {
field for _, field, _, _ in formatter.parse(template) if field
}
provided = set(kwargs)
missing = template_fields - provided
extra = provided - template_fields
if missing:
raise PromptConfigError(f"Template missing kwargs: {missing}")
if extra:
logger.warning("Extra kwargs not in template (renamed placeholder?): %s", extra)
return template.format(**kwargs)Run this at startup (not per invocation) so a renamed placeholder like {output_format} to {response_schema} fails fast instead of silently rendering a blank section at runtime.
Step 5: Write golden-output regression tests for critical behaviors
GOLDEN_TESTS = [
{
"input": "Review: def login(u, pw): return db.query(f\"SELECT * FROM users WHERE pw={pw}\")",
"must_contain": ["SQL injection", "parameterized"],
"must_not_contain": ["looks good", "no issues"],
"prompt_version": "v1.4.0",
},
]
def test_reviewer_golden_outputs():
for case in GOLDEN_TESTS:
out = run_reviewer_agent(case["input"], prompt_version=case["prompt_version"]).lower()
for phrase in case["must_contain"]:
assert phrase.lower() in out, f"Missing '{phrase}' in: {out[:200]}"
for phrase in case["must_not_contain"]:
assert phrase.lower() not in out, f"Banned '{phrase}' in: {out[:200]}"Run these in CI before promoting any prompt version to staging. Because model output varies, assert on stable substrings and JSON schema validity rather than exact strings, or wrap the assertions in an LLM-as-judge eval with a pass threshold.
How to confirm it’s fixed
- Tail logs across all instances for one input class and confirm every line shows the same
prompt_versionandprompt_hash. Identical hash across the fleet means no drift. - Send one identical request to each instance and diff the responses; format and key fields should match.
- In CI, run a drift check that compares each registry/prod prompt hash against the pinned version in
deployment/config.yamland fails the build on mismatch. - Confirm the golden-output suite is green for the pinned version before any promotion.
Prevention
- Store prompts as versioned text files under source control, or in a managed registry pinned by immutable reference — never as ad-hoc inline strings.
- Log the exact prompt version and SHA-256 hash with every run so any past run is reproducible.
- Pin explicitly in deployment config; ban
latest, barepull_prompt(...), and unset labels in production code. - Validate template placeholders at startup; fail loudly on a variable mismatch rather than rendering a blank section.
- Keep a
CHANGELOG.mdper agent documenting what each version added, removed, or changed. - When A/B testing, write the variant into run metadata and chart variant against quality metrics.
- Review prompt changes like code changes — prompt text is production logic, not documentation.
- Run a scheduled drift check (CI or cron) comparing the running prompt hash to the pinned version, so an out-of-band edit gets caught within a deploy cycle.
FAQ
Q: Should prompts be versioned separately from the agent code that uses them? A: Yes. Prompts change for different reasons (quality tuning, new requirements, instruction bug fixes) than code (features, refactors). Separate versioning lets you roll back a prompt regression without touching code, and vice versa. Keep the prompt files in the same repo for atomic PRs, but give them their own version numbers and changelog.
Q: How do I manage prompts for a fleet of 50+ agents?
A: Use a registry that stores each prompt with a version, agent ID, environment label, and content hash. Agents pin and fetch their version at startup and cache it. As of June 2026, LangSmith Prompt Hub (immutable commits + mutable :production/:staging tags) and Langfuse (immutable version integers + production/staging labels) both provide this with a diff view between versions. Add a lint rule that bans defining prompt strings outside the registry so nobody bypasses it.
Q: Should the prompt live in code or in a database/registry?
A: If only engineers edit it, code wins: version control, PR review, and git blame come free. If non-engineers (marketing, support) must edit it live, a registry with mandatory version fields and an audit log is better. Either way, never let production run an unpinned prompt.
Q: How do I roll back a bad prompt version fast?
A: With files, point prompt_version in deployment/config.yaml at the previous tag and redeploy. With a registry, move the mutable pointer: in LangSmith move the :production tag to the prior commit; in Langfuse move the production label to the prior version — no code deploy needed. Then verify the hash across instances and re-run the golden suite to confirm the regression is gone.
Q: How do I evaluate whether a new prompt version is actually better? A: Build an eval set of 20-50 inputs with rated expected outputs. Run both versions and compare scores. Only promote if the new version raises the average without dropping any individual case below the old version’s score. LangSmith and Langfuse both support running an eval over a dataset against a specific prompt version.