AI AI Tools Guidebook
Troubleshooting

Prompt Lacks Source Material Hierarchy

You attached three documents and the model treated them as equally authoritative — including the outdated draft.

Published: Author: AI Productivity Guide Team 🌐 查看中文版本

You attached three files: the current product spec, last month’s draft of the spec, and a slack thread debating one section. The model produced a summary that pulled from all three as if they were equally valid sources. The answer cited a feature requirement from the draft (which was rejected in the final spec) and ignored a change in the current version because the slack thread argued against it. The model has no way to know which document is authoritative. By default it averages, and averaging across “current” + “rejected draft” + “internal debate” gives you a hybrid that exists nowhere in reality.

This page walks through how to label sources with provenance and precedence so the model knows which to trust.

Common causes

1. Multiple sources, no labels

You attached files but did not name their role. The model sees three blobs of similarly-structured text.

How to spot it: you have 2+ sources and no “this is the canonical source” statement.

2. Old and new versions both present

Both v1 (rejected) and v3 (current) are in the prompt. The model has no temporal awareness — it cannot infer which is newer.

How to spot it: outdated facts from old versions appear in the output.

3. Mixed first-party and third-party

Your team’s spec + a third-party blog post about the same topic. The model treats them as equally authoritative. Worst case: the model preferred the third-party because it had more “confident” language.

How to spot it: output cites external content over your own.

4. Docs vs code conflict

Documentation says X, code does Y. The model picks whichever was longer or more recent in the prompt. Neither is necessarily right for your question.

How to spot it: doc + code disagree and the model picked the wrong one.

5. No citation requirement

If you do not require citation per claim, the model blends sources silently and you cannot trace where any fact came from.

How to spot it: claims in output, no way to know which source supplied them.

Before you change anything

  • List every source attached to your prompt.
  • For each, decide: canonical, supplementary, for-context-only, or remove entirely.
  • Identify temporal order: which is newest? Which is current? Which is rejected?
  • Decide your precedence rule: e.g., “current spec wins over draft; spec wins over slack; first-party wins over third-party.”
  • Plan to require citation per claim.

Information to collect

  • All sources you attached.
  • The output that pulled from wrong sources.
  • Your team’s actual precedence (who decides what is canonical).
  • Metadata for each source: date, author, status.
  • Model and any system prompt.

Shortest path to fix

Step 1: Label every source with name, date, role

<source name="product-spec" date="2026-05-15" role="CANONICAL">
... spec content ...
</source>

<source name="spec-draft" date="2026-04-01" role="REJECTED_DRAFT">
... old draft, kept for context only ...
</source>

<source name="slack-debate" date="2026-04-20" role="DEBATE">
... slack thread ...
</source>

Now the model has metadata to use.

Step 2: State precedence in one line

Precedence rules:
- Prefer CANONICAL over REJECTED_DRAFT over DEBATE.
- If CANONICAL and DEBATE conflict, CANONICAL wins.
- REJECTED_DRAFT is for context only; do not cite as authoritative.
- If sources agree, cite the canonical one.

Step 3: Remove obviously outdated sources

If the rejected draft is not needed for comparison, just delete it. The fewer sources, the fewer averaging mistakes.

Step 4: Mark first-party vs third-party

<source name="our-spec" role="FIRST_PARTY_CANONICAL">...</source>
<source name="competitor-blog" role="THIRD_PARTY_CONTEXT">...</source>

Rule: For factual claims about our product, prefer FIRST_PARTY.
       Use THIRD_PARTY only for industry context, never for our facts.

Step 5: Require citation per claim

For every factual claim in your output, cite the source in this format:
"[claim] [source=product-spec, section=2.3]"

If a claim cannot be cited, mark it [UNCITED] so the reviewer can verify.

This catches blending and lets you audit.

Step 6: For code vs docs, declare which wins

When docs disagree with code:
- For behavior questions: code is authoritative.
- For intent questions: docs are authoritative.
- Always cite which one you used.

How to confirm the fix

  • Every claim in the output has a citation.
  • Citations point to the canonical source, not drafts.
  • Outdated facts no longer appear.
  • A teammate can audit each claim against the cited source.
  • Re-running the prompt produces the same source distribution.

If it still fails

  1. Delete the secondary sources entirely; if the model needs only canonical, just give canonical.
  2. Use retrieval-augmented generation (RAG) where sources are chunked with explicit metadata.
  3. Two-pass: first prompt extracts facts from canonical only, second prompt does analysis.
  4. For high-stakes work, audit every citation manually before scaling.

Prevention

  • Default: every multi-source prompt declares precedence in one block.
  • Use filenames that encode role and date (spec-canonical-2026-05.md).
  • Maintain an “outdated, do not use” list for sources you keep around for comparison.
  • Require citation per claim in any analytical prompt.
  • For RAG setups, include source metadata in every chunk.
  • Audit production prompts: any multi-source prompt without precedence is a risk.

Tags: #Troubleshooting #Prompt #Prompt quality #Prompt engineering

Related Articles