Why Your LLM App Hallucinates (and How to Reduce It)

What Hallucination Actually Is

The term "hallucination" is misleading. It implies the model is malfunctioning — doing something it should not. That framing leads to the wrong engineering decisions.

A language model generates tokens by sampling from a probability distribution learned from its training corpus. At each step, it predicts the next most likely token given everything that came before. The model has no access to reality. It has no notion of "checking" a fact. It is pattern-completing. When a pattern suggests that a sentence should contain a specific name, date, or claim — and no information in the context contradicts it — the model produces that token with high confidence regardless of whether it is true.

This is not a bug. It is the mechanism. Hallucination is what pattern completion without grounding looks like when applied to factual questions.

The engineering implication: hallucination cannot be eliminated by improving the model call. It is addressed by what you put into the model's context, how you constrain the model's outputs, and how you validate those outputs.

The Four Root Causes

Understanding which cause is active in your system determines which mitigation to apply.

1. Parametric Memory Gaps

The model's training data has a cutoff date and coverage gaps. Questions about events after the cutoff, niche domains underrepresented in training data, or private organizational knowledge will trigger hallucinations because the model has no relevant parametric knowledge — but it still generates output.

Example: Ask a model about a product updated three months ago. The model confidently describes the old API because that is what its training data contains.

Mitigation: Retrieval. Inject current, authoritative information into the context. Article six covers this in detail.

2. No Grounding in the Current Context

The model has relevant parametric knowledge, but the prompt contains no specific context that constrains what it says. The model fills in from memory, which is approximated and imprecise.

Example: "What did our CEO say at the Q4 earnings call?" The model has no access to your CEO or your earnings call. With no grounding constraints, it produces a plausible-sounding response from generic executive speech patterns.

Mitigation: Context injection. If the answer must come from a specific source, provide that source in the prompt and constrain the model to only use it.

3. Insufficient Constraint

The model has the relevant information in its context but is not sufficiently constrained to use only that information. It supplements context with parametric memory, which may contradict the context.

Example: You inject a product spec and ask for a feature summary. For features not clearly described in the spec, the model invents plausible details from similar products it knows about.

Mitigation: Explicit scope constraints in the prompt. "Answer using only information from the provided document. If information is not present in the document, say so explicitly."

4. Overconfident Response on Low-Certainty Domains

The model cannot model its own uncertainty accurately. It assigns high-confidence outputs to domains where its training data is sparse, inconsistent, or contradictory.

Example: A legal question about jurisdiction-specific regulations. The model has seen some legal text, generates a confident answer, but the specific regulation the answer is based on does not exist.

Mitigation: Uncertainty elicitation. Ask the model to express uncertainty explicitly and validate any claims that require precision.

What Your Reliability Layer Does Not Catch

Articles three and four built structural validation: the output is valid JSON, required fields are present, types are correct, ranges are within bounds. This catches format failures.

It does not catch semantic failures. A structurally perfect response can be completely wrong.

{
  "summary": "The company reported revenue of $4.2 billion in Q3 2024, up 23% year-over-year.",
  "wordCount": 17,
  "keyPoints": [
    "Q3 revenue: $4.2 billion",
    "Year-over-year growth: 23%",
    "Strong performance across all segments"
  ]
}

This passes every structural check. If the source document reported $3.1 billion and 11% growth, your validation layer returned valid: true on a response that is materially wrong.

This is the regime your reliability layer cannot handle. It requires different tools.

Mitigation 1: Context Constraint Prompting

The most direct mitigation. You provide the relevant context and explicitly prohibit the model from going outside it.

You are a document question-answering system.

Answer the user's question using ONLY the information in the document below.
If the answer is not contained in the document, respond with: {"answer": null, "reason": "Information not found in document"}.
Do not infer, speculate, or supplement with outside knowledge.

DOCUMENT:
{{document_text}}

The key constraint: "If the answer is not contained in the document, respond with a null answer." This is critical. Without it, the model will attempt to answer anyway, using parametric memory. With it, you trade false confidence for honest incompleteness — which is the correct behavior.

Testing this constraint: Build a test set of questions where the answer is definitely NOT in the document. Verify that the model returns null answers, not invented ones. If the model returns non-null answers on questions with no grounding, the constraint is not tight enough.

Mitigation 2: Uncertainty Elicitation

For domains where the model has relevant knowledge but precision is critical, ask the model to express its uncertainty:

You are a technical information assistant.

Answer the question below. After your answer, provide a confidence field:
- "high": you are certain this information is accurate and current
- "medium": you believe this is accurate but recommend verification
- "low": this is your best approximation; authoritative verification is required

Return JSON:
{"answer": "<string>", "confidence": "high"|"medium"|"low", "caveats": ["<caveat>", ...]}

Important: models are systematically overconfident. The confidence field here is a rough signal, not a calibrated probability. Use it as a filtering mechanism (route low-confidence answers to human review), not an acceptance criterion.

Calibrate it explicitly: collect 100 responses with confidence labels, validate each against a ground truth, measure how often "high" confidence answers are actually correct. If "high" is right 90% of the time and "low" is right 60% of the time, you have a meaningful signal. If "high" is right 75% of the time, the signal is weak and you need additional validation.

Mitigation 3: Source Citation Enforcement

Force the model to cite the specific source material its answer is based on:

You are a document analyst. Answer questions about the provided document.

Rules:
- Quote the exact sentence(s) from the document that support your answer.
- If you cannot find supporting text, do not answer.
- Do not paraphrase beyond what the source text conveys.

Return JSON:
{
  "answer": "<string>",
  "citations": ["<exact quote from document>", ...]
}

Citations serve two purposes. First, they constrain the model — having to produce a citation changes the model's generation behavior, making it harder to hallucinate (hallucinated content would need a fabricated citation, which is harder to produce in a constrained schema). Second, they are verifiable post-hoc. Your validation layer can check whether the cited text actually appears in the source document:

function validateCitations(
  response: { answer: string; citations: string[] },
  sourceDocument: string
): string[] {
  const errors: string[] = [];
  for (const citation of response.citations) {
    if (!sourceDocument.includes(citation.trim())) {
      errors.push(`Citation not found in source document: "${citation.slice(0, 80)}..."`);
    }
  }
  return errors;
}

This is a deterministic check. If the citation does not appear in the source document verbatim, the model fabricated it. This is one of the most reliable hallucination detection signals you can build.

Mitigation 4: Constraining the Answer Space

For classification, entity extraction, or any task where the correct answer belongs to a known set, explicitly enumerate the valid answers:

Classify the document type.
Valid types: "contract", "invoice", "purchase_order", "correspondence", "other"
Return ONLY one of these values. Do not return any other value.

When the answer space is closed, the model cannot hallucinate outside it. The worst case is a wrong classification — still wrong, but deterministically within the expected value range. This eliminates an entire class of hallucination failures.

For open-ended tasks where the answer cannot be enumerated, you cannot fully close the answer space. Apply the context constraint and citation enforcement patterns instead.

Mitigation 5: Temperature as a Calibration Tool

Higher temperature increases the model's randomness, pulling it toward lower-probability tokens. At temperature 0, the model consistently picks the highest-probability completion. At temperature 1, it samples more broadly.

For factual, grounded tasks where you have provided context: use low temperature (0.1–0.3). The highest-probability completion given your context is more likely to be grounded in that context than a lower-probability sample.

For creative or open-ended generation: higher temperature is appropriate.

Temperature adjustment will not eliminate hallucination — a high-probability hallucination at temperature 0 is still a hallucination. But for context-grounded tasks, lower temperature measurably reduces the rate.

Do not use temperature 0 universally. Some tasks benefit from sampling diversity (brainstorming, variation generation). Set temperature per task type, not globally.

The Failure Walkthrough: Confident Wrong Answers

Scenario: A customer support assistant answers product questions. The product catalog is injected into the context. A user asks: "Does the Pro plan support single sign-on?"

Model response:

{
  "answer": "Yes, the Pro plan supports SAML-based single sign-on (SSO) and integrates with Okta, Azure AD, and Google Workspace.",
  "confidence": "high"
}

Validation result: passes all structural checks. Passes the confidence filter (high).

The problem: The Pro plan in the current catalog does not mention SSO. SSO was removed from the Pro plan two months ago and moved to Enterprise. The model's training data predates this change, and the injected catalog text did not explicitly state "SSO is not available on this plan." The model completed the pattern with its parametric memory.

Why this passed your reliability layer:

• Structurally correct: valid JSON, correct fields
• Semantically valid: a real feature described correctly for the old plan
• High confidence: the model is pattern-completing with strong prior probability

What would have caught it:

1. Explicit negative constraint: "If a feature is not mentioned in the provided catalog text, respond with \"answer\": null. Do not infer feature availability."

2. Citation enforcement: If the model is required to cite the specific catalog text, and SSO does not appear in it, the validation layer catches a missing or fabricated citation.

3. Ground truth evaluation: A test case in your eval dataset: "Does the Pro plan support SSO?" expected output: null or explicit "not available." Running this before deployment catches the regression.

Fix applied:

• Added explicit negative scope instruction to system prompt
• Added citation validation to the response pipeline
• Added 8 test cases specifically testing questions about removed/unsupported features

What Retrieval Changes About This

The mitigations in this article work when the authoritative information can fit in a single prompt — a product spec, a single document, a known set of classification labels. When your knowledge base is larger than your context window, you need retrieval: find the relevant information at query time and inject it.

That is article six. The patterns in this article assume the context is already assembled. Article six covers how to assemble it.

Hallucination Mitigation Checklist

• [ ] System prompt includes explicit prohibition against using knowledge outside provided context
• [ ] "Information not found" case has a defined output (null answer, explicit error field)
• [ ] For fact-critical tasks: citations are required and verified against source text
• [ ] For classification tasks: answer space is enumerated and constrained to the closed set
• [ ] Temperature is set per task type (low for factual, higher for creative)
• [ ] Confidence elicitation is calibrated against ground truth labels before use
• [ ] Eval dataset includes test cases specifically designed to test for hallucination on your domain