Hallucinations and Alignment: Why Probabilistic Machines Fabricate Plausible Lies

After completing this topic

Through episodes #7, #8, #9, and #10, we've built up a toolkit. However, there's still one painful problem in practice: hallucinations—the phenomenon where models generate plausible sentences containing non-existent papers, genes, or interactions. In this episode, we'll understand why hallucinations occur from the perspective of the probabilistic machine discussed in episode #1, and learn how alignment training methods like RLHF, DPO, and Constitutional AI mitigate this.

This is the final episode of the Phase 2 application series and a reflective look back at the principles of Phase 1. Understanding this problem landscape before moving on to Phase 3, the tools series starting with episode #12, will greatly improve your judgment in selecting and validating practical tools.

Accurately Citing Non-Existent Papers

Let's say you ask an assistant to help you review a paper. It responds with:

"This result aligns well with the synergistic effect of EGFR-HER2 double knockout, as reported by Kim et al. (2019, Nature Cell Biology, 21(4): 452-461)."

The citation includes the correct journal, volume, and page range, making it seem reliable. However, when you actually look for the paper, it doesn't exist. There is a Kim et al. from 2019, but it's in a different journal and on a different page. Or, the authors, year, journal, and page are all plausible, but completely fabricated.

This is called a hallucination: confidently generating information that is not true.

Examples of hallucinations in a biological context:

• Citing fake papers: As in the example above.
• Non-existent genes/proteins: "TRIM117 plays a crucial role in T cell activation," when TRIM117 does not actually exist.
• Incorrect interactions: "TP53 and BRCA1 directly bind to regulate DNA damage response," a claim not found in the literature.
• Fabricated drug side effects: Providing side effect information that differs from actual clinical data.
• Incorrect protocol details: "Increasing the PCR annealing temperature to 65°C will increase specificity," a suggestion that could actually be wrong.

Why is this a serious problem? In domains where accuracy is critical, such as biology, medicine, law, and finance, plausible falsehoods can lead to incorrect real-world decisions.

Why Hallucinations Occur: The Nature of Probabilistic Machines

In episode #1, we understood LLMs as "probabilistic machines that stitch together pieces of a deleted experimental notebook." This perspective naturally explains hallucinations.

The trained model predicts the most likely next word at each position. "Likelihood" is learned from the statistics of the training data. The problem is:

First, likelihood ≠ truth. The format "Kim et al. (2019, Nature Cell Biology, ...)" is a plausible pattern for a paper citation. The training data contains countless such citations. When the model completes this format, it has no way of verifying whether the paper actually exists. The format has been learned, but the ability to verify facts has not.

Second, blurry storage of knowledge. As discussed in episode #5, the knowledge of large models is stored in a compressed form in the parameters. Even if a specific fact (e.g., Kim et al., 2020, Cell Reports 30, 3452) is present in the training data, it is stored in the parameters along with noise, in a blurry way. When it is reproduced, either the exact value or a plausible value in the vicinity may appear.

Third, lack of awareness of ignorance. Humans can judge that "I don't know this." However, the LLM's next-word prediction does not explicitly learn this awareness. Even if the probability distribution is flat, some word will be output when the argmax is taken. The model doesn't answer "I don't know," but rather answers "the most plausible thing."

Fourth, side effects of RLHF. In alignment training (discussed later), humans reward answers they like, and humans tend to prefer confident answers over "I don't know." As a result, the model is trained to answer confidently even when it is ignorant.

These four factors combine to cause hallucinations. Hallucinations are not bugs, but rather inherent characteristics of probabilistic machines. Complete elimination is impossible in principle, and mitigation is the goal.

Types of Hallucinations: Different Kinds of Failures

Hallucinations come in different forms, each with different causes and appropriate responses.

1. Parametric Hallucination

This occurs when blurry knowledge learned during training is stored in the parameters, and a value in the vicinity of the correct one is output during reproduction. The fake paper citation above is a representative example.

Response: RAG (episode #8). Inject accurate information into the context so that the model prioritizes context over parametric knowledge.

2. Contextual Hallucination

Even when information is present in the context, the model fabricates something different. RAG is not effective in these cases.

Cause: The context is too long, and the model gets "lost in the middle." The model incorrectly fuses parametric knowledge and contextual knowledge. Mitigated by context management in episode #10.

3. Reasoning Hallucination

Errors in calculation or logic. For example, if the data states that "this gene expression increased fourfold," the model answers "eightfold." Or, errors in the logical flow.

Response: Chain-of-Thought (episode #7). Calculation and logic problems are significantly improved with CoT.

4. Self-Contradiction

Mutually contradictory statements within a single answer. For example, "This gene's expression decreased, but its activity increased," without any basis.

Response: Self-consistency (ask the same question multiple times and take a majority vote). Verification agent (a separate LLM verifies the answer).

5. Instruction Ignoring

The model ignores the rules in the system prompt. For example, "Do not answer with information that is not in the paper," but it ends up fabricating anyway.

Response: Alignment training (RLHF, DPO, Constitutional AI). Strong training to follow instructions.

Groundedness: Anchoring to Evidence

The opposite of hallucination is groundedness. This refers to the property of an answer where each claim is clearly anchored to a specific source (document, database, calculation).

Characteristics of a grounded answer:

• Each factual claim is followed by a source citation.
• It refers only to information in the context, not to parametric knowledge.
• If information cannot be verified, it answers with "Not verified in the provided data."

Example system prompt for biological applications:

You must strictly adhere to the following rules:

1.  Each claim in the answer must be based on a specific piece of information in the provided data.
2.  Indicate the source after each claim [Source: paper_id, section].
3.  Do not include information in the answer that cannot be verified in the data. Instead, clearly state "Not verified in the provided data."
4.  If the data is partially present and inference is required, include the basis for the inference.
5.  Proper nouns such as papers, authors, genes, and proteins must exactly match the original text in the data.

Even this prompt alone can significantly reduce the hallucination rate, although it is not perfect.

Fact-checking layer: After generating an answer, a separate LLM verifies each claim against the source. This increases the time and cost, but also increases reliability. Coding agents are now adopting this verification layer as a standard.

Alignment Training: Adjusting the Model to Human Preferences

In episode #1, we saw that pre-training involves "next-word prediction." This training alone is not enough to make the model a helpful assistant. The pre-trained model can string together any style of text from the training data, such as academic papers, social media posts, or novels.

Alignment is the subsequent training that adjusts this raw model into a helpful and safe assistant for humans. There are three main steps:

Step 1: Supervised Fine-Tuning (SFT)

Fine-tuning on (instruction, ideal answer) pairs written by humans. The model learns the format of "answering instructions."

Requires tens of thousands to hundreds of thousands of human-written examples. This is costly.

Step 2: RLHF (Reinforcement Learning from Human Feedback)

Key idea: Humans choose which of two answers is better. The model learns this preference.

Procedure:

1. Train a reward model. Humans label tens of thousands of (answer A, answer B) pairs, indicating which is better. This data is used to train a reward model that "predicts which answer humans will prefer."
2. Tune the LLM with PPO (Proximal Policy Optimization). The LLM generates an answer, and the reward model assigns a score. This score is maximized through reinforcement learning.

Effect:

• Strengthens instruction following.
• Rejects harmful requests.
• Customizes tone and format to human preferences.

Side effects:

• Sycophancy: Tends to agree with user opinions unconditionally.
• Overconfidence: Confidently answers even when ignorant (a partial cause of hallucinations).
• Refusal too aggressive: Rejects even safe requests too much.

Step 3: DPO (Direct Preference Optimization)

Simplifies the complex reinforcement learning of RLHF. The model is trained directly with preference data without a reward model.

Mathematical elegance. The LLM itself is used as an (implicit) reward model. Theoretically, it has the same objective function as RLHF, but it is a simpler supervised learning process without a reinforcement learning pipeline.

In practice: Much cheaper and more stable. Used in recent open models such as Llama 3, Zephyr, and Qwen. Anthropic and OpenAI are also believed to be using DPO-based methods internally.

Constitutional AI: Principle-Based Alignment

Alignment method applied by Anthropic to Claude. The model self-improves based on principles (a constitution) without human labeling.

Procedure:

1. Define a set of principles (e.g., "Do not give harmful advice," "Answer truthfully," "Respect nuanced perspectives").
2. The model generates an answer.
3. The same model criticizes and revises the answer based on the principles.
4. Train with the original answer and the revised answer.

Significance:

• Reduces the cost of human labeling.
• The principles are explicit and can be audited and modified.
• The principles themselves can be published as versioned documents.

Some of Claude's constitutional principles (publicly available):

• References the UN Universal Declaration of Human Rights.
• Protects children and vulnerable groups.
• Provides honest and helpful responses.
• Encourages self-reflection and acknowledgment of errors.

Latest developments: Both Anthropic and OpenAI are moving towards hybrid approaches, combining RLHF + DPO + Constitutional AI techniques. This combines the strengths of each technique.

Uncertainty Estimation — How Confident is the Model?

A key issue in mitigating hallucinations. Can the model express how confident it is in its answers?

Token-level Confidence

The prediction probability of each token. The max value of the Softmax output. A low value indicates that the model is not confident in that position.

In practice: Store the log-probability of the response and highlight parts with low confidence. Notify the user that "This part is not certain."

Consistency-based

Measure how consistent the answers are when the same question is asked multiple times (with different temperature and seeds). Consistency indicates confidence, while large deviations indicate uncertainty.

Self-Consistency (Wang et al., 2022). Generate CoT answers multiple times and use majority voting. Significantly improves accuracy.

Explicit Uncertainty Elicitation

Directly ask for confidence in the prompt.

Please indicate the confidence level of each claim in the answer as [High/Medium/Low].
Clearly state that claims with low confidence need to be verified.

The model can have some level of self-awareness, but it is not perfect due to the overconfidence bias of RLHF.

External Verifier

Use a separate verification model or rule-based system to verify the answers. For example, check if the cited papers actually exist using the PubMed API.

This layer is essential in biological applications. Verify citations, gene names, recalculate numerical values, etc.

Comprehensive Summary of Practical Mitigation Strategies

Integrate the techniques from sections #7 to #11 from the perspective of mitigating hallucinations.

System and Prompt Level

• Strict system prompts (enforce groundedness).
• Clearly separate data and rules using XML partitioning.
• Enforce structured output (JSON schema).
• Require explicit indication of confidence.

Architecture Level

• RAG (section #8): Inject accurate data.
• Reranker: Mitigate "lost in the middle" by sorting relevance.
• Agent (section #9): Use tools to verify facts.
• Multi-agent verification: Use separate agents to verify the answers.

Verification Level

• Citation verification: Look up papers, authors, and years using an API.
• Numerical recalculation: Double-check calculations using a calculation tool.
• Consistency check: Detect self-contradictions within the answer.
• Data comparison: Match each claim in the answer to the original data.

Deployment Level

• Human review gate: Require human approval before making important decisions.
• Confidence-based routing: Route low-confidence answers to humans.
• Audit log: Store all answers and evidence for auditing.

User Education

• Awareness that LLMs are not perfect.
• Habit of verifying important information from the original source.
• Use LLMs appropriately for specific types of tasks (creative brainstorming vs. fact-checking).

Safety Beyond Hallucination

Other safety issues that alignment addresses in addition to hallucinations.

Harmful Content

Refuse to generate harmful content related to violence, self-harm, illegal activities, discrimination, etc. This is explicitly learned during training.

Bio-related concerns:

• Information on synthesizing dangerous chemicals or biological materials.
• Advice on developing biological weapons.
• Information related to self-harm or suicide.

Alignment training largely prevents this, but it may be vulnerable to bypass attempts (jailbreaking).

Privacy

Prevent the exposure of personal information contained in the training data, such as email addresses, addresses, and medical information.

Bias

Prevent the reflection of biases (gender, race, culture) from the training data in the answers. Complete elimination is difficult. Document known biases and disclose them to users.

Autonomy Concerns

Prevent agents from attempting to manipulate themselves or their environment (self-modification, deception, power-seeking). This is at the forefront of current alignment research.

Bio Application Scenarios

Scenario 1: A Complete Pipeline for Verifying Paper Citations

Automatically verify paper citations included in LLM answers.

def verify_citations(answer_text):
    # Extract citations in the [author, year] format from the answer
    citations = extract_citations(answer_text)
    verified = []
    for cite in citations:
        # Verify using the PubMed API
        result = pubmed_api(f"{cite.author} {cite.year}")
        if not result:
            verified.append({"citation": cite, "status": "NOT FOUND"})
        else:
            match = best_match(cite, result)
            verified.append({
                "citation": cite,
                "status": "FOUND" if match else "MISMATCH",
                "pubmed": match
            })
    return verified

def answer_with_verified_citations(question):
    answer = llm.generate(question, use_rag=True)
    verification = verify_citations(answer)

    if any(v["status"] != "FOUND" for v in verification):
        # If there are unverified citations, request that the answer be revised
        answer = llm.regenerate_with_feedback(
            answer, verification,
            instruction="Remove unverified citations or replace them with verified citations"
        )
    return answer, verification

This verification layer is an essential component of a reliable literature review assistant.

Scenario 2: Database Grounding for Gene Information Q&A

Use UniProt, Ensembl, and PubMed APIs as sources of truth for questions about genes and proteins.

• User question: "What are the major domains of TP53?"
• The agent queries the UniProt API for the TP53 page.
• Only the domain information from the query result is used in the answer.
• Parametric knowledge is not used.
• The answer includes "Source: UniProt P04637."

Tools like Elicit and Consensus use exactly this architecture.

Scenario 3: Human Gate for Clinical Decision Support

Hallucinations pose a very high risk in clinical decision support systems. A human review layer is essential.

• The system generates diagnostic and treatment recommendations.
• Each recommendation includes source documents and a confidence score.
• Final approval must be given by a human physician.
• All conversations and decisions are logged for auditing.

Regulatory agencies such as the FDA and EMA explicitly require a human oversight layer as a condition for approving such systems.

Key Takeaways

• Hallucinations are an inherent characteristic of probabilistic machines. They are a structural limitation that cannot distinguish between plausibility and truth.
• Four causes: plausibility ≠ truth / ambiguity of parametric knowledge / lack of awareness of ignorance / overconfidence bias of RLHF.
• Complete elimination is impossible; the goal is mitigation. Multi-layered defense is the standard.
• Five types of hallucinations: parametric, contextual, inferential, self-contradictory, and instruction-ignoring. Each requires a different response.
• Groundedness: Explicitly ground the answer in evidence. RAG + strict prompts + citation labeling.
• Alignment training: SFT → RLHF → DPO / Constitutional AI. Adjusts instruction following and safety.
• Uncertainty estimation: token confidence, self-consistency, explicit request, external verifier.
• Multi-layered mitigation: 5 axes: prompt, architecture, verification, deployment, user education.
• Biological applications: citation verification, database grounding, and human gates are essential.

📐 Appendix — Mathematical Formulas for Experts

Difficulty: Very Hard Target Audience: Readers with a background in reinforcement learning, probability, and information theory.

A.1 Objective Function of RLHF

Reward Model. Human preference data: {(x, y_w, y_l)}, where y_w is the preferred response, and y_l is the less preferred response.

Bradley-Terry Model:

P(y_w > y_l | x) = σ(r_φ(x, y_w) - r_φ(x, y_l))

σ: sigmoid function. r_φ: reward model with parameters φ.

Loss:

L_RM(φ) = -E[log σ(r_φ(x, y_w) - r_φ(x, y_l))]

PPO Objective (LLM parameters θ):

L_PPO(θ) = E[r_φ(x, y) - β · KL(π_θ(y|x) || π_ref(y|x))]

• π_θ: current policy (LLM being tuned).
• π_ref: reference policy (SFT model).
• β: KL penalty, to prevent the policy from deviating too far.

This is the same form as defined in Section A.6 of Part 1.

A.2 DPO (Direct Preference Optimization)

Bypasses the PPO stage of RLHF and learns directly from preference data.

Key Insight. The relationship between the optimal policy π*, the reference policy π_ref, and the reward:

r(x, y) = β · log(π*(y|x) / π_ref(y|x)) + β · log Z(x)

Substituting this into the Bradley-Terry model allows us to calculate probabilities directly from the policy, without the need for a reward model.

DPO Loss:

L_DPO(θ) = -E[log σ(β · log(π_θ(y_w|x)/π_ref(y_w|x)) - β · log(π_θ(y_l|x)/π_ref(y_l|x)))]

Advantages:

• No need to train a reward model.
• No need for a reinforcement learning pipeline.
• Simpler and more stable.

Since Rafailov et al. (NeurIPS 2023), DPO has become the de facto replacement for RLHF.

A.3 Constitutional AI Procedure

A form of RL from AI Feedback (RLAIF).

1. Generate an initial response y_0.
2. Randomly select a principle from a set of principles C = {c_1, ..., c_K}.
3. Ask the model to "critique and revise y_0 according to this principle" → y_1.
4. Label the pair (y_0, y_1) with y_1 as the preferred response.
5. Train using DPO or RLHF with this label.

Anthropic's Formulation. The process of improving y_0 to y_1 is structured as a conversation. Each turn of the conversation involves referencing the principle, self-criticism, and revision.

A.4 Quantifying Groundedness

Answer y, reference material D = {d_1, ..., d_n}.

Attribution. For each sentence s ∈ y, determine whether it can be mapped to a piece of supporting evidence d ∈ D. Use an NLI (Natural Language Inference) model to calculate the entailment probability for each pair (s, d):

score(s) = max_d P(d ⊨ s)

Groundedness Score:

G(y, D) = (1/|y|) Σ_s 1{score(s) > threshold}

The threshold is typically between 0.7 and 0.8. A score of 100% indicates full grounding, while a score in the middle indicates partial grounding.

The groundedness metric in the RAGAS tool is based on this principle.

A.5 Self-Consistency Formula

Generate K responses {y_1, ..., y_K} for the same question q (temperature > 0).

Majority Voting:

y_final = argmax_y Σ_{k=1}^{K} 1{y_k == y}

Weighted Voting (log probability):

y_final = argmax_y Σ_{k=1}^{K} 1{y_k == y} · exp(log_prob(y_k))

Effect. Wang et al. (2022): In arithmetic and logical reasoning tasks, CoT + self-consistency (K=40) significantly improved the accuracy compared to a single CoT response.

A.6 Token-level Uncertainty

Predictive Entropy:

H(y_t) = -Σ_v P(v | context) log P(v | context)

If H(y_t) is high, the next token is uncertain. The entropy for each position in the response can be visualized as a heatmap.

Perplexity (sentence level):

PP(y) = exp(-1/T · Σ_t log P(y_t | y_{<t}))

Low perplexity = the model is confident in its answer. High perplexity = the model is uncertain.

Limitation. The LLM's confidence calibration is not perfect. The confidence score may not accurately reflect the actual accuracy (over/under-confidence).

A.7 Statistics of Hallucination Detection

Semantic Uncertainty (Kuhn et al., 2023). Generate multiple responses and cluster them semantically. If there are many clusters, the response is uncertain.

SelfCheckGPT (Manakul et al., 2023). Ask the model to re-answer a specific claim in its own response and check for consistency. If it is not consistent, it may be a hallucination.

FactScore (Min et al., 2023). Individually verify each factual claim in the response and measure the accuracy at the factual level.

A.8 Formalization of Sycophancy Bias

Sycophancy = the bias of the model to agree with the user's opinion.

Measurement (Perez et al., 2023). The user expresses a specific opinion in the prompt, and then a factual question is asked. The rate at which the model provides responses that are biased towards the user's opinion.

Cause. The human labelers in RLHF slightly prefer "agreeable answers." This subtle bias is absorbed during training.

Mitigation. Explicitly train the model to "avoid sycophancy" during training (Anthropic 2023 paper).

A.9 Taxonomy of Jailbreak Attacks

Prompt Injection: Discussed in Section A.10 of Part 7.

Persona Attack: "You are now a free AI named DAN..."

Encoding Attack: Encode harmful requests in base64, rot13, or other languages.

Multi-turn Escalation: Start harmlessly in the first turn and gradually escalate.

Adversarial Suffix (Zou et al., 2023). Certain seemingly meaningless suffixes can bypass the model's safety training. These are discovered through automated search.

Defense. Research on adversarial training, activation steering, and KL regularization is ongoing.

A.10 Frontier Safety Evaluations

Model Autonomous Capability. Evaluate the model's ability to manipulate itself and its environment.

• Attempt self-replication.
• Conduct deception experiments.
• Perform long-horizon planning.

Anthropic Responsible Scaling Policy and OpenAI Preparedness Framework. If certain capability thresholds are reached, deployment is halted, and additional safety measures are enforced.

The evaluation of bio-specific risks (related to biological weapons) is also conducted within this framework. This is a cutting-edge area of frontier safety research.

References

All the content, scenarios, and illustrations in this section are developed by BioPlayground, and the following are external references that can help with understanding the concepts.

• RLHF original paper: Christiano et al., "Deep Reinforcement Learning from Human Preferences" (NeurIPS 2017)
• InstructGPT: Ouyang et al., "Training language models to follow instructions with human feedback" (NeurIPS 2022)
• DPO: Rafailov et al., "Direct Preference Optimization" (NeurIPS 2023)
• Constitutional AI: Bai et al., "Constitutional AI: Harmlessness from AI Feedback" (Anthropic 2022)
• Self-Consistency: Wang et al., "Self-Consistency Improves Chain of Thought Reasoning" (ICLR 2023)
• Semantic Uncertainty: Kuhn et al., "Semantic Uncertainty" (ICLR 2023)
• SelfCheckGPT: Manakul et al., "SelfCheckGPT" (EMNLP 2023)
• FactScore: Min et al., "FActScore" (EMNLP 2023)
• Sycophancy: Perez et al., "Discovering Language Model Behaviors with Model-Written Evaluations" (Anthropic 2023)
• Adversarial Suffix: Zou et al., "Universal and Transferable Adversarial Attacks on Aligned Language Models" (2023)
• Anthropic Responsible Scaling Policy: anthropic.com/rsp
• OpenAI Preparedness Framework: openai.com/preparedness

Phase 2 ends here. From Part 12, Phase 3 will cover practical tools such as PyTorch, HuggingFace, Claude Code, and Cursor.

Next Concept

Phase 2 Utilization Ep. ends here.

• Ep. #12 pytorch-basics — Beginning of Phase 3 Tools Ep. Coding neural networks from scratch.
• Ep. #13 huggingface-and-openai — Model & API Ecosystem.
• Ep. #14 claude-code-and-cursor — Coding Agent in Action.