What Is an LLM — A Probabilistic Machine That Reconstructs Erased Lab Notes

What You'll Learn From This Chapter

You'll understand the underlying principle of how ChatGPT, Claude, and Gemini actually generate their replies. You'll be able to stitch together, into a single story, how a simple rule called next-token prediction develops through pre-training, autoregressive generation, and alignment via human feedback into today's AI assistants; why they sometimes hallucinate; and how a biology researcher can put them to work in the lab.

This chapter is the first in the AI Native track and serves as a map. All the concepts introduced here will be explored in depth across the next 14 chapters, so for now, focus on grasping the big picture.

Imagine a Ruined Lab Notebook

Suppose you're a graduate student. You've just inherited an old lab notebook from a senior who left for study abroad. But the moment you open it, disappointment sets in. Someone must have spilled coffee on it many times, because the right half of every page is smeared beyond recognition. The left side still shows the experimental sequence, materials, and conditions, but the observations, results, and interpretations that should be on the right are illegible.

You need to restore this notebook. And, as luck would have it, you happen to have a magical machine at hand. Given any lab record, if you feed the beginning to this machine, it predicts the single most plausible next word to follow. What could you do with such a machine?

The procedure is simple. Feed the entire surviving left half into the machine. The machine predicts the first word that should come next. You paste that word into the notebook. Now feed the whole updated sentence back into the machine as input. It predicts the next word. Repeat this cycle, and the erased right half restores itself. The restored sentences won't be identical to what your senior originally wrote, but the results will follow the logic of the experiment plausibly.

This is precisely the fundamental operating principle behind the ChatGPT, Claude, and Gemini that you use every day. The conversations we have with these machines actually work this way. Following the user's question, the AI simply predicts, one word (more precisely, one token) at a time, what to say next, and appends them.

This perspective saves us from a cognitive illusion. An LLM is not a "smart spirit that understands your question and knows the answer." It is merely an elaborate machine that picks the statistically most plausible next word and chains them together. Hold onto this truth, and every concept that follows will connect naturally.

How Is This Different From Search?

Let's clear up a common confusion. Google Search and ChatGPT are completely different things.

Google Search finds documents that already exist somewhere on the internet and points to them with links. The answer itself lives inside those links; Google just plays the role of finding them well. It's closer to a database lookup. In our lab-notebook analogy, it's like finding an original notebook someone else left behind that resembles yours, and handing the whole thing over.

LLM is not a database. Instead of retrieving stored documents, it generates the answer anew each time. That's why asking the same question twice can yield slightly different answers. In the notebook analogy, the LLM doesn't hand you an original; it makes up, on the spot, sentences that would plausibly fit the erased sections.

This distinction is decisive. Search is nothing more than matching between query and document, but an LLM uses patterns from the text it has learned to produce sentences in combinations it has never seen before. This property is the source of its creativity — and, simultaneously, the source of the hallucinations we'll examine later.

A Precise Definition — A Function That Predicts the Probability of the Next Token

Now let's define it more precisely.

An LLM (Large Language Model) is a very sophisticated mathematical function that, given some text, predicts the word that comes next.

There's a subtle point here. It does not deterministically predict a single word; it computes probabilities for every possible word that could come next.

For example, if you feed the sentence "The PCR reaction produced a band that" into an LLM, it produces a probability distribution like this:

"appeared"        probability: 0.42
"was observed"    probability: 0.18
"was absent"      probability: 0.11
"faintly"         probability: 0.08
"was two"         probability: 0.05
...
"cat"             probability: 0.0000001

You pick the word with the highest probability and append it. And then? Feed the entire updated sentence back in as input, and compute the probability distribution over the next word again. Repeat this cycle, and the sentence is built one word at a time.

In CS and statistics, this is called an autoregressive model. It means feeding what you produce back into your own input. The "GT" in GPT stands for Generative Pre-trained Transformer — the GT is the autoregressive generator.

Why Does the Answer Vary a Little Each Time?

Here's an interesting point. LLMs are deterministic models. That is, given the same parameters and the same input, the probability distribution comes out exactly identical. And yet we get slightly different answers to the same question each time. Why?

Because we don't always pick just the highest-probability word — we occasionally sample a lower-probability word at random. The value that controls the degree of randomness is called temperature.

• Low temperature picks only the highest-probability word, so answers are consistent and safe. But they can be dull and repetitive. Temperature 0 is fully deterministic.
• High temperature samples low-probability words often, producing creative and varied answers. But the risk of odd answers also grows.

Inject artificial randomness into a deterministic model — this is the real reason the same question yields different answers each time. The function itself always computes the same probabilities, but randomness enters at the step where we roll the dice on those probabilities.

In CS terms, the autoregressive generation loop can be summarized as the pseudocode below:

context = user_prompt
while not stop_condition:
    prob_distribution = model.next_token_probability(context)
    next_token = sample(prob_distribution, temperature=1.0)
    context = context + next_token
    yield next_token

A practical bio tip. When you need variety over accuracy — summarizing paper abstracts, brainstorming experimental designs — raise the temperature a bit (0.7–1.0). When you need precise gene names, chemical formulas, or numbers, lower it (0.0–0.3). This is exactly why tools like Cursor and Claude Code default to low temperatures — for code, randomness is a bug.

What Exactly Is a Token?

So far I've been comfortably calling them "words," but strictly speaking, they are tokens. A token is often a smaller unit than a word.

For example, the English word unbelievable is usually split into three tokens: un, believe, able. Korean tends to be split by frequently used phrase units, but rare words may be split into syllables or even smaller pieces. 프로스타글란딘 (prostaglandin), for example, might be broken into 프로스, 타글, 란딘.

Why bother splitting them like this? Because you can't fit every word in the world into the vocabulary. If you did, the vocabulary would explode. If instead you keep only the frequently used fragments in the vocabulary, even rare words can be expressed as combinations of those fragments. This approach is called subword tokenization. Algorithms like BPE (Byte-Pair Encoding), WordPiece, and SentencePiece do this job.

Practical implications of tokens. The pricing of GPT, Claude, and Gemini is usually calculated by input tokens + output tokens. Korean produces roughly 1.5–2x more tokens than English for the same meaning. In other words, the same sentence written in Korean can cost more.

A biology-specific note. When you feed DNA or protein sequences into an LLM, tokenization behaves oddly. Sequences like ATGCTA are unlikely to have exact fragments in the vocabulary, so they usually get split into individual characters. This means processing long sequences causes token counts to explode. Sequence-specific models (e.g., ESM, ProGen) use their own tokenization schemes.

How Does It Predict So Well? — The Scale of Pre-training

Here's where the truly astonishing part begins. LLMs are trained on enormous amounts of text data collected from the internet. How much, exactly?

GPT-3's training data is said to be roughly 400 billion tokens (about 300 billion words). Here's a thought experiment to grasp the scale. Assume you read all 1 million books held in a Korean four-year university library. If the average book has 300 pages and each page 300 words, the entire library totals about 90 billion words. That makes GPT-3's training data worth 3–4 such university libraries.

And it doesn't stop there. Recent frontier models like GPT-4, Claude, and Gemini are trained on far more. Imagine: all the books in every four-year university library in Korea + all of Wikipedia + active repositories from open-source code hosts (GitHub) + a substantial fraction of academic paper databases (arXiv, PubMed, and so on) — that's roughly the reality of pre-training data.

How is this actually learned? By adjusting the countless dials inside the model. These dials — called parameters or weights — determine what a language model outputs. Change the dials, and how the model predicts the next word changes with them.

The "Large" in Large Language Model refers to the count of these parameters. These are neural networks with tens of billions to hundreds of billions of parameters. GPT-3 has about 175 billion; the latest frontier models have far more.

There's no way a human sets all these parameter values by hand. They start as randomly initialized values and, through repeated training, get tuned toward values that make increasingly plausible predictions.

Training works as follows.

1. Prepare a text snippet from the training data, ranging from a few words to thousands of words.
2. Feed the model everything but the last word, and see how the model predicts that last word.
3. Adjust the model's parameters ever so slightly so the prediction moves closer to the correct answer.
4. This adjustment uses an algorithm called backpropagation to steadily converge toward the correct answer.

Repeat this cycle countless times, and the model doesn't just predict the next word well on the training data — it can also make plausible predictions on sentences it has never seen. This is called generalization.

How Much Computation is Needed? — Gauging Scale with a Biology Experiment

Processing this immense number of parameters and data points requires an unimaginable amount of computation. Let's try to grasp the scale with something familiar to biology researchers.

The cell cycle of a mammalian cell is approximately 24 hours. This means that one cell takes a day to divide. The total amount of computation required to train GPT-3 is approximately 3 × 10^23 FLOPS (floating-point operations). If we analogize one cell division to one computation, then the amount of computation needed to train GPT-3 is equivalent to 3 × 10^23 cell divisions.

This is a huge number, and it's hard to wrap your head around it. The number of cells in an adult human body is approximately 30 trillion (3 × 10^13). That means that even if all the cells in your body divide simultaneously once, it would only amount to 3 × 10^13 computations. Training GPT-3 requires 10 billion times more computation than that. It's a scale that would be achieved if the entire body went through a division event once a day for 10 billion days. 10 billion days = approximately 27 million years.

To actually handle this computation, we need parallel processing and specialized computer chips, namely GPUs. Today, training the most advanced models involves thousands to tens of thousands of GPUs running in parallel for months. A single top-of-the-line AI accelerator like the NVIDIA H100 can perform approximately 4,000 trillion operations per second (4 × 10^15 FLOPS). Running 10,000 of these GPUs for 3 months would result in approximately 3 × 10^25 FLOPS, which is close to the training scale of GPT-4.

This is why there are only a handful of companies in the world that have this infrastructure: OpenAI, Anthropic, Google, Meta, and others. The training cost alone is estimated to be in the hundreds of billions of won.

Pre-training Alone Isn't Enough — Fine-tuning and Directional Evolution

A model that has only been pre-trained is essentially just a machine that strings together internet text. It's not yet a conversational partner that can answer questions. For example, if you ask it, "Tell me how to take care of a cat," a raw, pre-trained model will simply append the question, such as "Tell me how to take care of a cat. And also a dog. And also a bird." It's more like a text auto-completion engine than a conversational partner.

There are two steps that follow to transform this raw model into the conversational assistants we know, such as ChatGPT and Claude.

Step 1: Fine-tuning: The model is further trained with human-created conversational example data (question-answer pairs). In this step, the model learns the conversational format of "when a question comes in, it should be followed by an answer."

Step 2: RLHF (Reinforcement Learning from Human Feedback): Human evaluators compare multiple answers from the model and indicate their preference for which one is better. This preference data is used to tune the model through reinforcement learning. The result is a model that is tuned to predict the next word in a way that users prefer.

These two steps are collectively called alignment. This is the process of aligning the model with human expectations and safety standards.

If you translate this process into the language of biology experiments, it becomes surprisingly familiar. Think of directed evolution screening using phage display or antibody libraries. You create a large number of mutant antibodies in a test tube, and then you select the ones that bind well to the target antigen, amplify them, and then select the ones that bind even better from among them, and amplify them again... This cycle is repeated to produce an antibody with extremely high target binding affinity.

Alignment (RLHF) is conceptually exactly the same as this cycle:

• Raw model = initial antibody library (probabilistic utterances in all directions)
• Human evaluator's preference selection = "screening that selects only the ones that bind well to the target"
• Retraining the model in that direction = "amplifying the selected clones"
• Repeating this cycle = "multiple rounds of directed evolution"

The result is a model that probabilistically prioritizes responses that are aligned with human common sense, safety standards, and expectations. "Pre-training" and "alignment" are completely separate engineering steps. The process of the model becoming smarter (the construction of a genetic library that creates antibodies) and the process of taming it to be safe for humans are different steps.

This distinction is important because even with the best alignment, there will always be a zone of probability somewhere in the multi-dimensional parameter space that has not been screened out. Just like directed evolution doesn't achieve perfection in just a few rounds. Attempts to break through this remaining zone are prompt injection or jailbreak attacks.

Transformer — The Architecture That Enabled Parallel Processing

Underneath all of this is the Transformer neural network architecture.

Before 2017, most language models processed text sequentially, one word at a time. Architectures like RNNs (Recurrent Neural Networks) and LSTMs. This approach was not suitable for parallel processing because it had to finish processing one word before moving on to the next. Even with multiple GPUs, it was impossible to break the sequential flow.

In 2017, a research team at Google published a paper titled "Attention Is All You Need," which introduced a new architecture called the Transformer. This architecture processes the entire sentence in parallel from the beginning to the end, rather than reading the text sequentially. This single paper fundamentally changed the AI landscape over the next eight years.

Inside the Transformer, each word is first converted into a numerical vector. This conversion is called embedding. Because AI training can be done with continuous numerical data, encoding language into numbers.

Each numerical vector contains the meaning and context of the word. In the vector space, words with similar meanings are placed close to each other. From this perspective, the vectors for "cell" and "mitochondria" are close, while the vectors for "cell" and "matrix multiplication" are far apart.

The real core of the Transformer is the attention operation. Attention allows the vectors to exchange information with each other, and it allows each word to appropriately adjust its meaning according to the surrounding context.

For example, consider the ambiguous abbreviation "PC," which frequently appears in biology papers.

• In the sentence "PC 12 cells were cultured," PC is interpreted as pheochromocytoma cells.
• In the sentence "The clusters were clear in the principal component analysis (PC1, PC2)," PC is Principal Component.
• It is also used as "positive control."

The vector location moves depending on the surrounding context, even though it is the same "PC." This is the power of attention. It's the ability to understand with context, which humans do, and the model learns this ability with parameters.

In addition to attention, the Transformer also includes an operation called a feed-forward network. This part is known to serve as the model's knowledge repository. This is where facts like "the spike protein of SARS-CoV-2 binds to the ACE2 receptor" are compressed and stored as a specific combination of parameters.

By repeating the attention and feed-forward operations in multiple layers, each word vector becomes increasingly contextually aware. It compresses and compresses the information needed to accurately predict the next word and transmits it to the final stage.

In the final stage, a vector that reflects the entire context is used to predict the probability distribution of the next word. You can think of this as the model predicting the probability for all possible words that could come next.

The structure of the model is designed by researchers, but what the model actually outputs is determined by the billions of parameters that are automatically adjusted through training. This is why it is difficult to accurately explain after the fact why the model gave a particular answer. This leads to the interpretability problem, which we will discuss later.

Emergent Abilities — Abilities That Suddenly Appear at Large Scales

Here is an interesting observation. Train models with 1 million, 100 million, 1 billion, and 10 billion parameters. Most abilities improve gradually in proportion to the scale.

However, when you exceed a certain scale, new abilities suddenly appear. Abilities such as logical reasoning, coding, multi-step calculations, following instructions, and even translating into other languages suddenly become statistically significant beyond a certain scale. This is called emergent abilities.

Biology researchers are familiar with the analogy of quorum sensing. Individual bacteria do not exhibit any special behavior, but when the density exceeds a certain threshold, they suddenly begin to exhibit collective behavior such as biofilm formation or bioluminescence. The ability of individual bacteria remains the same, but the entire system undergoes a phase transition when it crosses a certain threshold density.

The emergence of LLMs is conceptually similar. When the parameter scale exceeds a certain threshold, previously non-existent abilities appear. However, there is one truth that you should not be lulled into by this magic: in the end, all of these abilities are based on trillions of matrix multiplications and additions that we are learning about behind the scenes. It's not mysterious. It's just a sophisticated and vast probabilistic prediction machine, and because its scale is beyond our comprehension, the results seem magical.

But Why Does It Hallucinate?

LLMs sometimes provide completely incorrect answers, as if they "know" them for sure.

• They cite non-existent papers with plausible authors and publication years.
• They create Python functions or R packages that don't exist.
• They invent gene names or drug names that are not real.
• They confidently present experimental protocols that are completely different from actual results.

This is called hallucination.

Understanding the underlying principles makes it clear why this happens. LLMs do not have a step to verify "whether this is true". They simply generate the next token based on the learned probabilities. They also generate plausible combinations, even for things that are not in the training data.

It may be more likely to generate plausible nonsense than to answer "I don't know" – this is the root of hallucination.

Hallucinations are more frequent in the following situations:

• Rare knowledge: Topics that appear less frequently in the training data (e.g., a specific subtype of a rare tumor)
• Latest information: Events that occurred after the training data was collected (if the training was up to 2024, then a paper published in 2026)
• Precise citations: Specific papers, pages, or numbers
• Long chains of reasoning: Logical steps that involve multiple stages
• Ambiguous questions: Prompts that can be interpreted in multiple ways

It is impossible to completely eliminate hallucinations. It is inherent in the fact that LLMs are based on probabilistic prediction. However, there are several ways to mitigate it.

• RAG (Retrieval-Augmented Generation): Search for reliable documents before answering and include them as context → see #9
• Low temperature: Suppress the selection of low-probability words
• Explicit requests for uncertainty: Explicitly state in the prompt that the model should answer "I don't know" if it doesn't know.
• Mandatory citations: "Must include citation sources."
• Fact-checking pipeline: A separate verification step

Reinforcement Learning from Human Feedback (RLHF) also contributes to reducing hallucinations. If human evaluators give high scores to responses that answer "I don't know" during the iterative cycle of goal-directed evolution, the model will gradually be biased in that direction. However, as mentioned earlier, there is a mathematically inevitable area of probability somewhere in the multi-dimensional space that did not pass the screening, so perfect defense is impossible.

Practical Applications for Biology Researchers: 3 Scenarios

How can the theories discussed so far be used in a biology research lab? Here are three real-world scenarios.

Scenario 1: Summarizing Papers and Brainstorming

Select 10 related paper abstracts from PubMed and paste them into the prompt, then ask:

"My experiment is observing metabolic changes in cell lines where a specific tumor suppressor gene is knocked out using CRISPR-Cas9. Identify three points of contact between the papers above and my experiment, and suggest five ideas for follow-up experiments."

This task, which would take tens of minutes to a few hours to research and organize, can be completed in a few minutes. However, it is essential to re-verify the citations with the original source. This is because the LLM may cite non-existent papers or distort the conclusions of existing papers due to hallucinations.

In this case, it is best to set a slightly higher temperature (around 0.7). The goal is to brainstorm from various perspectives, so a more creative response will be helpful.

Scenario 2: Biopython Code Review and Debugging

You have written a script to parse a FASTA file and calculate the GC content, but the result always comes out as 0. Paste the entire code and ask:

"Find the reason why the GC content is always 0 in this code. Also, improve it to a version that is memory-safe for large files (several GB)."

The LLM usually provides an accurate diagnosis (e.g., not handling case, only counting "G" and missing "C", string slicing resulting in O(n^2) complexity, etc.). It also suggests improved code. The user must verify the code with test data.

In this case, set a lower temperature (around 0.2). Randomness is a bug in code.

Scenario 3: Automating Data Organization

You received an Excel file from a collaborating hospital that is full of merged cells, color coding, formula errors, and typos. Show the LLM a few rows and ask:

"Create a Python code to organize this into a pandas DataFrame. Fill merged cells with the top value, extract color information into a separate column, and normalize frequently occurring typos using regular expressions."

This task, which would take several hours in practice, can be completed in 5 minutes.

All three scenarios accurately leverage the LLM's ability to "generate answers". These are tasks that cannot be done by searching. In the subsequent parts of this series, you will learn prompt engineering and tool workflows to extract more sophisticated results from each scenario.

Looking at It Again from the Perspective of Goal-Directed Evolution

Earlier, we compared RLHF to goal-directed evolution screening. If we broaden this perspective, the entire process of LLM training becomes a familiar concept for biology researchers.

Pre-training = Collecting the Genetic Diversity of Nature. The process of collecting vast amounts of text from the internet is structurally similar to collecting the gene sequences of all species in nature and creating a large database. The goal is not to be interested in the specific origin of individual sentences, but to understand the statistical patterns of the entire distribution.

Fine-tuning = Narrowing Down to a Specific Antibody Library. Fine-tuning, which refines a pre-trained model into a specific conversational format, is conceptually similar to narrowing down a general antibody library to create a subsequent library containing only target areas of interest.

RLHF = Multiple Rounds of Goal-Directed Evolution Screening. RLHF, which strengthens the model in the direction of what humans prefer by selecting the best answers from several answers, is exactly the same concept as goal-directed evolution, which creates multiple variant antibodies in a test tube and selects and amplifies those with the strongest target binding.

Understanding this mapping opens up an interesting insight. The "selection → amplification" cycle, which is familiar to biology researchers, is also at work in the alignment stage of LLM training. This perspective will be further elaborated in subsequent parts, especially #4 (backpropagation) and #11 (alignment and hallucinations).

Key Takeaways

Let's summarize it.

• LLMs are autoregressive models that predict the probability distribution of the next token.
• They are not search engines, but generators that create new content each time, which makes them creative, but also prone to hallucinations.
• Pre-training (compressing knowledge patterns with the equivalent of 3-4 university libraries), and fine-tuning and RLHF (goal-directed evolution) to align with conversational format and safety criteria.
• At the bottom, there is a parallel processing architecture of transformers + attention + feedforward.
• Beyond a certain scale, emergent abilities suddenly appear (analogy to the "critical mass" detection), but at its core, it is still probabilistic prediction.
• Always remember the fact that it does not have the ability to determine "whether it really knows this".

📐 Appendix — Mathematical Formulas for Experts

Difficulty: Very Hard Target Audience: Readers who already have a graduate-level understanding of probability, calculus, and linear algebra.

This section provides a rigorous mathematical summary of the concepts discussed in the main text, which focuses on intuition and illustrations. Feel free to skip this appendix if you are new to these concepts. It serves as a reference point to which you can return when reading papers or implementing models.

A.1 Autoregressive Factorization

LLMs model the joint probability of an entire text by decomposing it into a product of conditional probabilities.

P(w_1, w_2, ..., w_n)
  = P(w_1) · P(w_2 | w_1) · P(w_3 | w_1, w_2) · ... · P(w_n | w_1, ..., w_{n-1})
  = ∏_{t=1}^{n} P(w_t | w_{<t})

Here, w_t is the token at time step t, and w_{<t} = (w_1, ..., w_{t-1}) represents all tokens up to that point. The model approximates this conditional probability P(w_t | w_{<t}; θ) using a neural network with parameters θ (hundreds of billions to trillions).

A.2 Softmax with Temperature

The output of the model's last layer is a set of real-valued scores (logits) for each token in the vocabulary. To convert these scores into a probability distribution, we use Softmax with temperature.

P(w_t = i | context) = exp(z_i / T) / Σ_j exp(z_j / T)

• z_i is the logit for token i.
• T is the temperature.
• T → 0: The probability of the highest logit converges to 1 (equivalent to argmax, deterministic).
• T → ∞: The distribution converges to a uniform distribution (completely random).
• T = 1: Standard softmax.

Numerical Stability Tip: In practice, subtract the maximum value, such as exp(z_i / T - max(z) / T), to prevent overflow.

A.3 Cross-Entropy Loss — Objective Function for Pre-training

In a single step, if the correct token is y ∈ {1, ..., V} (vocabulary size V), the cross-entropy loss with the model's predicted distribution p̂ = softmax(z) is:

L(θ) = -log p̂_y = -log( exp(z_y) / Σ_j exp(z_j) )

This is the negative logarithm of the probability assigned to the correct token. The loss is 0 when the model assigns a probability of 1 to the correct answer, and it increases as the probability decreases.

The total loss for the entire dataset is the average of this value over all time steps and documents:

L_total(θ) = -1/N · Σ_{document} Σ_{time step t} log P(w_t | w_{<t}; θ)

Minimizing this loss with respect to the parameters θ is the mathematical goal of pre-training. Optimization is performed using stochastic gradient descent (SGD) or its variants (Adam, AdamW, etc.).

A.4 Information-Theoretic Interpretation — Perplexity

The exponential of the cross-entropy loss is called perplexity.

Perplexity = exp(L_total)

This is a measure of "how many candidate tokens the model is, on average, narrowing down to for the next token". A lower perplexity indicates that the model is predicting more confidently. The perplexity of a random predictor (a uniform distribution over vocabulary size V) is exactly V, and a well-trained large model has a perplexity of around 10-30 in natural language.

A.5 RLHF Preference Optimization — Bradley-Terry Model

In RLHF, if a human evaluator compares two responses y_w (winning, preferred) and y_l (losing, less preferred), we train the reward model r_φ(x, y) to predict the following probability well:

P(y_w ≻ y_l | x) = σ( r_φ(x, y_w) - r_φ(x, y_l) )

Here, σ(z) = 1 / (1 + exp(-z)) is the sigmoid function. This is in the form of a Bradley-Terry model, and maximizing the log-likelihood is the goal of preference learning.

L_reward(φ) = -E_{(x, y_w, y_l) ~ D} [ log σ( r_φ(x, y_w) - r_φ(x, y_l) ) ]

Subsequently, the policy model (i.e., the LLM) is fine-tuned using reinforcement learning (e.g., PPO) to maximize the reward model r_φ. At this time, the KL divergence from the original pre-training model is added as a penalty to prevent it from deviating too much.

J(π_θ) = E_{x~D, y~π_θ(·|x)} [ r_φ(x, y) - β · KL( π_θ(·|x) || π_ref(·|x) ) ]

• π_θ: The policy model being trained.
• π_ref: The reference model (pre-trained or SFT model).
• β: The KL penalty coefficient (typically 0.01 to 0.1).

If the balance of this term is incorrect, there are two failure modes. If β is too large, the model will not be able to escape the pre-training state and will not be properly aligned. If β is too small, reward hacking or model collapse may occur.

If we revisit this dilemma with a biological analogy, if β is too large, it will not be able to escape the initial library, and the screening effect will be absent. If β is too small, the screening pressure will be too strong, and diversity will be completely lost, converging to only a few clones—diversity collapse.

A.6 Recent Alternatives — DPO (Direct Preference Optimization)

Since 2023, DPO has emerged, which does not use a separate reward model but directly optimizes the policy model from preference data.

L_DPO(θ) = -E_{(x, y_w, y_l) ~ D} [ log σ( β · log(π_θ(y_w|x) / π_ref(y_w|x))
                                          - β · log(π_θ(y_l|x) / π_ref(y_l|x)) ) ]

This formulation is stable and computationally efficient, allowing DPO to be adopted in several open-source models without PPO.

A.7 Transformer Attention Formula (Scaled Dot-Product Attention)

The self-attention in a Transformer is defined by the following formula:

Attention(Q, K, V) = softmax( Q · K^T / sqrt(d_k) ) · V

• Q ∈ R^{n×d_k}: Query matrix.
• K ∈ R^{n×d_k}: Key matrix.
• V ∈ R^{n×d_v}: Value matrix.
• d_k: Dimension of the key and query (typically 64 or 128).
• sqrt(d_k): Scaling constant (prevents the inner product from exploding).

For each position, the query calculates the dot product with all positions' keys, normalizes it with softmax, and then takes the weighted sum of the value matrix. This operation automatically learns from the data which positions are related to which and to what extent.

Multi-Head Attention performs this operation in parallel with multiple pairs of (Q, K, V) projections and then concatenates the results.

MultiHead(X) = Concat(head_1, ..., head_h) · W_O
where head_i = Attention(X · W_Q^i, X · W_K^i, X · W_V^i)

This structure is the key to the Transformer's parallel processing capabilities and the source of the power of today's LLMs. Detailed visual intuition on this part can be found in episode #6 attention-mechanism.

A.8 Reference — Sense of Scale of Training Loss

In a large pre-training dataset (e.g., terabytes of web text), the model calculates the loss for thousands to tens of thousands of tokens in parallel for each batch. A single step's loss is the average cross-entropy over this batch, and this is repeated for millions of steps (over several months) until the model converges. The total computational cost of training a top-tier model is approximately 3 × 10^24 to 3 × 10^25 FLOPS.

Comparing this value to the "10 billion times the division of all cells in the body" we saw earlier, you can get a sense of how extreme the actual training is in terms of parallel processing infrastructure.

References

All content, scenarios, analogies, and figures in this episode are BioPlayground's original creations, and the following are external references that may be helpful for learning the concepts.

• Transformer original paper: Vaswani et al., "Attention Is All You Need" (NeurIPS 2017)
• GPT-3 paper: Brown et al., "Language Models are Few-Shot Learners" (NeurIPS 2020)
• RLHF principles: Christiano et al., "Deep Reinforcement Learning from Human Preferences" (NeurIPS 2017)
• DPO paper: Rafailov et al., "Direct Preference Optimization" (NeurIPS 2023)
• Deep learning visualization education: 3Blue1Brown "Deep Learning" series (YouTube) — for pedagogical reference.
• Standard textbook for natural language processing: Jurafsky & Martin, "Speech and Language Processing" (publicly available online).

This episode is a guide and a starting point. In the following 14 episodes, we will delve into each concept one by one, and return to this guide in the final episode.

Next Concept

This first episode is about the role of maps. We will delve into each one in the following episodes.

Phase 1 — Principles

• Ep. #2 neural-network-basics — What is a neural network? A static structure made of dials and switches.
• Ep. #3 how-nn-learns — How does it learn? Gradient descent from the perspective of free energy minimization.
• Ep. #4 backpropagation-intuition — The democratic tug-of-war of errors. Backpropagation intuition.
• Ep. #5 transformer-and-embedding — The moment tokens are placed in a semantic space.
• Ep. #6 attention-mechanism — How to distinguish the various meanings of "PC."

Phase 2 — Application

• Ep. #7 prompt-engineering-basics — Guiding probability in the desired direction.
• Ep. #8 structured-output — Using it like a function in JSON mode.
• Ep. #9 context-window-and-rag — How much can it hold, and RAG.
• Ep. #10 embedding-and-vector-db — Searching with vectors.
• Ep. #11 hallucination-and-alignment — The mathematical inevitability of hallucinations and countermeasures.

Phase 3 — Tools & Workflow

• Ep. #12 cursor-and-claude-code — Vibe coding in practice.
• Ep. #13 mcp-and-agent — MCP and AI Agent.
• Ep. #14 local-llm — Serving personal models with Ollama.

Phase 4 — Bio Integration

• Ep. #15 bio-ai-integration — A bridge to DevBench in practice.