Attention Mechanism: Contextual Judgement of the Tumor Microenvironment

After completing this topic

We will delve into attention, the heart of the Transformer, which we marked in section #5 and put aside for later. Why is it necessary? What does it do? And how is the 128K token context that modern LLMs handle calculated?

This section is the culmination of the story. The components from sections #1-5 – next word prediction, 33.62 million dials, gradient descent, backpropagation, embedding, residual connections, and normalization – come together with attention to complete the Transformer. Prompt engineering, RAG, and agents, which will be covered from section #7, also have their roots in the properties of attention.

Back to the "liver" problem in section #1

Let's recall the polysemy example from section #1.

• "My liver hurts" – a body organ (肝)
• "The seasoning is just right" – the degree of saltiness
• "Three years have passed" – the passage of time

As we saw in section #5, the embedding of the word "liver" is a fixed, single vector. However, this vector is used with different meanings in the three sentences above. How does a trained neural network distinguish between these three cases and process them?

The method is as follows: other words in the context flow into this "liver" representation, transforming it into a contextually appropriate meaning. If "hurts" is present, it leans towards the body organ; if "seasoning" is present, it leans towards saltiness; if "three years" is present, it leans towards the passage of time. The component responsible for this flow is attention.

In other words, attention is the process by which the representation of a word is contextualized within the Transformer. If the embedding vector is a static, pre-existing representation, then the vector after attention is a living meaning within the sentence.

Now, let's learn how attention accomplishes this.

The failure of the naive approach: the problem of uniform averaging

The simplest way to pass information between words is to simply average all the token vectors in the sentence.

contextualized(liver) = (vec(liver) + vec(i) + vec(hurts) + vec(".")) / 4

Problem: all words are equally weighted. "Hurts" is crucial in determining the meaning of "liver", while "i" is not very important, but uniform averaging does not distinguish between them.

What we need: a method that differentially weights how much each word is reflected, based on the situation. For example, "hurts" 80%, "three years" 5%, and "i" 5% when determining the meaning of "liver".

Attention calculates this "how much to reflect". It calculates an attention weight for each pair of words (query-key), and uses this to perform a weighted average of the information.

A pathologist judging T cells in a pathology slide

A bio analogy, continuing from the pathology slide we saw in section #2.

Suppose you are a pathologist observing a single T cell in a tissue slide. You need to determine what state this T cell is in – is it activated, exhausted, or a regulatory T cell?

This judgment cannot be made by looking only at the morphology of the T cell itself. You need to look at the surrounding cellular environment (the tumor microenvironment).

• If there are many tumor cells right next to this T cell? → It is likely attacking the tumor or in a state of being neutralized.
• If there are many other T cells and dendritic cells nearby? → This is a location where the immune response is activated.
• If the surroundings are thickened by fibroblasts and a collagen matrix? → It is trapped within the stromal barrier.
• If it is close to a regulatory T cell (Treg)? → It is likely being suppressed.

The importance of each of the surrounding cells in this judgment varies. The density of tumor cells is crucial, while the thickness of the stroma is relatively less crucial. The pathologist unconsciously assigns different weights to each surrounding cell and makes a judgment.

This can be quantified to produce the exact structure of attention. Three components are needed.

• Query: What does the T cell, which is being judged, "want to know"? For example, "I am a CD8+ T cell, and I want to know if I am currently attacking the tumor or if I am exhausted."
• Key: What information can each of the surrounding cells "provide"? For example, tumor cells can provide "PD-L1 expression levels", dendritic cells can provide "antigen presentation status", and stromal cells can provide "the degree of physical barrier".
• Value: The actual content of the information being transmitted. For example, the PD-L1 expression level of the tumor cell, the activation state of the dendritic cell, and so on.

The Query of the current T cell is compared with the Key of each of the surrounding cells to score "how well they match". The Value of cells that match well is highly reflected in the judgment of this T cell, and cells that do not match are ignored.

This is the skeleton of attention.

Query, Key, Value: The three faces of a word

The attention of the Transformer also has exactly this triangular structure.

Each word vector in the sentence is transformed into three derived vectors.

• Query (Q): "What does this word want to know in the context?"
• Key (K): "How does this word advertise itself to other words?"
• Value (V): "What information does this word actually transmit?"

Each is obtained by multiplying the embedding vector of the word by three learned weight matrices, W_Q, W_K, and W_V.

Q_i = W_Q · x_i
K_i = W_K · x_i
V_i = W_V · x_i

x_i is the embedding vector of the i-th word. These three vectors are usually of smaller dimension than the original vector (e.g., original 4096 dimensions → Q/K/V each 128 dimensions).

Intuition: You can think of a word vector as having three "faces". The Query face asks other words, "Are you useful to me?" The Key face answers the questions of other words, "I have this information." The Value face transmits the actual information.

It is important that the three faces of Q, K, and V are different. The way a word advertises itself (Key) and the way it investigates other words (Query) can be different, and the information it actually transmits (Value) can also be different from these two. This separation is what allows attention to express very flexible information flow.

The formula for Scaled Dot-Product Attention

Now, let's look at the actual calculation. The attention score that word i calculates for word j:

score(i, j) = ⟨Q_i, K_j⟩ / sqrt(d_k)

This is the dot product of the two vectors. If the directions of the two vectors are similar, the score is high, and if they are different, the score is low.

The reason for dividing by sqrt(d_k). Q and K are d_k-dimensional vectors, but if the vectors are random, the magnitude of the dot product increases approximately as sqrt(d_k) as d_k becomes larger. If this division is not done, the score will explode as the vector dimension increases, and the subsequent softmax will go to extremes. This scaling is the hidden hero of Transformer training stability. The exact reason can be derived in Appendix A.2.

Now, from the perspective of word i, we have calculated the scores for all the words in the sentence. These scores are converted into a probability distribution using softmax.

α(i, j) = exp(score(i, j)) / Σ_k exp(score(i, k))

Σ_j α(i, j) = 1. Each word distributes 100% of its attention to other words.

Softmax is a function we have already encountered in Appendix A.2 of section #1 and Appendix A.4 of section #3. Here, it creates a probability distribution that represents the competition between words.

Finally, the Value vectors are weighted and averaged using this weight:

output_i = Σ_j α(i, j) · V_j

This is the attention-based representation of word i.

If written in matrix form for the entire sentence, it becomes much more concise.

Attention(Q, K, V) = softmax(Q · K^T / sqrt(d_k)) · V

This single line is the heart of the Transformer. The title of the original paper, "Attention Is All You Need," refers to this equation.

Multi-Head Attention: Judging in Parallel from Multiple Angles

Even with a single attention mechanism, information flow is possible, but in practice, we always use multiple attentions in parallel. This is called multi-head attention.

Each head has its own W_Q, W_K, and W_V, and calculates attention independently. The results are concatenated and then combined again using a single output weight matrix W_O.

head_h = Attention(Q · W_Q^h, K · W_K^h, V · W_V^h)
MultiHead = Concat(head_1, ..., head_H) · W_O

H is the number of heads. 16 in GPT-2 medium, 96 in GPT-3, and 64 in LLaMA-70B.

Why are multiple heads necessary? Each head is specialized in only one "perspective". For example, one head might focus on the "previous word," another on the "earlier noun in the sentence," and another on "matching parentheses." With multiple heads, it is possible to process various grammatical and semantic relationships simultaneously.

Bio analogy: judging multiple markers in parallel. When a pathologist judges a T cell, they do not look at only one characteristic. They simultaneously and in parallel examine CD8/CD4 markers, activation markers (CD69, HLA-DR), exhaustion markers (PD-1, TIM-3, LAG-3), and regulatory markers (FoxP3), and combine the results to make a judgment. Each marker examination corresponds to one attention head, and the final combination corresponds to the W_O combination. This parallel perspective analogy is the key to multi-head attention.

Interpretability: Studies that have examined the trained attention heads of Transformers have shown that different heads actually handle different linguistic patterns. For example, some heads in GPT-2 focus only on the "previous word," while other heads track the "previous occurrence of the same noun." Recent mechanistic interpretability research is cataloging these head-by-head functions.

Causal Masking: The Problem of Not Being Able to See the Future

LLMs are trained with next word prediction (section #1). An important constraint at this time: it should not refer to future words that have not yet been predicted.

For example, if you have seen "The cat likes fish," you should not refer to the next word, "to," when predicting the next word. Training should always predict the next word based on "what has been seen so far."

Attention is a structure that can, in principle, refer to all words in the sentence. Therefore, if it is necessary to prevent it from seeing the future, it must be explicitly blocked. This is called causal masking.

The method is as follows: set the upper triangle (locations where word i has j > i) in the score matrix to -∞. After passing through the softmax, -∞ becomes a probability of 0, so future words are ignored.

score(i, j) = ⟨Q_i, K_j⟩ / sqrt(d_k)  if j ≤ i
score(i, j) = -∞                      if j > i

This attention with masking is called causal attention or decoder attention. GPT-like models use this. Attention without masking is called bidirectional attention. BERT-like models use this. Most LLMs are causal.

Bio analogy: Subtle, but a possible analogy. In developmental biology, when a cell makes a differentiation decision, it cannot use information from downstream cells that have not yet developed. It only reflects the signals from upstream cells in the temporal order. Causal masking forces this temporal causality.

Self-Attention vs. Cross-Attention

Up until now, we've been discussing attention mechanisms where Q, K, and V all come from the same set of tokens within a sentence. This is called self-attention. It's a structure where tokens draw information from within the sentence itself.

Cross-attention is where Q comes from one set, and K and V come from another set. For example, in a translation model, when the target language tokens refer to the source language tokens. Or, in image captioning, when the text tokens refer to the image patches. This is a fundamental component of multimodal models.

• Self-attention: Q, K, and V all come from the same sequence. Most LLMs use this.
• Cross-attention: Q comes from one sequence, and K and V come from another sequence. Used in multimodal models and translation models.

The encoder-decoder architecture of the original Transformer paper uses both self-attention and cross-attention. Pure decoder LLMs (like the GPT family) use only self-attention. Recently, cross-attention has become prominent again in vision-language models (like GPT-4V and Claude Vision).

KV Cache — Speeding Up Inference by 100x

When we move from training to inference, an interesting optimization technique emerges.

When an LLM generates text, it proceeds as follows:

• Input the prompt: "The cat".
• The model predicts the next token: "eats".
• The prompt becomes: "The cat eats". This is fed back into the model.
• The next prediction: "fish".
• The prompt becomes: "The cat eats fish". This is fed back into the model.
• ...

If we recalculate the entire prompt at each step, the computational cost at step t is O(t^2). Generating 100 tokens would require 10,000 units of computation.

However, the K and V values calculated in the previous step are the same in the next step. Thanks to causal masking, the Q, K, and V values of past tokens do not change even when new tokens are added. So, we can cache them.

This is the KV cache. The K and V vectors of each layer are stored in GPU memory, and at each step, only the Q value of the new token is calculated and used with the cached K and V values in the attention mechanism. This reduces the computational cost per step to O(t). Generating 100 tokens now takes 1,000 units of computation, a 100x speedup.

The tradeoff. It consumes a lot of GPU memory. 96 layers × 96 heads × 12288 dimensions × 128K context × batch size... This can easily add up to hundreds of GB. This is why large models with long context windows are limited by GPU memory.

This KV cache optimization is a crucial reason why ChatGPT and Claude can have real-time conversations today. Without the cache, responses would take tens of seconds.

GQA and MQA — Evolving to Reduce KV Cache Size

Because the KV cache is so large, new ideas have emerged. One is to have multiple Query heads share a single Key/Value head.

• Multi-Head Attention (MHA): Q, K, and V each have H heads. This is the basic form.
• Multi-Query Attention (MQA): Q has H heads, while K and V each have 1 head. This is an extreme approach, reducing the KV cache size by a factor of H.
• Grouped-Query Attention (GQA): An intermediate approach. Q has H heads, while K and V each have G heads (G < H). For example, if H=32 and G=8, then 4 Q heads share 1 K/V head.

MQA maximizes cache reduction but can lead to some loss of quality. GQA is a compromise that provides significant cache reduction while maintaining almost the same quality. The latest models, such as LLaMA-2, LLaMA-3, Qwen, and Mistral, use GQA. This is now the standard.

Flash Attention — Breaking Through the Memory Wall with Implementation

The bottleneck in attention computation is not actually the amount of computation, but rather the GPU memory bandwidth. Q · K^T is an n × n matrix (where n is the sequence length), so a context of 128K requires a 128000 × 128000 matrix. The time it takes to transfer this large matrix back and forth between HBM (GPU memory) and SRAM (on-chip cache) is much longer than the actual computation.

Flash Attention (Dao et al., 2022) is an implementation that minimizes this back-and-forth transfer. It divides the large attention matrix into blocks and processes them within the SRAM, without materializing the large matrix in HBM. The mathematical result is the same, but it is 2 to 10 times faster and reduces memory usage from O(n²) to O(n).

Flash Attention is a key reason why context windows of 128K, 200K, and 1M tokens have become practical. Today, almost all large models use Flash Attention for both training and inference. It continues to be improved with Flash Attention 2 and Flash Attention 3.

The key point. It's not just about algorithmic theory, but about hardware-aware implementation optimizations that drive the expansion of LLMs today. This highlights the importance of deep learning systems engineering.

Beyond Attention — To the Next Token Prediction

Let's recap the components we've covered. The attention block contextualizes the token vectors, and then the FFN (Feedforward Network), discussed in #5, further processes this contextualized representation. After passing through several blocks, the language modeling head (a matrix of size d × V similar to an embedding) generates the probability distribution for the next token.

During training, the cross-entropy between this distribution and the actual next token is used as the loss. The backpropagation in #3 and #4 propagates this loss to the attention parameters, enabling learning. As a result, attention naturally learns to be the "most useful way to contextualize language".

Interpreting the attention matrix. Visualizing the attention scores α(i, j) of a trained model reveals specific language patterns. For example:

• Previous token head: Scores are concentrated just below the diagonal. Grammatical adjacency.
• Previous noun reference head: Scores point from pronouns to previous nouns. Coreference resolution.
• Sentence ending head: Scores point from the end-of-sentence token to the beginning of the sentence. Summarizing sentence structure.

This visualization is the starting point for mechanistic interpretability research. Anthropic, OpenAI, and Google are using attention pattern analysis to reverse engineer the internal workings of LLMs.

Bio Application Scenarios

Scenario 1 — ESM's Amino Acid Contact Prediction

The attention mechanism in the ESM protein language model, mentioned in #4 and #5, exhibits remarkable properties. When the attention matrix of a trained ESM is examined, it is found that the attention is naturally strongly focused on pairs of amino acids that are in physical contact in the 3D structure.

This means that the model automatically learns to predict 3D contacts even without being explicitly trained to "predict the structure," but only being trained to predict the next amino acid. This attention map was used as a contact prediction tool in the early stages of AlphaFold.

This is a case that demonstrates the general power of attention. The principle of "paying attention to relevant information" applies not only to language but also to biological sequences.

Scenario 2 — Enformer's Enhancer-Promoter Linkage

The attention mechanism in Enformer (genomic Transformer), mentioned in #5, links promoters (gene start sites) with enhancers (regulatory elements) that are tens of kb apart within a 100 kb context window. After training, some of the attention heads automatically learn to identify enhancer-promoter pairs.

Traditionally, enhancer-promoter matching requires 3D genome contact experiments (Hi-C), but Enformer can predict this relationship based only on the sequence. This is due to attention's "ability to learn long-range interactions".

Scenario 3 — Cell Context Judgment in Pathology Images

This is a practical implementation of the scenarios in #2 and #6. A Vision Transformer (ViT)-based pathology model calculates attention for each patch (small square region) in the slide to determine the tumor microenvironment. Each T cell patch establishes an attention relationship with surrounding tumor cell and stromal cell patches, and this is combined to predict the T cell state.

Models in this family (e.g., HIPT, CTransPath) are beginning to achieve pathologist-level performance in precision tumor diagnosis and prognosis prediction.

Key Takeaways

• Attention is the process of contextualizing token representations. This is where the "bank" polysemy processing from #1 takes place.
• Each token has three faces: Q, K, and V. The dot product of Q and K calculates the "attention" between tokens, which is then normalized using softmax, and the information is passed on as a weighted average of V.
• The formula: Attention(Q, K, V) = softmax(Q · K^T / sqrt(d)) · V.
• Multi-head attention processes information in parallel from multiple perspectives. This is analogous to a pathologist's multi-marker panel analysis.
• Causal masking prevents information from the future from being used. This is essential for LLM training.
• KV cache speeds up inference by 100x. GQA reduces the cache size.
• Flash Attention makes context windows of 128K+ practical. Hardware-aware implementation drives the expansion of LLMs today.
• Attention is a general principle that applies not only to language but also to proteins, genomes, and images. The idea of "paying attention to relevant information" can be reused across domains.

📐 Appendix — Mathematical Formulas for Experts

Difficulty: Very Hard Target Audience: Readers with a graduate-level understanding of linear algebra, probability, and numerical analysis.

A.1 Scaled Dot-Product Attention: Complete Formula

Sequence length n, embedding dimension d, Q/K dimension d_k, V dimension d_v.

Input:

• Q ∈ ℝ^{n × d_k} (Query)
• K ∈ ℝ^{n × d_k} (Key)
• V ∈ ℝ^{n × d_v} (Value)

Output:

Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V

Softmax applied row-wise:

softmax(M)_{ij} = exp(M_{ij}) / Σ_k exp(M_{ik})

Each row is a probability distribution (sum is 1).

Final output shape: ℝ^{n × d_v}. Each row is the contextually-aware representation of the corresponding word.

A.2 Derivation of Why to Divide by sqrt(d_k)

Assume that each element of Q and K is a random variable with mean 0 and variance 1.

Dot product of Q_i and K_j:

⟨Q_i, K_j⟩ = Σ_{k=1}^{d_k} Q_{ik} K_{jk}

If each term Q_{ik} K_{jk} is independent, it has mean 0 and variance 1. The variance of the sum is:

Var(⟨Q_i, K_j⟩) = d_k
Std(⟨Q_i, K_j⟩) = sqrt(d_k)

Thus, as d_k gets larger, the magnitude of the dot product also grows by a factor of sqrt(d_k). If d_k = 128, the standard deviation is approximately 11.3.

If we directly input this into the softmax, the large values will dominate, and the softmax output will become close to a one-hot distribution. The softmax gradient will be virtually 0, preventing training.

Dividing by sqrt(d_k) keeps the standard deviation close to 1, allowing the softmax to output a smoother distribution. This enables gradients to flow and training to proceed.

A.3 Partial Derivative of Softmax

When p = softmax(z):

∂p_i/∂z_j = p_i · (δ_{ij} - p_j)

δ_{ij} is the Kronecker delta (1 if i=j, 0 otherwise).

Combined with Cross-Entropy Loss:

Given the ground truth y (one-hot), the loss is L = -Σ y_i log p_i.

Using the chain rule:

∂L/∂z_j = p_j - y_j

This is a beautiful result, as seen in section A.4. The gradient of attention training also repeatedly utilizes this principle.

A.4 Multi-Head Attention: Formula

H heads. For each head h:

Q_h = X · W_Q^h,  K_h = X · W_K^h,  V_h = X · W_V^h
head_h = Attention(Q_h, K_h, V_h)

W_Q^h, W_K^h ∈ ℝ^{d × d_k}, W_V^h ∈ ℝ^{d × d_v}. Typically, d_k = d_v = d/H.

Concatenation:

MultiHead(X) = Concat(head_1, ..., head_H) · W_O

W_O ∈ ℝ^{H · d_v × d}.

Number of parameters:

• Each of Q, K, V projections has d × d. So, 3 d^2.
• Output projection has d^2.
• Total: 4 d^2 per block.

This is consistent with the calculation in section A.10.

A.5 Implementation of Causal Mask

Mask matrix:

M_{ij} = 0        if j ≤ i
M_{ij} = -∞       if j > i

In the attention calculation:

Attention_masked(Q, K, V) = softmax((Q K^T / sqrt(d_k)) + M) V

-∞ in the softmax becomes exp(-∞) = 0, assigning probability 0 to future positions.

In practice, use a large negative number (e.g., -1e9) instead of -∞ to ensure numerical stability.

A.6 KV Cache Algorithm

During inference:

• Step 1: Calculate Q_1, K_1, V_1. Store [K_1], [V_1] in the cache.
• Step 2: Calculate Q_2, K_2, V_2 for the new word. Append to the cache: [K_1, K_2], [V_1, V_2].
• Step t: Calculate only the new word's Q_t. Perform attention using the cached K and V.

Memory:

• Number of heads per layer: H, head dimension: d_h, context length: n, batch size: B
• KV cache for one layer: 2 · B · n · H · d_h · bytes_per_value

Example: LLaMA-70B, H=64, d_h=128, n=128K, B=1, fp16:

2 · 1 · 128000 · 64 · 128 · 2 = 4.19 GB per layer

With 80 layers, this is 335 GB. This is why large models with long contexts are limited by memory.

A.7 GQA (Grouped-Query Attention)

• Number of Q heads: H_Q
• Number of K, V heads: H_KV
• Group size: g = H_Q / H_KV

Q heads within the same group share a single K and V head:

head_i = Attention(Q_i, K_{i // g}, V_{i // g})

KV cache size is reduced by a factor of g. If g = H_Q (all Q share one KV), it becomes MQA.

LLaMA-2-70B: H_Q = 64, H_KV = 8, g = 8. The cache is saved 8 times.

A.8 Flash Attention Block Algorithm

Key idea: Calculate the large n × n attention matrix block-by-block using tiling, without materializing the entire matrix in SRAM.

Online Softmax: An algorithm for streaming the softmax calculation block-by-block. Each block maintains the partial maximum and partial exponential sum, which are then combined.

Block size B_r (rows) × B_c (columns):

• Load Q in chunks of B_r.
• Iterate through K and V in chunks of B_c.
• For each (Q_block, K_block), calculate the partial scores and partial exponential sums.
• Combine with the results from the previous block (online softmax formula).

Results:

• Time: Theoretically, the FLOPs are the same, but the actual wall-clock time is 2-4 times faster (memory bottleneck is resolved).
• Memory: O(n²) → O(n)
• Accuracy: Identical (negligible numerical error).

A.9 Sparse Attention Series

Approximate methods for handling long contexts.

• Sliding Window Attention: Each token only references a neighborhood of w tokens. Computation is O(nw). Adopted by Mistral and Longformer.
• Longformer: Sliding window + a few global tokens.
• BigBird: Sliding + global + random attention.
• Sparse Attention (GPT-3): Attention using specific patterns (strided, factorized).

Trade-offs: Since it's an approximation, there is some loss of quality. With the success of Flash Attention, it is now less necessary. However, it is still useful for very long contexts (1M+).

A.10 Computational and Memory Complexity of Attention

Sequence length n, dimension d:

• Full Attention: Time O(n²d), memory O(n² + nd)
• Flash Attention: Time O(n²d), memory O(n · d) (only the block size is in SRAM)
• Sliding Window: Time O(nwd), memory O(nd)

When n = 128K, n² is 16 billion. Flash Attention made it practical to compute this scale of matrix. The long context capabilities of modern large language models are the result of this algorithm and system collaboration.

References

All content, scenarios, analogies, and figures in this section are developed in-house by BioPlayground. The following are external references that may be helpful for learning the concepts.

• Original Transformer paper (defining attention): Vaswani et al., "Attention Is All You Need" (NeurIPS 2017)
• Flash Attention: Dao et al., "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness" (NeurIPS 2022)
• Flash Attention 2: Dao, "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning" (2023)
• Multi-Query Attention: Shazeer, "Fast Transformer Decoding: One Write-Head is All You Need" (2019)
• Grouped-Query Attention: Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models" (EMNLP 2023)
• Interpretation of attention heads: Elhage et al., "A Mathematical Framework for Transformer Circuits" (Anthropic 2021)
• Attention head catalog: Olsson et al., "In-context Learning and Induction Heads" (Anthropic 2022)
• ESM contact prediction: Rao et al., "Transformer protein language models are unsupervised structure learners" (ICLR 2021)
• Vision Transformer: Dosovitskiy et al., "An Image is Worth 16x16 Words" (ICLR 2021)
• Deep learning visualization tutorial: 3Blue1Brown "Deep Learning" Ch 6 & 7 (YouTube) — for pedagogical reference.

This concludes Phase 1, the principles section, with section 6. From section 7 onwards, we will discuss how to actually utilize this trained transformer — prompt engineering, RAG, agents.

Next Concept

• Ep. #7 prompt-engineering — Designing prompts to effectively guide the attention of trained transformers.
• Ep. #8 rag-and-context — RAG, which expands the information that attention can handle.
• Ep. #11 hallucination-and-alignment — How failures of attention are connected to hallucinations.