RAG and Context Expansion: Adding a Lab's Knowledge Base to an LLM

After completing this topic

The document summarization assistant we created in Part 7 has a fundamental limitation: the model doesn't know about our lab's internal data. The 30 papers you've published in the last five years, the 200 protocols in the lab wiki, the 100 slides from conference presentations – none of this is included in the LLM's training data.

This section deals with RAG (Retrieval-Augmented Generation), an architecture that solves this problem. It creates an external knowledge base and allows the LLM to retrieve and inject relevant parts whenever needed to answer questions. This is a natural point to see how the embeddings from Part 5 and the attention from Part 6 are recycled in practice.

Two Locations for LLM Knowledge

In Part 1, we understood LLMs as "probability machines," and in Parts 5 and 6, we saw how these machines compress and store statistical patterns from training data in billions of parameters. If we look more closely at where this knowledge resides, there are two locations.

First location: Parameters. These are the things learned during training. The reason GPT-4 can generate the sentence "Mitochondria are the powerhouses of the cell" is because this sentence (or similar ones) was repeatedly exposed in the training data, leaving a trace in the parameters. This knowledge is built into the model itself and is referenced independently of the context.

Second location: Context window. This is the information contained in the prompt for each call. Examples include the few-shot examples in Part 7, the name of the mentor, and the text of the paper being reviewed. This knowledge is referenced only through attention. When the model predicts the next word, the attention mechanism focuses on the relevant parts of the context, and this information is reflected in the answer.

The two locations have completely different characteristics.

If we want the LLM to reference the 30 papers in our lab, we have two options:

• Option A: Retrain or fine-tune the LLM with these papers. Embed the knowledge in the parameters.
• Option B: For each call, paste the relevant papers into the context window.

Option A requires tens of GB of GPU and weeks of training time. Furthermore, it needs to be retrained every time a new paper is added. Option B allows for instant updates and no training. In practice, most approaches fall under Option B.

The problem is how to implement Option B. Putting 30 papers in the context would exceed 200K tokens. Furthermore, if it expands to 100 or 1,000 papers, no context model with 200K tokens can handle it.

This is where RAG comes in: find only the necessary parts and put them in. For each question, search for a few relevant papers or paragraphs and inject them into the context.

The Framework of a RAG Pipeline

The overall flow of RAG.

[Preparation Phase - One time only]
30 papers, 200 protocols, 100 slides
   ↓
Chunking (split into paragraphs or sections)
   ↓
Calculate embeddings for each chunk (embedding model from Part 5)
   ↓
Store in a vector database (embedding + original text)

[Query Phase - Each call]
User question: "Were there any cases in the lab where CRISPR editing efficiency was low?"
   ↓
Calculate the embedding of the question (same embedding model)
   ↓
Search for the top K chunks in the vector DB using cosine similarity
   ↓
Paste the retrieved chunks into the prompt
   ↓
Call the LLM → Generate an answer

Three components are needed: embedding model, vector DB, and LLM. The first two components handle retrieval, and the last one handles generation. It's a combination of Retrieval + Generation, hence the name RAG.

Bio analogy: Recycling the integration of cell signals. In Part 6, we saw attention as "the principle by which a cell focuses on relevant information from its surroundings." RAG extends this principle one step further. It's like adding a system that dynamically expands the neighboring cells (context) that a cell (LLM) can reference. If attention is the principle of focusing on information within the context, then RAG is the principle of reconstructing the context itself each time.

Chunking: Chunk Size Affects Retrieval Quality

If you embed the entire document, too much meaning is crammed into a single embedding, reducing retrieval accuracy. Conversely, if you split it into word-sized pieces, the embedding becomes meaningless because of the lack of context. An appropriate chunk size is required.

General guidelines:

• Chunk size: 200-800 tokens (approximately 200-1000 words)
• Overlap between chunks: 10-20% (to prevent information loss at the boundary)
• Split at paragraph or section boundaries (do not forcibly split in the middle of a sentence)

Recursive Character Splitter (LangChain/LlamaIndex standard). First split using a large separator (paragraph boundary), and if the chunk is too large, recursively split using the next separator (sentence boundary), and so on.

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""]
)
chunks = splitter.split_text(paper.text)

Bio domain-specific chunking:

• Papers: Each section (abstract, introduction, methods, results, discussion) is a separate chunk. Append the section header to the beginning of the chunk (to preserve context).
• Protocols: Each step is a separate chunk. The list of reagents is a single chunk.
• Slides: One slide = one chunk. Include presenter notes if available.
• Tables and figures: Use the caption as the header of the chunk. Keep table data in CSV or Markdown table format.

Attach metadata: Store the original paper information (author, year, journal, DOI) as metadata for each chunk. This will be used to display the source in the answer later.

Embedding Search: Recycling the Vector Space from Part 5

The embedding space we talked about in Part 5 reappears here. This time, instead of individual words, it's the embedding of the entire document chunk. This is called a document embedding or sentence embedding.

Embedding model: A neural network that takes a chunk as input and outputs a vector (typically 512-4096 dimensions). It is trained so that "semantically similar chunks have vectors close to each other, and different chunks have vectors far apart."

Major embedding models:

• OpenAI text-embedding-3: One of the practical standards. small (1536 dimensions) and large (3072 dimensions). API call.
• Cohere embed v3: Strong in multilingualism, low latency.
• Voyage AI: Top-ranked in recent benchmarks.
• Sentence-BERT series: Open source. all-MiniLM-L6-v2 (384 dimensions), all-mpnet-base-v2 (768 dimensions), etc.
• BGE/E5 series: Top open-source models. Good multilingual support.

Bio domain-specific: General embedding models are relatively weak in biological terms (gene names, protein names, abbreviations). If necessary:

• BioBERT/PubMedBERT: BERT series pre-trained on PubMed. Can be fine-tuned for sentence embedding.
• SPECTER: Scholarly embedding trained on paper citations.
• Self-fine-tuning: Fine-tune sentence-transformers with lab data. Best performance, but requires data and training.

Cosine similarity is used to measure the similarity between two vectors.

similarity(u, v) = ⟨u, v⟩ / (||u|| · ||v||)

If the two vectors have the same direction, it is 1, if they are irrelevant, it is 0, and if they are opposite, it is -1. Embeddings are typically stored normalized (norm 1), so cosine similarity is just the inner product. A reduced form of the attention score in Part 6.

Search procedure:

1. Embed the question using the embedding model to get the question vector q.
2. Calculate the cosine similarity between q and all chunk vectors in the vector DB.
3. Return the top K chunks (e.g., K=10).

Since it is slow to iterate over tens of thousands to millions of chunks for each query, an Approximate Nearest Neighbor (ANN) index is used. HNSW (Hierarchical Navigable Small World) is the standard. See Appendix A.7.

Vector Database: A Warehouse to Store Embeddings

Vector DB: An infrastructure that stores embeddings + metadata and performs fast similarity searches. Major choices:

• Chroma: Open source, local, lightweight. Good for getting started.
• Qdrant: Open source, production-grade. Supports both self-hosting and cloud.
• Weaviate: Open source, strong schema, GraphQL interface.
• Pinecone: Managed service, paid. Easy to use.
• pgvector: PostgreSQL extension. Leverages existing SQL infrastructure.
• Milvus: Large-scale, enterprise-grade.

Practical scale: 30 papers (each with 20 chunks) is 600 vectors. Chroma is sufficient for local use. 100,000 papers (2 million chunks) require production-grade tools like Qdrant or Pinecone. Start with Chroma and migrate as needed.

Storage format: For each vector:

{
  "id": "paper_2023_smith_chunk_3",
  "vector": [0.024, -0.113, ..., 0.087],  // 1536-dimensional float
  "content": "In this study, we investigated CRISPR editing efficiency in HEK293T cells...",
  "metadata": {
    "paper_id": "2023_Smith_Nature",
    "section": "Results",
    "authors": ["Smith J", "Lee H"],
    "year": 2023,
    "doi": "10.1038/..."
  }
}

## Dense + Sparse: The Power of Hybrid Search

The weakness of embedding search (dense retrieval): It's weak at **exact word matching**. If a question contains the gene name "TP53" and the document also contains "TP53", it may be mixed with other gene names in the embedding space. Dense search struggles to identify precise identifiers such as gene names, drug names, chemical formulas, and equations.

**Sparse retrieval** complements this weakness. It is a traditional information retrieval method.

**BM25 (Best Matching 25)**. It has been the standard for search engines for over 20 years. It calculates relevance based on word frequency, document length, and inverse document frequency. It is strong at exact word matching.

**Hybrid Search**. Combines dense and sparse results.

- Search for the top K results using each method.
- Combine the rankings of the two results using **Reciprocal Rank Fusion (RRF)**.

RRF_score(doc) = Σ_i 1 / (k + rank_i(doc))

`k = 60` is the default value. A higher rank in each method results in a higher score.

**Practical Guidelines**. In domains with many specialized terms and identifiers (bio, medical, legal, patents), hybrid search provides a significant improvement over dense search alone. Hybrid search generally outperforms in search performance benchmarks.

**Recent Trends**. Learned sparse embeddings, such as **ColBERT and SPLADE**, are emerging. They combine the advantages of dense and sparse methods in a single model. They are at the top of the latest benchmarks.

---

## Reranking: The Power of the Final Filter

Embedding search is fast, but it is not **perfectly accurate**. Some of the top 10 search results may not be relevant.

**Reranker**. A model that re-evaluates the K candidate results retrieved and provides a more accurate ranking. Cross-encoder models are the standard.

**Bi-encoder vs. Cross-encoder**.

- **Bi-encoder** (embedding search): Transforms the question and document into embeddings **separately** and then calculates their similarity. It is fast, but the interaction between the two texts is compressed into a single similarity score.
- **Cross-encoder** (reranker): Puts the question and document **together** into a transformer and directly calculates the relevance score. It is accurate but slow.

Cross-encoders can make much more nuanced judgments because they calculate attention between all pairs of words in the two texts (see #6). However, the computational cost is high, so it is not possible to process all documents. A two-step pipeline that first uses embedding to narrow down to 100 documents and then uses a cross-encoder to re-rank the top 10 is standard.

**Key rerankers**.

- **Cohere Rerank v3**: Managed API.
- **BGE-Reranker**: Open source, multilingual.
- **Voyage Rerank**: Top-ranked in benchmarks.

**Effect**. It is common to see the accuracy improve from 60% to 80% by adding a reranker when using only the top 10 embedding results.

---

## Limitations of RAG: Where Does It Fail?

RAG is not a panacea. Common types of failures encountered in practice.

**Limitation 1: If the search fails, the answer fails.** Even if the answer is in the vector database, the LLM cannot reference it if the search fails to find it. This can happen if the chunking is bad, the embedding model is not good for the domain, or the question is not expressed in a way that the search system can understand. → Mitigations: **query rewriting** and **multi-query search**.

**Limitation 2: Difficult to combine information from multiple documents.** If the answer spans multiple documents and requires integration. For example, "a list of cell lines used by our lab in the last 5 years" needs to be compiled by referencing the methods section of 30 papers. It is not possible to answer by looking at each document alone. → Mitigations: **agentic RAG** and **multi-hop retrieval**.

**Limitation 3: Accuracy of citations.** Even if the LLM creates an answer from the retrieved documents, it may hallucinate and invent information that is not in the documents (see #11). The existence of search results does not guarantee complete protection against this. → Mitigations: Explicitly request citations and force the display of sources.

**Limitation 4: Negative questions.** RAG struggles with questions that ask about the absence of something, such as "What cell lines have not been used in our lab?". This is because it is impossible to search for something that does not exist. → Mitigation: Invert the question.

**Limitation 5: Real-time information vs. delayed updates to the search repository.** Vector databases are not real-time, so they may miss the latest information. → Mitigations: Incremental updates and real-time indexing.

---

## RAG in the Age of Long Context: Is It Still Necessary?

In 2024-2025, models with million-token context windows (Gemini 1.5, Claude 3.5, GPT-4.1, etc.) have emerged, sparking debate. The question is, "Now that the context window is so large, is RAG still necessary?"

**Why RAG is still necessary**.

- **Cost**. Putting 1M tokens into each call will explode the cost and latency. RAG puts in only K chunks (~10K tokens), which is much cheaper and faster.
- **Accuracy**. Even with a 1M token context, the model exhibits the **"lost in the middle"** problem. It misses information in the middle. Putting in only the relevant parts with RAG actually improves accuracy.
- **Scalability**. 1M tokens can't hold 1000 papers. The scale in practice is always larger than that.
- **Auditability and Transparency**. It is possible to track which documents were used to answer the question. This is essential for regulated industries.

**Usefulness of long context**. Long context is better for specific tasks that actually require the entire context (e.g., in-depth analysis of an entire paper). RAG and long context are not substitutes, but rather **complementary**.

**Hybrid Architecture**. A common practice is to use RAG to extract 100-1000 relevant chunks and then feed them into a long-context model. This combines the search capabilities of RAG with the combining power of long context.

---

## Bio Application Scenarios

### Scenario 1: Lab Paper and Protocol Knowledge Chatbot

Index your lab's 30 papers + 200 protocols + professor's lecture notes with RAG. Ask questions via a Slack bot or web UI.

```python
# Preparation (one-time)
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings

splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
docs = load_lab_documents()  # papers, protocols, notes
chunks = splitter.split_documents(docs)

embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory="./lab_db")

# Query (for each call)
def ask_lab(question):
    docs = vectorstore.similarity_search(question, k=10)
    context = "\n\n".join([d.page_content for d in docs])
    
    prompt = f"""You are an assistant that answers questions by referencing our lab's materials.

Answer the question based on the following materials and indicate the source of each piece of evidence in the answer with [author, year, section].
If the information is not in the materials, answer with "Not found in our lab's materials".

Materials:
{context}

Question: {question}
Answer:"""
    
    return llm(prompt, temperature=0)

# Usage
ask_lab("Were there any cases where CRISPR editing efficiency was below 20%?")

Force source indication in the system prompt + reject information outside the materials. Combine with the prompt rules in #7.

Scenario 2: Clinical Trial Literature Search Assistant

Comprehensive search of clinical trial results for a specific disease or treatment.

• Data sources: ClinicalTrials.gov, PubMed, conference abstracts.
• Chunking: Each trial as one document. Abstract, protocol, and results as separate chunks.
• Hybrid search: BM25 is strong for drug names and gene names, so hybrid is essential.
• Reranker: Accuracy in judging clinical relevance is critical to answer quality.
• Metadata filter: Combine vector search with filters such as "Phase 3", "2020 or later", and "adult only".

Such assistants are now being deployed in CROs and hospital research teams.

Scenario 3: Gene and Protein Knowledge QA

BioBERT embedding + UniProt and Ensembl data for gene QA.

• Each UniProt page of a gene is divided into multiple chunks (function, structure, disease, PTM, etc.).
• Use a domain-specific embedding (BioBERT).
• User question: "What kinases interact with TP53?" → Search for relevant chunks → LLM summarizes.

This is the root technology of recent commercial bio tools (Elicit, Consensus, etc.).

Key Takeaways

• The two places where LLM knowledge resides: parameters (built-in, requires retraining) and context window (immediate, finite).
• RAG is an architecture that dynamically injects external knowledge into the context. It is a combination of search and generation.
• Pipeline: chunking → embedding → vector database → search → prompt injection → LLM.
• Chunking is 200-800 tokens + 10-20% overlap. Respect section boundaries.
• Embedding search is a natural extension of the embedding space (see #5). Cosine similarity.
• Vector database - Start with Chroma, scale with Qdrant and Pinecone.
• Hybrid search (dense + BM25) enhances exact identifier matching. Combine with RRF.
• Reranker (cross-encoder) improves final accuracy. Two-step pipeline.
• Limitations: search failure, combining information, citation errors, negative questions.
• RAG is still valid in the age of long context. Reasons: cost, accuracy ("lost in the middle"), and scalability.

📐 Appendix — Mathematical Formulas for Experts

Difficulty: Very Hard Target Audience: Readers with a background in information retrieval, probability, and optimization.

A.1 Cosine Similarity and Inner Product

For two vectors u, v ∈ ℝ^d:

cos_sim(u, v) = ⟨u, v⟩ / (||u||_2 · ||v||_2)

If the vectors are normalized (||u||_2 = 1), then the cosine similarity equals the inner product:

cos_sim(u, v) = ⟨u, v⟩ = Σ_i u_i v_i

Connection to Attention Scores (Section 6). The ⟨Q_i, K_j⟩ in attention is also in the form of an inner product. Retrieval in RAG is conceptually isomorphic to the initial score calculation in attention.

A.2 Contrastive Learning for Training Embedding Models

Adopted by the Sentence-BERT family. Similar pairs are brought closer, and dissimilar pairs are pushed farther apart.

Triplet Loss:

L = max(0, ||f(a) - f(p)||^2 - ||f(a) - f(n)||^2 + margin)

a: anchor, p: positive (similar), n: negative (dissimilar). The negative example is pushed farther away by margin (e.g., 0.5).

InfoNCE Loss (SimCLR family, industry standard):

L = -log[ exp(sim(f(a), f(p)) / τ) / Σ_k exp(sim(f(a), f(k)) / τ) ]

τ: temperature. Other examples in the mini-batch are used as automatic negative examples.

A.3 BM25 Formula

Document D, query Q = {q_1, ..., q_n}:

BM25(D, Q) = Σ_i IDF(q_i) · TF(q_i, D) · (k_1 + 1) / (TF(q_i, D) + k_1 · (1 - b + b · |D| / avgdl))

• IDF(q_i) = log((N - n_i + 0.5) / (n_i + 0.5) + 1). N: total number of documents, n_i: number of documents containing q_i.
• TF(q_i, D): frequency of q_i in document D.
• avgdl: average document length.
• Parameters: k_1 = 1.2~2.0, b = 0.75.

Intuition. Frequent words have a lower IDF and thus a lower weight. If a word appears frequently in a document, the TF is high. Longer documents are normalized. The 30-year standard for search engines.

A.4 Reciprocal Rank Fusion (RRF)

Combines the rankings of multiple search methods:

RRF_score(d) = Σ_{r ∈ R} 1 / (k + rank_r(d))

• R: set of search methods (e.g., dense, sparse).
• rank_r(d): rank of document d in method r (1 is the best).
• k = 60 (empirical default value).

Advantage: Uses only the ranks of each method, not the absolute scores. Can ignore scale differences.

A.5 Cross-Encoder Rerank Formula

Cross-encoder input:

[CLS] question [SEP] document [SEP]

The transformer (BERT/RoBERTa family) performs attention calculations on this entire sequence. The final representation of the [CLS] token passes through a linear layer to obtain a relevance score:

score = W · h_{CLS} + b

Computational Cost. Bi-encoders compute the two encoders independently, so for searching K documents, it involves passing through 2K encoders. Cross-encoders pass through the encoder for each (Q, D) pair, so it involves passing through K encoders, but this is much slower if K is large.

In practice: bi-encoder to retrieve 100, then cross-encoder to rerank top-10 pipeline.

A.6 HNSW Index

Hierarchical Navigable Small World. The current standard for approximate nearest neighbor search.

Structure: Multi-layered graph. Higher layers are sparse (fewer nodes, longer edges), and lower layers are dense (more nodes, shorter edges). During search, the algorithm roughly navigates the upper layers and then performs a precise search in the lower layers.

Time Complexity: O(log N) — much faster than exact search O(N).

Accuracy: Tunable with parameters (M, ef_construction, ef_search). Typically, recall of 95% or higher can be maintained.

Memory: The vectors themselves + the graph structure. The graph overhead is approximately 1-2 times the size of the vectors.

A.7 Query Rewriting Techniques

HyDE (Hypothetical Document Embeddings):

1. Generate "hypothetical answer documents" by feeding the question into an LLM.
2. Embed these hypothetical answers and use them for retrieval.

Reason: Questions and answers have different representations, but answers and answers have similar representations. Searching with hypothetical answers helps match documents that contain the actual answer.

Multi-Query:

1. Rewrite the question in multiple ways using an LLM (different expressions, different angles).
2. Search with each query and combine the results.

A.8 "Lost in the Middle" Phenomenon in Long-Context

Observed by Liu et al. (2023). LLMs tend to find information well at the beginning and end of long contexts, but tend to miss information in the middle.

Accuracy Curve. If you plot the context position on the x-axis and the answer rate on the y-axis, you will get a U-shaped curve. Beginning/end 90%, middle 60%.

Mitigation:

• Place important information at the beginning or end of the context.
• Use a reranker to place the top snippets at the beginning of the context.
• Use RAG to compress the context to only the relevant parts instead of using long-context.

A.9 Probabilistic Regularization of RAG

Question q, answer a, document snippet d.

Naive Bayes Perspective:

P(a | q) = Σ_d P(a | q, d) · P(d | q)

• P(d | q): retrieval probability (embedding similarity).
• P(a | q, d): probability of the answer given the document (LLM).

Fusion-in-Decoder (FiD). Each of the retrieved K documents is encoded independently and then combined in the decoder. RETRO and Atlas use this approach.

Retrieval-Augmented Language Model (RALM). Integrates retrieval into the model parameters themselves. RETRO and REALM. References during each training step.

A.10 RAG Performance Metrics

Retrieval Stage:

• Recall@K: the proportion of times the correct answer is included in the top K.
• Precision@K: the proportion of the top K that are actually relevant.
• MRR (Mean Reciprocal Rank): the average of the reciprocals of the ranks of the correct answers.
• NDCG (Normalized Discounted Cumulative Gain): rank-weighted relevance.

Generation Stage:

• Faithfulness: whether the answer is based on the retrieved documents.
• Answer Relevancy: whether the answer is relevant to the question.
• Context Precision/Recall: whether the retrieved context contains the information needed for the answer.

Tools: Frameworks such as RAGAS and TruLens automatically measure these metrics.

References

All the content, scenarios, analogies, and figures in this section are developed in-house by BioPlayground, and the following are external references that can help with concept learning.

• RAG original paper: Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (NeurIPS 2020)
• Dense Passage Retrieval: Karpukhin et al., "Dense Passage Retrieval for Open-Domain Question Answering" (EMNLP 2020)
• BM25: Robertson & Zaragoza, "The Probabilistic Relevance Framework: BM25 and Beyond" (2009)
• Sentence-BERT: Reimers & Gurevych, "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks" (EMNLP 2019)
• HNSW: Malkov & Yashunin, "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs" (TPAMI 2020)
• Lost in the Middle: Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (TACL 2023)
• HyDE: Gao et al., "Precise Zero-Shot Dense Retrieval without Relevance Labels" (ACL 2023)
• Fusion-in-Decoder: Izacard & Grave, "Leveraging Passage Retrieval with Generative Models" (EACL 2021)
• RETRO: Borgeaud et al., "Improving language models by retrieving from trillions of tokens" (ICML 2022)
• BioBERT: Lee et al., "BioBERT: a pre-trained biomedical language representation model" (Bioinformatics 2020)
• LangChain documentation: python.langchain.com
• LlamaIndex documentation: docs.llamaindex.ai
• Anthropic Contextual Retrieval: anthropic.com/news/contextual-retrieval

In Section 8, you learned how to add external knowledge to LLMs. In Section 9, we will discuss agent patterns where LLMs use tools (search, calculation, API calls) on their own.

Next Concepts

• Ep. #9 agent-and-tool-use — Agent pattern where LLM calls tools. RAG is also one of the agent's tools.
• Ep. #10 context-window-management — Practical techniques for managing context windows wisely.
• Ep. #11 hallucination-and-alignment — Why RAG cannot completely prevent hallucinations.