Neural Networks: Pattern-Recognizing Machines Made of Dials and Switches

After completing this topic, you will:

Understand why neural networks are sophisticated mathematical functions, not just "imitations of the brain." You'll see how the four components—neurons, weights, biases, and activation functions—work together to transform low-level data, such as pixels, into high-level decisions, such as tumor detection. This will also connect to the LLMs you learned about in Part 1 as extensions of this structure.

This section serves as the foundation for the following sections (Part 3: Gradient Descent, Part 4: Backpropagation, and Part 5: Transformers).

The Problem of a Single Pathology Slide

Let's say you're a graduate student on a temporary assignment at a hospital's pathology department. Your task for today is to analyze 500 tissue slides that have just arrived from the lab. Each slide is an image of a tissue section stained with H&E, and you need to determine whether "this tissue contains tumor cells, and if so, is it benign or malignant."

500 slides. Even if you carefully examine each slide for just 5 minutes, it will take 40 hours. Moreover, even expert pathologists often disagree on such interpretations, as the analysis is often subtle. It's not that each individual pixel has a special meaning, but rather that the arrangement, shape, and density patterns formed by the pixels are the basis for the judgment.

But what if you had a magical machine? A machine that takes all the pixel values in an image as input and outputs probabilities for three possible classifications: normal, benign, and malignant. With this machine, you could process 500 slides in seconds, and you would only need to carefully review the ambiguous cases where the probabilities are uncertain.

This machine is a neural network. And the LLMs you learned about in Part 1 are just very sophisticated and large special forms of this neural network. In this section, we will examine what components this machine consists of and how low-level data, such as pixels, can lead to high-level decisions, such as tumor detection.

Neural Networks Are Not "Imitations of the Brain"

First, let's clarify a common misconception. Because it's called an "Artificial Neural Network," people often imagine it as a sophisticated imitation of the brain or even an artificial brain. This is not the case.

An artificial neural network is a highly sophisticated multivariate mathematical function. The reason the word "neural" is in its name is that the early developers were inspired by the firing structure of biological neurons, but modern neural networks are very different from actual brain neurons in both structure and how they work. While biological neurons process information through spike timing, chemical signaling, and the collaboration of thousands of ion channels, the neurons in artificial neural networks are simply computational units that take a weighted sum of input values, add a bias, and pass it through a non-linear function.

It's important to keep this perspective in mind because the concepts we will learn in the future are about understanding the form of mathematical functions, not about understanding the mysteries of the brain.

Let's start with a precise definition:

An artificial neural network is a multivariate mathematical function that maps high-dimensional input data (e.g., image pixel values) to the desired answer space (e.g., normal, benign, and malignant probabilities) through a process of hierarchical feature extraction.

The two key words here are hierarchical and mapping. These two words encompass everything that a neural network does.

The Reality of a Neuron: A Single Number with an Activation State

The smallest component that makes up a neural network is the neuron, or node. The reality of a single neuron is surprisingly simple:

One neuron = one number. Usually a real number between 0 and 1.

This number is called the activation state. A value close to 0 means "this neuron is not particularly active," and a value close to 1 means "this neuron is strongly active." The activation state of an individual neuron is not a physical entity but simply the result of a particular calculation.

Let's go back to the pathology slide example. If the slide image is 256x256 pixels, there are 65,536 pixels. If you treat the brightness value of each pixel as a single neuron, then the input layer has 65,536 neurons. The brightness value of each pixel (normalized to a value between 0 and 1) is the activation state of that neuron.

Of course, the final output is three probabilities for normal, benign, and malignant. So the output layer has three neurons, and the activation state of each neuron is the probability of each class.

Between the input layer and the output layer are intermediate layers called hidden layers. The reason they are called "hidden" is because they are the intermediate calculation results that we don't directly see. The activation states of the hidden layer neurons are not directly interpreted but are passed to the next layer.

The overall structure looks like this:

Input Layer (65,536 neurons, pixel brightness)
   ↓ [connection]
Hidden Layer 1 (e.g., 512 neurons, what it represents is determined by learning)
   ↓ [connection]
Hidden Layer 2 (e.g., 128 neurons, more abstract features)
   ↓ [connection]
Hidden Layer 3 (e.g., 32 neurons, even more abstract features)
   ↓ [connection]
Output Layer (3 neurons, normal, benign, and malignant probabilities)

This structure is called a multilayer perceptron or a feedforward neural network. Because information flows in only one direction from input to output, it is called "feedforward."

Hierarchical Feature Detection: From Low-Level to High-Level

This is where the real magic of neural networks happens. Why stack it into so many layers? Because different levels of abstraction occur in each layer.

Let's imagine what's actually happening in a pathology slide.

The neurons in Hidden Layer 1 detect very local features around each pixel. For example, some neurons may respond to the signal "there is a sharp change in brightness here" – acting as a cell boundary detector. Other neurons may respond to the signal "this area is particularly dark" – acting as a cell nucleus detector.

The neurons in Hidden Layer 2 combine several signals from Hidden Layer 1 to detect slightly larger patterns. "Several cell boundaries are arranged in a circular pattern" → individual cell detector. "The nucleus is large and there are several clustered together" → nucleus abnormality detector.

The neurons in Hidden Layer 3 combine the signals from Hidden Layer 2 to see even larger patterns. "Cells are tangled in a disorganized manner" → tissue structure collapse detector. "Normal tissue arrangement is maintained" → normal tissue arrangement detector.

The three neurons in the output layer synthesize these features from Hidden Layer 3 to make the final judgment. If the tissue structure collapse signal and the nucleus abnormality signal are strong, then → malignant probability increases. If the normal arrangement signal is strong, then → normal probability increases.

This hierarchical feature detection is what the "deep" in deep learning means. The reason for stacking it deeply is to allow abstraction to flow naturally from low-level to high-level. Pixel → boundary → cell → tissue arrangement → tumor detection. This flow is the key to why neural networks can handle complex data such as images, language, and sound.

Interesting fact: We don't tell the neurons in the hidden layer in advance, "You detect cell boundaries." We simply give it training data (e.g., thousands of pathology slides with correct answers) and train it, and the neural network figures out on its own what features to place in each hidden layer. Whether a neuron becomes a cell boundary detector or a nucleus detector is determined by the training. This will be discussed in more detail in Part 3 (Gradient Descent) and Part 4 (Backpropagation).

This perspective may be particularly familiar to biological researchers. The signal transduction pathways in our bodies are exactly this structure. Cell surface receptor (hundreds) → secondary messenger (dozens) → transcription factor (several) → changes in gene expression. A hierarchical structure in which low-level signals lead to high-level cell responses. The flow of hidden layers in a neural network is exactly the same concept.

Dials and Thresholds: Weights and Biases

Now let's see how the neurons and neurons are connected. Let's pick a specific neuron in Hidden Layer 1. How is the activation state of this neuron determined?

Each of the 65,536 neurons in the input layer is connected to this neuron in the hidden layer by a connection line, and each of these connection lines is assigned a number. This number is called the weight.

A large weight (e.g., +2.0) means "strongly pay attention to this input neuron's signal." A weight of 0 means "ignore this input neuron's signal." A negative weight (e.g., -1.5) means "if this input neuron is on, I will react in the opposite direction."

The activation state of the hidden layer neuron is determined as follows:

1. Multiply the activation state of each connected input neuron by its connection weight and add them all together. This is called the weighted sum.
2. Then, add a single number called the bias.
3. Pass the result through an activation function, which is a non-linear function that squashes the input value to a range between 0 and 1 (or another specified range).

Written as a formula (precisely organized in Appendix A.1, but only the concept is described here):

Activation state = activation_function(weighted sum + bias)

The weight is a dial, and the bias is a threshold. The dial adjusts "how much this neuron will pay attention to this input," and the threshold adjusts "how easily this neuron will react." If the bias is a large positive number, the neuron will turn on even with a little stimulation, and if it is a large negative number, it will only turn on with strong stimulation.

Why is the activation function necessary? If we just calculate the weighted sum and pass it on as is, the entire neural network will be just a single large linear function. No matter how many linear functions we put together, it will still be a single linear function. This will make hierarchical feature detection impossible. A non-linear activation function must be present in each layer to enable different types of calculations in each layer.

Two of the most commonly used activation functions:

Sigmoid: Smoothly compresses the input to a value between 0 and 1. It has been used since a long time ago and is still used in the output layer because it is easy to interpret as a probability.

ReLU (Rectified Linear Unit): max(0, x). Negative values are set to 0, and positive values are passed through as is. It is much simpler, but it works very well in the hidden layer. Most modern neural networks use this as the activation function for the hidden layer.

Appendix A.3 discusses the precise formulas and graphical forms of each function.

Scale Matters — How Many Knobs Are We Talking About?

Let's start with a simple example to get a sense of scale.

Imagine we're building a very small neural network for pathology slide analysis. The architecture is: Input 65,536 → Hidden 512 → Hidden 128 → Hidden 32 → Output 3.

We'll assume that the connections between each layer are fully connected. This means that every neuron in the previous layer is connected to every neuron in the next layer.

• Input Layer → Hidden Layer 1: 65,536 × 512 = 33,554,432 weights
• Hidden Layer 1 → Hidden Layer 2: 512 × 128 = 65,536 weights
• Hidden Layer 2 → Hidden Layer 3: 128 × 32 = 4,096 weights
• Hidden Layer 3 → Output Layer: 32 × 3 = 96 weights

That's a total of roughly 33.62 million weights. Plus, each neuron has a bias, adding 675 biases. These weights and biases are the values that will be adjusted during training.

Could a human adjust these 33.62 million values by hand? Even if they spent all day turning one dial at a time, it would take 100 years. Clearly, we need an automatic adjustment algorithm. This is the reason for the learning algorithm we will discuss in Part 3 (gradient descent).

This 33.62 million is still a very small neural network. Real-world pathology AI (e.g., PathAI, Paige) uses hundreds of millions or even billions of parameters. As we saw in Part 1, GPT-3 has 175 billion, and the latest frontier models have even more. The scale of the parameters directly determines the complexity and capability of the model.

Let's translate this into a scale that is familiar to biology researchers. The human genome has about 20,000 genes, and each gene is connected to approximately dozens or hundreds of transcription factors (TF binding sites, enhancers, silencers). The "number of connections" in the overall genome regulatory network is roughly in the millions or tens of millions. This means that the scale of the neural network's parameters is comparable to or even larger than this genome regulatory network. Since the genome regulatory system within the body regulates development, homeostasis, and disease with this level of complexity, it's not surprising that a neural network of this scale can perform complex pattern recognition.

Forward Propagation — Calculation Flows in One Direction

Now, let's outline the process by which a neural network actually makes a decision. This process is called forward propagation or feed-forward.

1. Input: Set the activation states of the input layer neurons to the 65,536 pixel brightness values of the slide image.
2. Hidden Layer 1 Calculation: For each neuron in hidden layer 1, calculate the (weighted sum + bias) and pass it through the activation function to determine the activation state.
3. Hidden Layer 2 Calculation: Use the activation states of hidden layer 1 as input and repeat the same calculation.
4. Hidden Layer 3 Calculation: Use the activation states of hidden layer 2 as input.
5. Output Layer Calculation: Use the activation states of hidden layer 3 as input for the final calculation. The output layer typically uses Softmax as the activation function to make the values of the three neurons a probability distribution (sum of 1).
6. Decision: Classify the input into the class with the highest probability. Or, use the probability itself as a risk score.

Here's the process in pseudocode:

def neural_network_prediction(pixel_values):
    a = pixel_values  # Input vector, size 65,536
    for each layer (weight matrix W, bias vector b) in network_layers:
        weighted_sum = W · a + b
        a = activation_function(weighted_sum)  # ReLU or Sigmoid
    return Softmax(a)  # Final probability distribution

That's it. Multiplication and addition, and then passing through a non-linear function. Once trained, the neural network performs this calculation in milliseconds per slide image.

Training is a completely separate process. During training, the neural network actually does is slightly adjust the weights and biases to make more accurate predictions. The adjustment algorithm is gradient descent, which we will discuss in Part 3, and backpropagation in Part 4.

How Does This Relate to the LLMs in Part 1?

Let's briefly return to Part 1. The LLMs we learned about in Part 1 are also neural networks. However, there are some key differences:

1. Nature of the input. The pathology neural network takes image pixels (continuous real numbers) as input. LLMs take text tokens (discrete symbols) as input. The process of converting text tokens into vectors that a neural network can handle is called embedding, and we will discuss this in more detail in Part 5.

2. Architectural differences. The feedforward neural network discussed in this part has fully connected layers. LLMs use a different architecture called Transformer. The Transformer also contains feedforward layers, but it also includes a special operation called attention. We will discuss this in Parts 5 and 6.

3. Scale. If the pathology neural network has hundreds of millions of parameters, the latest LLMs have 100 to 1000 times more. The larger scale leads to the emergent abilities we saw in Part 1.

Similarities. The fundamental structure is all made up of a multivariate function consisting of weights, biases, and activation functions. The principle of abstracting low-level data into high-level representations as it passes through layers is the same. The concepts learned in this part will serve as the foundation for all the neural network-based models that will be introduced later.

Bio Application Scenarios

How does this theory translate into practice?

Scenario 1 — Single-Cell RNA Sequencing Cell Type Classification

In single-cell RNA sequencing (scRNA-seq), each cell is represented by a vector of expression values for thousands of genes. Feeding this vector into a neural network to automatically classify cell types (T cells, B cells, macrophages, etc.) is a direct application of the concepts in this part.

• Input layer: Thousands of genes (e.g., the top 2,000 variable genes)
• Hidden layer 1: Low-level combinations of gene expression patterns (e.g., activation of specific signaling pathways)
• Hidden layer 2: Combinations of cellular programs (e.g., active vs. resting state)
• Output layer: Probability distribution of cell types

Libraries like scVI and scanpy have standardized this approach.

Scenario 2 — Automated Pathology Slide Reading (The Actual Scenario from the Opening of This Part)

Deep learning in pathology has rapidly entered clinical practice over the past five years. Commercial solutions such as PathAI, Paige, and Aiforia are used for automated reading of slides for breast cancer, prostate cancer, and colorectal cancer. In Korea, research in this area is also active at Seoul National University Hospital and Samsung Medical Center.

The neural networks used here are more sophisticated than the fully connected architectures discussed in this part; they are Convolutional Neural Networks (CNNs). However, the fundamental principles (hierarchical feature detection, weights, biases, and activation functions) are the same. Although we will not cover CNNs separately in this series, if you understand the concepts in this part, you will be able to easily understand CNNs.

Scenario 3 — Drug Response Prediction

We can train a neural network on data from experiments in which several drug candidates were tested on cell lines to create a model that predicts drug response (IC50) in new cell lines.

• Input: Cell line's genetic profile + drug's chemical characteristic vector
• Hidden layer: Extraction of interaction patterns between genes and drug characteristics
• Output: Predicted IC50 value (real number)

This is a regression problem, not a classification problem, but the neural network structure is almost the same. The main difference is that the activation function of the output layer is changed from Softmax to an identity function. This is an approach that is actually used in drug discovery by companies such as Genentech and Novartis.

Key Takeaways

• Artificial neural networks are not imitations of the brain, but rather sophisticated multivariate mathematical functions. Don't be fooled by the name.
• Neuron = activation state (a single number). Connection = weight (knob). Threshold = bias. Activation function = non-linear transformation.
• Neural networks transform low-level data into high-level decisions through hierarchical feature detection. Pixel → edge → cell → tissue arrangement → tumor classification.
• The reason for stacking multiple layers to create "depth" is here.
• The number of parameters (weights and biases) ranges from millions to billions. Humans cannot adjust them by hand. Training algorithms are essential.
• LLMs are built on the principle of neural networks, with the addition of Transformer and attention, which are special structures.

📐 Appendix — Mathematical Formulas for Experts

Difficulty: Very Hard Target Audience: Readers with a graduate-level understanding of linear algebra, calculus, and probability.

While the main text focused on intuition, this section summarizes the rigorous mathematics of neural networks. Feel free to skip it if you're encountering it for the first time.

A.1 Activation Calculation for a Single Neuron

Let a^{l-1} ∈ R^{n_{l-1}} be the activation vector of the previous layer, w_j^{l} ∈ R^{n_{l-1}} be the weight vector of the j-th neuron in this layer, and b_j^{l} ∈ R be the bias. Then, the activation of the j-th neuron is:

z_j^{l} = Σ_i w_{ji}^{l} · a_i^{l-1} + b_j^{l}       (Weighted sum)
a_j^{l} = σ(z_j^{l})                                  (Passing through the activation function)

Here, σ is the activation function (Sigmoid, ReLU, etc.).

A.2 Vectorized Layer Calculation

Representing the entire layer as a vector and matrix makes it more elegant. From layer l-1 to layer l:

z^{l} = W^{l} · a^{l-1} + b^{l}
a^{l} = σ(z^{l})

• W^{l} ∈ R^{n_l × n_{l-1}}: Weight matrix of layer l. W_{ji}^{l} is the weight from the i-th neuron in the previous layer to the j-th neuron in this layer.
• a^{l-1} ∈ R^{n_{l-1}}: Activation vector of the previous layer.
• b^{l} ∈ R^{n_l}: Bias vector of this layer.
• a^{l} ∈ R^{n_l}: Activation vector of this layer.

This vectorization is the foundation of GPU parallel processing. GPUs perform large matrix-vector multiplications in parallel using thousands of cores.

A.3 List of Activation Functions

Sigmoid — Smooth S-shaped curve, output (0, 1):

σ(z) = 1 / (1 + exp(-z))

The gradient is σ(z)(1 - σ(z)). It is a cause of the vanishing gradient problem because the gradient becomes very small when z is large or small, so it is not commonly used in hidden layers of deep neural networks.

Tanh — Smooth S-shaped curve, output (-1, 1):

tanh(z) = (exp(z) - exp(-z)) / (exp(z) + exp(-z))

A center-shifted version of Sigmoid. Slightly better than Sigmoid in hidden layers. The vanishing gradient problem still exists.

ReLU — Output is 0 for negative values, and the value itself for positive values:

ReLU(z) = max(0, z)

Standard in hidden layers since the 2010s. The derivative is essentially 1 (z > 0) or 0 (z < 0), so the computation is extremely fast. However, there is a phenomenon called "dying ReLU" where the neuron stops learning when z < 0 because the gradient is 0.

Leaky ReLU — Reflects negative values slightly:

LeakyReLU(z) = max(0.01 · z, z)

Mitigates the dying ReLU problem. The coefficient 0.01 is a hyperparameter.

GELU (Gaussian Error Linear Unit) — Standard in Transformers and LLMs:

GELU(z) = z · Φ(z)

Φ is the cumulative distribution function of the standard normal distribution. Similar to a smooth ReLU, but slightly different around z=0. Used in GPT and BERT, etc.

A.4 Softmax — Standard for Multi-Class Output Layers

When the output needs to be a probability distribution over multiple classes (e.g., a 3-way classification of normal/positive/malignant). Given logits z_1, ..., z_K for K classes:

Softmax(z_k) = exp(z_k) / Σ_j exp(z_j)

Characteristics:

• All outputs are in the (0, 1) range.
• The sum of all outputs is exactly 1 (satisfies the probability distribution).
• The probability is concentrated on the largest logit (soft argmax).

Numerical stability: Calculate exp(z_k - max(z)) to prevent overflow.

A.5 General Formula for Forward Propagation

The entire neural network with L hidden layers can be represented as a single function f_θ:

a^{(0)} = x                                    (Input)
z^{(l)} = W^{(l)} · a^{(l-1)} + b^{(l)}        (l = 1, ..., L)
a^{(l)} = σ^{(l)}(z^{(l)})                      (l = 1, ..., L-1)
a^{(L)} = Softmax(z^{(L)})                      (Output, in the case of classification)
f_θ(x) = a^{(L)}

Parameter set θ = {W^{(1)}, b^{(1)}, ..., W^{(L)}, b^{(L)}}. Adjusting these parameters through training is the topic of the next section (gradient descent).

A.6 Formula for Calculating the Number of Parameters

When the layer sizes are [n_0, n_1, ..., n_L], the total number of parameters is:

Total parameters = Σ_{l=1}^{L} (n_{l-1} · n_l + n_l)
                = Σ_{l=1}^{L} n_l · (n_{l-1} + 1)

Example: Structure [65536, 512, 128, 32, 3]:

• Layer 1: 512 × (65536 + 1) = 33,554,944
• Layer 2: 128 × (512 + 1) = 65,664
• Layer 3: 32 × (128 + 1) = 4,128
• Layer 4: 3 × (32 + 1) = 99
• Total: 33,624,835 parameters

This is the exact value for the "approximately 33.62 million" mentioned in the main text.

A.7 Universal Approximation Theorem

Theoretical background. Even with only one hidden layer, a neural network can approximate any continuous function with arbitrary accuracy if the number of neurons is sufficiently large (Cybenko 1989, Hornik 1991).

Mathematically: If f : [0,1]^n → R is a continuous function, then for any ε > 0, there exists a neural network g with a single hidden layer such that for all x ∈ [0,1]^n, |f(x) - g(x)| < ε.

This theorem is powerful, but it has practical limitations. "Sufficiently large" can be exponentially large in practice, and there is no guarantee that training will find the optimal point. The empirical finding in deep learning is that stacking layers deeply is much more parameter-efficient.

A.8 Initialization

How you set the weights before training is important. Initializing all weights to 0 leads to symmetry problems, so random initialization is essential.

Xavier / Glorot Initialization (for Sigmoid, Tanh):

W ~ Uniform(-√(6/(n_in + n_out)), √(6/(n_in + n_out)))

He Initialization (for ReLU):

W ~ Normal(0, √(2/n_in))

Biases are usually initialized to 0. Poor initialization can prevent training or significantly slow it down.

References

All content, scenarios, analogies, and figures in this section are developed by BioPlayground, and the following are external references that may be helpful for learning the concepts.

• Standard Deep Learning Textbook: Goodfellow, Bengio, Courville, "Deep Learning" (MIT Press, free online)
• Neural Network Visualization Education: 3Blue1Brown "Neural Networks" series (YouTube) — for pedagogical reference.
• Universal Approximation Theorem Original Paper: Cybenko, "Approximation by Superpositions of a Sigmoidal Function" (1989)
• Initialization Methods: He et al., "Delving Deep into Rectifiers" (ICCV 2015)
• Pathological Deep Learning Clinical Cases: Campanella et al., "Clinical-grade computational pathology using weakly supervised deep learning on whole slide images" (Nature Medicine 2019)
• scRNA-seq Deep Learning: Lopez et al., "Deep generative modeling for single-cell transcriptomics (scVI)" (Nature Methods 2018)

This section is both the beginning of the principle section and the foundation for the following sections. In section #3, we will discuss how this neural network "learns" — how the 33.62 million dials are automatically adjusted.

Next Concepts

• Ep. #3 how-nn-learns — How do we automatically adjust 33.6 million dials? The intuition of gradient descent.
• Ep. #4 backpropagation-intuition — How does the error signal of the last layer lead to the adjustment of weights in the preceding layers?
• Ep. #5 transformer-and-embedding — What extensions did the neural network of the LLM make on the structure of this episode?