Vectorization — Why You Should Abandon For Loops

After Completing This Topic

You will be able to structurally explain why vectorization is fast and develop the habit of replacing for loops with NumPy/pandas vector operations in your code.

Intuition — Same Result, 100x Difference

Two ways to square 1 million numbers:

python

import numpy as np
import time

data = list(range(1_000_000))
arr = np.array(data)

# Method 1: for loop
start = time.time()
result_loop = [x ** 2 for x in data]
print(f"for loop: {time.time() - start:.3f} seconds")

# Method 2: NumPy vector operation
start = time.time()
result_vec = arr ** 2
print(f"Vectorization: {time.time() - start:.3f} seconds")

text

for loop: 0.142 seconds
Vectorization: 0.002 seconds    ← Approximately 70x faster

The results are the same. The code is shorter. The speed is tens of times faster.

Why is There Such a Difference?

The reason Python's for loop is slow is that the Python interpreter intervenes with each iteration:

text

for loop (Python):
  Iteration 1: Type check → Object creation → Operation → Object storage
  Iteration 2: Type check → Object creation → Operation → Object storage
  ... (1 million iterations)

Vector operation (NumPy):
  Once: C code processes continuous memory data in bulk → Returns result array

NumPy internally runs the loop in C/Fortran code. There is no overhead from the Python interpreter, and the data is stored contiguously in memory, allowing for efficient use of the CPU cache.

Pattern 1 — Arithmetic Operations

python

import numpy as np

prices = np.array([1000, 2500, 3000, 1500, 4000])
tax_rate = 0.1

# ❌ for loop
total_loop = []
for p in prices:
    total_loop.append(p * (1 + tax_rate))

# ✅ Vectorization
total_vec = prices * (1 + tax_rate)
print(total_vec)  # [1100. 2750. 3300. 1650. 4400.]

prices * (1 + tax_rate) — performs the operation on the entire array at once.

Pattern 2 — Conditional Filtering

python

scores = np.array([85, 42, 91, 67, 55, 78, 93, 38])

# ❌ for loop
passed = []
for s in scores:
    if s >= 60:
        passed.append(s)

# ✅ Boolean indexing
passed = scores[scores >= 60]
print(passed)  # [85 91 67 78 93]

scores >= 60 creates a boolean array [True, False, True, True, False, True, True, False], and using this as an index extracts only the elements where the value is True.

Pattern 3 — The Same Applies to Pandas

python

import pandas as pd

df = pd.DataFrame({
    "name": ["A", "B", "C", "D", "E"],
    "revenue": [500, 800, 300, 1200, 900],
    "cost": [200, 600, 150, 700, 400]
})

# ❌ for loop
profits = []
for _, row in df.iterrows():
    profits.append(row["revenue"] - row["cost"])
df["profit_loop"] = profits

# ✅ Vectorization
df["profit_vec"] = df["revenue"] - df["cost"]

Iterating through rows with df.iterrows() is almost always a bad approach in pandas. Column-wise operations are vectorized.

Pattern 4 — Conditional Value Assignment

python

# ❌ for loop
for i, row in df.iterrows():
    if row["profit_vec"] > 500:
        df.at[i, "grade"] = "A"
    else:
        df.at[i, "grade"] = "B"

# ✅ np.where
df["grade"] = np.where(df["profit_vec"] > 500, "A", "B")

np.where(condition, value_if_true, value_if_false) — applies an if/else statement to the entire array at once.

For more complex multiple conditions:

python

conditions = [
    df["profit_vec"] > 500,
    df["profit_vec"] > 200,
    df["profit_vec"] > 0
]
choices = ["A", "B", "C"]
df["grade"] = np.select(conditions, choices, default="D")

When Vectorization is Not Possible

Not all operations can be vectorized:

python

# Cumulative calculation that depends on the value of the previous row
# (Cannot be parallelized because each row needs to reference the result of the previous row)
result = [data[0]]
for i in range(1, len(data)):
    result.append(result[-1] + data[i])

In these cases, look for dedicated functions like np.cumsum(), and if none exist, use a for loop, but limit its scope as much as possible.

Benchmark Rules

Data Size	for loop	Vectorization	Difference
1,000	~0.2ms	~0.01ms	~20x
100,000	~20ms	~0.1ms	~200x
10,000,000	~2s	~20ms	~100x

The larger the data, the greater the benefit of vectorization.

Key Takeaways

Situation	Use
Arithmetic operations	`arr * 2`, `arr + arr2`
Conditional filtering	`arr[arr > 0]`
Conditional assignment	`np.where()`, `np.select()`
Aggregation	`arr.sum()`, `arr.mean()`
Row-by-row iteration	❌ Use column operations instead of `iterrows()`

apply() — Between Vectorization and For Loops

When you have complex logic that cannot be expressed with vector operations, use apply():

python

def categorize(row):
    if row["revenue"] > 1000 and row["profit_vec"] > 300:
        return "Excellent"
    elif row["profit_vec"] > 0:
        return "Average"
    else:
        return "Loss"

df["category"] = df.apply(categorize, axis=1)

apply() is faster than iterrows() but slower than pure vector operations. Use apply() for complex logic and vectorization for simple operations.

String Vectorization — .str Accessor

python

names = pd.Series(["Alice Smith", "Bob Jones", "Charlie Brown"])

# ❌ for loop
upper_names = []
for name in names:
    upper_names.append(name.upper())

# ✅ Vectorization
upper_names = names.str.upper()
first_names = names.str.split(" ").str[0]
has_e = names.str.contains("e", case=False)

print(first_names)  # ["Alice", "Bob", "Charlie"]

pandas' .str accessor vectorizes string methods. Most string operations, such as split, replace, contains, and extract, can be performed without a for loop.

Refactoring Checklist

When you find a for loop in existing code, check the following in this order:

Is it an arithmetic/comparison operation? → Use the operator directly (arr + 1, arr > 0)
Is it a conditional branch? → np.where() or np.select()
Is it string processing? → .str accessor
Is it aggregation? → .sum(), .mean(), .groupby()
Is it complex logic? → apply()
Does it depend on the previous row? → cumsum(), shift(), look for dedicated functions
None of the above applies → for loop (minimum scope)

groupby — Group-by Vector Operations

python

df = pd.DataFrame({
    "department": ["Sales", "Development", "Sales", "Development", "Development"],
    "salary": [5000, 6000, 4500, 7000, 5500]
})

# ❌ for loop
departments = df["department"].unique()
for dept in departments:
    mask = df["department"] == dept
    print(f"{dept}: {df.loc[mask, 'salary'].mean()}")

# ✅ groupby vector operation
print(df.groupby("department")["salary"].mean())

groupby() applies vector operations to each group. It is internally C-optimized, making it faster and more concise than a for loop.

dtype Considerations

The speed of vector operations is also affected by the dtype:

python

# object dtype (column with mixed strings) → vectorization not possible
mixed = pd.Series([1, "two", 3])        # dtype: object → slow
clean = pd.Series([1, 2, 3])            # dtype: int64 → fast

# Check dtype
print(df.dtypes)
# If a numeric column is mistakenly captured as an object, it needs to be converted
df["price"] = pd.to_numeric(df["price"], errors="coerce")

When reading a CSV, if a numeric column has even one empty string, the entire column will be the object dtype, and vector operations will not work. Use pd.to_numeric() to convert it before use.

"Before writing a for loop, first check if there is a vectorized method." This one habit can greatly improve data processing performance.

BioPlayground

Vectorization — Why You Should Ditch the For Loop

Vectorization — Why You Should Abandon For Loops

After Completing This Topic

Intuition — Same Result, 100x Difference

Why is There Such a Difference?

Pattern 1 — Arithmetic Operations

Pattern 2 — Conditional Filtering

Pattern 3 — The Same Applies to Pandas

Pattern 4 — Conditional Value Assignment

When Vectorization is Not Possible

Benchmark Rules

Key Takeaways

apply() — Between Vectorization and For Loops

String Vectorization — .str Accessor

Refactoring Checklist

groupby — Group-by Vector Operations

dtype Considerations

💬 Questions & Comments