groupby — Split-Apply-Combine Pattern

After completing this topic

You will be able to explain the internal workings of pandas groupby() using the Split-Apply-Combine pattern, and you will be familiar with common group-by aggregation, transformation, and filtering patterns used in practice.

Why is group-by operation necessary?

Calculating the "overall average" is as simple as df["score"].mean(). However, most real-world questions are in this format:

What is the average salary by department?
What is the total sales by month?
What is the best-selling product by category?

The moment you add a "~ by" phrase, a group-by operation is required. This is the role of groupby().

Split-Apply-Combine Pattern

This pattern, named by Hadley Wickham in 2011, is the most frequently repeated thought process in data analysis.

text

Original data
┌────────┬──────┬───────┐
│ dept   │ name │ score │
├────────┼──────┼───────┤
│ Sales  │ Alice│   85  │
│ Sales  │ Bob  │   90  │
│ Dev    │ Carol│   95  │
│ Dev    │ Dave │   88  │
│ HR     │ Eve  │   78  │
└────────┴──────┴───────┘

1. SPLIT — Split by department
   Sales: [85, 90]
   Dev:   [95, 88]
   HR:    [78]

2. APPLY — Apply a function to each group (e.g., average)
   Sales: 87.5
   Dev:   91.5
   HR:    78.0

3. COMBINE — Combine the results into one
   ┌────────┬───────┐
   │ dept   │ score │
   ├────────┼───────┤
   │ Dev    │  91.5 │
   │ HR     │  78.0 │
   │ Sales  │  87.5 │
   └────────┴───────┘

In pandas, these three steps can be done in a single line:

python

df.groupby("dept")["score"].mean()

Basic Usage

python

import pandas as pd

df = pd.DataFrame({
    "dept": ["Sales", "Sales", "Dev", "Dev", "HR"],
    "name": ["Alice", "Bob", "Carol", "Dave", "Eve"],
    "score": [85, 90, 95, 88, 78],
    "salary": [50000, 52000, 70000, 68000, 45000]
})

# Average score by department
df.groupby("dept")["score"].mean()
# dept
# Dev      91.5
# HR       78.0
# Sales    87.5

# Total salary by department
df.groupby("dept")["salary"].sum()
# dept
# Dev      138000
# HR        45000
# Sales    102000

What is a GroupBy Object?

python

grouped = df.groupby("dept")
print(type(grouped))  # <class 'pandas.core.groupby.DataFrameGroupBy'>

groupby() does not compute immediately. It is a lazy evaluation — the calculation is only performed when a function (mean, sum, etc.) is applied. This prevents unnecessary operations in large datasets.

Checking Group Contents

python

for name, group in df.groupby("dept"):
    print(f"\n--- {name} ---")
    print(group)

# --- Dev ---
#   dept   name  score  salary
# 2  Dev  Carol     95   70000
# 3  Dev   Dave     88   68000
# --- HR ---
# ...

Multiple Aggregation Functions

One column, multiple functions

python

df.groupby("dept")["score"].agg(["mean", "min", "max", "count"])
#        mean  min  max  count
# dept
# Dev    91.5   88   95      2
# HR     78.0   78   78      1
# Sales  87.5   85   90      2

Different functions for each column

python

df.groupby("dept").agg(
    avg_score=("score", "mean"),
    total_salary=("salary", "sum"),
    headcount=("name", "count")
)
#        avg_score  total_salary  headcount
# dept
# Dev         91.5        138000          2
# HR          78.0         45000          1
# Sales       87.5        102000          2

Named aggregation (column_name=("original_column", "function")) is the cleanest way because it allows you to specify the column name of the result at once.

Grouping by Multiple Columns

python

sales = pd.DataFrame({
    "region": ["East", "East", "West", "West", "East", "West"],
    "category": ["A", "B", "A", "B", "A", "A"],
    "revenue": [100, 200, 150, 250, 120, 180]
})

sales.groupby(["region", "category"])["revenue"].sum()
# region  category
# East    A           220
#         B           200
# West    A           330
#         B           250

When you pass multiple columns as a list, a MultiIndex is created.

python

# Convert MultiIndex to a regular column
result = sales.groupby(["region", "category"])["revenue"].sum().reset_index()
#   region category  revenue
# 0   East        A      220
# 1   East        B      200
# 2   West        A      330
# 3   West        B      250

reset_index() returns the group keys from the index to a regular column. This form is more convenient for subsequent operations (visualization, saving, etc.).

transform — Group-by Transformation

agg() returns a single value for each group (5 rows → 3 rows). transform() returns a result of the same size as the original (5 rows → 5 rows).

python

# Attach department average to the original data
df["dept_avg"] = df.groupby("dept")["score"].transform("mean")
print(df)
#    dept   name  score  salary  dept_avg
# 0  Sales  Alice     85   50000      87.5
# 1  Sales    Bob     90   52000      87.5
# 2    Dev  Carol     95   70000      91.5
# 3    Dev   Dave     88   68000      91.5
# 4     HR    Eve     78   45000      78.0

Each row is appended with the average score of its department. This allows you to calculate the score compared to the department average:

python

df["vs_avg"] = df["score"] - df["dept_avg"]
# Alice: 85 - 87.5 = -2.5 (below department average)
# Carol: 95 - 91.5 = +3.5 (above department average)

filter — Group-by Filtering

It keeps or removes the entire group that satisfies a condition.

python

# Keep only departments with 2 or more members
df.groupby("dept").filter(lambda g: len(g) >= 2)
#    dept   name  score  salary
# 0  Sales  Alice     85   50000
# 1  Sales    Bob     90   52000
# 2    Dev  Carol     95   70000
# 3    Dev   Dave     88   68000

HR (1 member) was completely removed. The key to filter() is that it filters at the group level, not at the individual row level.

Practical Pattern — Monthly Sales Analysis

python

orders = pd.DataFrame({
    "date": pd.to_datetime([
        "2026-01-05", "2026-01-15", "2026-02-03",
        "2026-02-20", "2026-03-10", "2026-03-25"
    ]),
    "product": ["Widget", "Gadget", "Widget", "Gadget", "Widget", "Widget"],
    "amount": [1200, 800, 1500, 900, 1100, 1300]
})

# Total monthly sales
monthly = orders.groupby(orders["date"].dt.to_period("M"))["amount"].sum()
print(monthly)
# date
# 2026-01    2000
# 2026-02    2400
# 2026-03    2400

# Average monthly sales by product
orders.groupby("product").agg(
    total=("amount", "sum"),
    avg=("amount", "mean"),
    count=("amount", "count")
)
#         total     avg  count
# Gadget   1700   850.0      2
# Widget   5100  1275.0      4

Using .dt.to_period("M") to create monthly groups from date data is a very common pattern.

Key Summary

Method	Operation	Result Size
`agg()`	Group-by aggregation (sum, mean, count...)	Number of groups (decreases)
`transform()`	Group-by transformation (maintains original size)	Same as original
`filter()`	Keeps groups that satisfy the condition	Less than or equal to original

groupby() is the most powerful and most frequently used feature in pandas. If you can visualize the Split-Apply-Combine pattern in your head, you can naturally break down complex aggregations into three steps: "which column to split by, which function to apply, and how to combine the results."

BioPlayground

groupby — Split-Apply-Combine Pattern

groupby — Split-Apply-Combine Pattern

After completing this topic

Why is group-by operation necessary?

Split-Apply-Combine Pattern

Basic Usage

What is a GroupBy Object?

Checking Group Contents

Multiple Aggregation Functions

One column, multiple functions

Different functions for each column

Grouping by Multiple Columns

transform — Group-by Transformation

filter — Group-by Filtering

Practical Pattern — Monthly Sales Analysis

Key Summary

💬 Questions & Comments