Back to List

Cellpose 3 + SAM Cell Segmentation Pipeline β€” A Practical Guide to Identifying Individual Cells in Fluorescence Microscopy Images

A hybrid pipeline using Cellpose 3 (optimized for 2024 image restoration) + SAM 2 to automatically detect cell boundaries in fluorescence microscopy images. This complete pipeline includes background noise restoration, cell instance segmentation, area, circularity, and intensity morphological quantification, and napari visualization.

Intermediate
|
30min
|
Verified (2026-07)
Progress0/15 (0%)

Cellpose 3 + SAM Cell Segmentation Pipeline: A Practical Guide to Identifying Individual Cells in Fluorescence Microscopy Images

Anyone who has manually traced cell boundaries in fluorescence microscopy images will immediately understand why this guide is necessary. If you have 500 cells per slide and 100 slides per day, labeling 50,000 cells manually will take days. However, quantitative analysis of cell shape, area, and intensity is a fundamental input for downstream analysis (drug response, phenotypic screening, histopathology), so addressing this bottleneck will accelerate the entire experimental cycle. This guide presents a practical pipeline that combines the image restoration capabilities of Cellpose 3 and Meta's Segment Anything 2 to reduce this process to about 30 minutes.

πŸ“š Recommended Prerequisite

This is an advanced AI x Biology topic. We strongly recommend that you first review the following DryBench tutorials before diving in.

Without the prerequisites, it will be difficult to follow the practical code, as this guide will proceed without re-explaining the principles of U-Net-based encoder-decoder architectures and PyTorch tensor/GPU transfer.


We Already Learned This in DryBench

In DryBench ai-native #2, we learned how neural networks learn representations by repeatedly applying weighted multiplication and non-linear activation functions to inputs, and in particular, why U-Net-based encoder-decoder architectures are strong for pixel-by-pixel prediction (segmentation) of images. In #12, we covered the basics of PyTorch tensor manipulation, GPU transfer, and model forward passes.

However, real-world experimental images present some additional challenges. These include uneven illumination (vignetting), out-of-focus blur, cell crowding, color shifts between channels, and low signal-to-noise ratio (SNR) in fluorescence conditions. We will examine in practice how robust pretrained CNNs are under these conditions, and how the latest foundation models (SAM 2) can further enhance this robustness.

Defining the Problem

Practical Requirements for Cell Segmentation

From a single fluorescence image (2048x2048), we need to automatically extract the following four pieces of information:

  • Instance Segmentation: Unique ID + pixel mask for each cell. Distinguish overlapping cells.
  • Morphological Statistics: Area (pxΒ²), circularity, and eccentricity for each cell.
  • Intensity Statistics: Mean, maximum, and standard deviation of fluorescence channels for each cell.
  • Spatial Relationships: Distance between neighboring cells, whether they form clusters.

Target metrics:

  • Cell detection recall β‰₯ 0.92 (compared to human labeling).
  • Cell detection precision β‰₯ 0.90.
  • Processing speed: 3-5 seconds per GPU, 30-60 seconds per CPU for a 2048x2048 image.

Spectrum of Existing Approaches

  • Classical rule-based (Otsu threshold + watershed): Recall of 0.7-0.8 for low-density, high-SNR images; often fails for overlapping cells.
  • StarDist (star-convex polygon): Strong for segmenting round nuclei; weak for complex cell bodies.
  • Cellpose 1/2 (U-Net + flow field): Covers a wide range of cell shapes; strong zero-shot performance with pre-trained weights. Cellpose 3 added an image restoration backbone in 2024.
  • SAM (Segment Anything Model): Trained on natural images, but surprisingly effective for bioimages in a zero-shot manner; requires prompts.
  • SAM 2: Integrates video and 2D images; strong mask propagation. The Cellpose team announced the integration of Cellpose-SAM (2025).

The combination of Cellpose 3 + SAM 2 is currently the most robust zero-shot approach.

Tool Stack and Infrastructure Requirements

ToolRoleLicense
Cellpose 3 (cellpose>=3.0)Image restoration + cell segmentationBSD-3-Clause
Segment Anything 2 (segment-anything-2)Mask refinement, prompt-based precision segmentationApache 2.0
napariInteractive visualization and label verificationBSD-3-Clause
scikit-imageCalculation of morphological and intensity statisticsBSD-3-Clause
tifffileReading multi-channel TIFF microscopy imagesBSD-3-Clause
numpy, pandasNumerical and tabular data processingBSD

Infrastructure requirements:

  • Cellpose 3 inference: Small consumer GPU (RTX 3060 6GB or higher recommended). CPU fallback is possible, but 10-20 times slower.
  • SAM 2: Small to medium consumer GPU (RTX 4060 8GB or higher recommended). CPU inference is very slow.
  • 16GB or more of RAM (for batch processing of 2048x2048 multi-channel images).
  • Disk: Cellpose weights approximately 30MB, SAM 2 base weights approximately 160MB, large weights approximately 900MB.

Estimated cost for learners to reproduce: API cost 0 (completely local). GPU time for processing 100 images is 5-10 minutes (estimated for RTX 4060). Dataset is open (Cellpose official example images are free).

Practical Implementation of the Pipeline

Overall Flow:

mermaid

Step 1. Loading and Preprocessing TIFF Images

Microscope images are typically multi-channel TIFFs (e.g., DAPI + FITC + TRITC). Normalization is important as the dynamic range of each channel can vary greatly.

python
from pathlib import Path
from typing import NamedTuple
import numpy as np
import tifffile
class MicroscopyImage(NamedTuple):
"""Microscope image container."""
data: np.ndarray # shape: (channels, height, width) or (height, width)
channel_names: list[str]
pixel_size_um: float # What is the physical size (in micrometers) of one pixel?
filename: str
def load_microscopy_image(
path: Path,
channel_names: list[str] | None = None,
pixel_size_um: float = 0.325,
) -> MicroscopyImage:
"""Loads TIFF images. Pixel size can be automatically extracted from OME-TIFF metadata."""
data = tifffile.imread(path)
if data.ndim == 2:
data = data[np.newaxis, ...] # (H, W) β†’ (1, H, W)
if channel_names is None:
channel_names = [f"ch{i}" for i in range(data.shape[0])]
return MicroscopyImage(
data=data,
channel_names=channel_names,
pixel_size_um=pixel_size_um,
filename=path.name,
)
def normalize_channel(
channel_data: np.ndarray,
lower_percentile: float = 1.0,
upper_percentile: float = 99.5,
) -> np.ndarray:
"""Percentile-based normalization. More robust to outliers than min-max."""
p_low = np.percentile(channel_data, lower_percentile)
p_high = np.percentile(channel_data, upper_percentile)
if p_high - p_low < 1e-6:
return np.zeros_like(channel_data, dtype=np.float32)
normalized = np.clip((channel_data - p_low) / (p_high - p_low), 0, 1)
return normalized.astype(np.float32)

Step 2. Cellpose 3 Image Restoration + Segmentation

Cellpose 3 integrates image restoration (denoise/deblur/upsample) as a front-end to the segmentation, significantly improving results for low-quality images [1].

python
from cellpose import models, io as cp_io
from cellpose.denoise import DenoiseModel
class CellposeSegmenter:
"""Integrated Cellpose 3 restoration and segmentation."""
def __init__(
self,
model_type: str = "cyto3", # cyto3: latest cell body model, nuclei: dedicated to nuclei
restore_type: str = "denoise", # denoise / deblur / upsample / None
device: str = "cuda",
):
self.model = models.CellposeModel(
gpu=(device == "cuda"),
model_type=model_type,
)
self.restore_type = restore_type
if restore_type:
self.denoiser = DenoiseModel(
model_type=f"{restore_type}_{model_type}",
gpu=(device == "cuda"),
)
else:
self.denoiser = None
def segment(
self,
image: np.ndarray,
diameter: float | None = None,
channels: list[int] = [0, 0],
cellprob_threshold: float = 0.0,
flow_threshold: float = 0.4,
) -> dict:
"""
image: (H, W) grayscale or (C, H, W) multi-channel.
diameter: Estimated cell diameter (in pixels). If None, it is automatically estimated.
channels: [cell body channel, nucleus channel] indices. [0, 0] = grayscale.
Returns: {masks, flows, styles, diams}
"""
if self.denoiser is not None:
image = self.denoiser.eval(image, channels=channels)[0]
masks, flows, styles = self.model.eval(
image,
diameter=diameter,
channels=channels,
cellprob_threshold=cellprob_threshold,
flow_threshold=flow_threshold,
)
return {
"masks": masks, # (H, W) int, 0=background, 1..N=cell ID
"flows": flows, # gradient flow visualization
"diameter_used": diameter or self.model.diam_labels,
}

Step 3. SAM 2 Refinement (Optional)

Refine boundaries that Cellpose segments ambiguously or cells that overlap using SAM 2. SAM 2 returns a refined mask based on point/box/mask prompts [2].

python
from segment_anything_2 import SAM2ImagePredictor
class SAM2Refiner:
"""Refine Cellpose masks using SAM 2."""
def __init__(self, model_id: str = "facebook/sam2-hiera-base", device: str = "cuda"):
self.predictor = SAM2ImagePredictor.from_pretrained(model_id)
self.predictor.model.to(device)
def refine_mask(
self,
image: np.ndarray,
cellpose_mask: np.ndarray,
cell_id: int,
) -> np.ndarray:
"""Refine the Cellpose mask of a specific cell ID using SAM 2."""
self.predictor.set_image(image)
# Use the bounding box and centroid of the Cellpose mask as the SAM prompt
cell_mask = (cellpose_mask == cell_id)
if not cell_mask.any():
return cell_mask
ys, xs = np.where(cell_mask)
bbox = np.array([xs.min(), ys.min(), xs.max(), ys.max()])
centroid = np.array([[xs.mean(), ys.mean()]])
refined_masks, scores, _ = self.predictor.predict(
point_coords=centroid,
point_labels=np.array([1]), # foreground
box=bbox,
multimask_output=True,
)
# Select the mask with the highest score
best_idx = scores.argmax()
return refined_masks[best_idx]
def refine_uncertain_cells(
self,
image: np.ndarray,
cellpose_result: dict,
uncertainty_threshold: float = 0.5,
) -> np.ndarray:
"""Refine only cells with low confidence in the Cellpose flow using SAM."""
masks = cellpose_result["masks"].copy()
# flows[2] is the cell probability map
cell_probs = cellpose_result["flows"][2] if len(cellpose_result["flows"]) > 2 else None
if cell_probs is None:
return masks
for cell_id in np.unique(masks):
if cell_id == 0:
continue
cell_region = (masks == cell_id)
mean_prob = cell_probs[cell_region].mean()
if mean_prob < uncertainty_threshold:
refined = self.refine_mask(image, masks, cell_id)
masks[cell_region] = 0
masks[refined] = cell_id
return masks

Step 4. Morphological and Intensity Feature Quantification

Use skimage.measure.regionprops to extract standard metrics for each cell.

python
from dataclasses import dataclass, asdict
from typing import Iterable
from skimage.measure import regionprops, regionprops_table
import pandas as pd
@dataclass
class CellFeatures:
"""Morphological and intensity features of a single cell."""
cell_id: int
area_px: int
area_um2: float
perimeter_px: float
circularity: float # 4π·area / perimeterΒ² (circular=1)
eccentricity: float # eccentricity (0=circle, 1=line)
solidity: float # area / convex_hull_area
centroid_y: float
centroid_x: float
mean_intensity_per_channel: dict[str, float]
max_intensity_per_channel: dict[str, float]
def extract_features(
mask: np.ndarray,
intensity_channels: dict[str, np.ndarray],
pixel_size_um: float,
) -> list[CellFeatures]:
"""
mask: (H, W) int, 0=background, 1..N=cell ID.
intensity_channels: {channel_name: (H, W) array}.
"""
features = []
for prop in regionprops(mask):
perimeter = prop.perimeter if prop.perimeter > 0 else 1e-6
circularity = (4 * np.pi * prop.area) / (perimeter ** 2)
mean_intensity = {
name: float(ch[prop.coords[:, 0], prop.coords[:, 1]].mean())
for name, ch in intensity_channels.items()
}
max_intensity = {
name: float(ch[prop.coords[:, 0], prop.coords[:, 1]].max())
for name, ch in intensity_channels.items()
}
features.append(CellFeatures(
cell_id=int(prop.label),
area_px=int(prop.area),
area_um2=float(prop.area * (pixel_size_um ** 2)),
perimeter_px=float(prop.perimeter),
circularity=float(circularity),
eccentricity=float(prop.eccentricity),
solidity=float(prop.solidity),
centroid_y=float(prop.centroid[0]),
centroid_x=float(prop.centroid[1]),
mean_intensity_per_channel=mean_intensity,
max_intensity_per_channel=max_intensity,
))
return features
def features_to_dataframe(features: Iterable[CellFeatures]) -> pd.DataFrame:
"""Convert to a tabular format for downstream analysis."""
rows = []
for f in features:
row = asdict(f)
for ch_name, val in row.pop("mean_intensity_per_channel").items():
row[f"mean_{ch_name}"] = val
for ch_name, val in row.pop("max_intensity_per_channel").items():
row[f"max_{ch_name}"] = val
rows.append(row)
return pd.DataFrame(rows)

Step 5. napari Visualization + Manual Inspection

napari is a real-time 4D image viewer that allows users to quickly inspect and correct the automated results [3].

python
def visualize_with_napari(
image: np.ndarray,
masks: np.ndarray,
channel_names: list[str],
) -> None:
"""Visualize image and mask overlays in napari.
A GUI window will open upon execution. On a remote server, use X11 forwarding or the napari web viewer (napari-remote).
"""
import napari
viewer = napari.Viewer()
for i, ch_name in enumerate(channel_names):
viewer.add_image(
image[i] if image.ndim == 3 else image,
name=ch_name,
colormap=["green", "red", "blue"][i % 3],
blending="additive",
)
viewer.add_labels(masks, name="cell masks", opacity=0.5)
napari.run()

Integrated Pipeline

python
def full_pipeline(
image_path: Path,
channel_names: list[str],
pixel_size_um: float,
output_csv: Path,
use_sam_refinement: bool = False,
device: str = "cuda",
) -> pd.DataFrame:
"""Image (1 image) β†’ Cell feature DataFrame."""
img = load_microscopy_image(image_path, channel_names, pixel_size_um)
# Normalize the cell body channel (e.g., FITC)
cell_channel_idx = channel_names.index("FITC") if "FITC" in channel_names else 0
normalized = normalize_channel(img.data[cell_channel_idx])
segmenter = CellposeSegmenter(model_type="cyto3", restore_type="denoise", device=device)
result = segmenter.segment(normalized)
masks = result["masks"]
if use_sam_refinement:
refiner = SAM2Refiner(device=device)
masks = refiner.refine_uncertain_cells(normalized, result)
intensity_channels = {
name: normalize_channel(img.data[i])
for i, name in enumerate(channel_names)
}
features = extract_features(masks, intensity_channels, pixel_size_um)
df = features_to_dataframe(features)
df["source_image"] = img.filename
df.to_csv(output_csv, index=False)
return df
## Performance, Cost, and Known Failure Cases
### Performance Reference (Using Public Benchmarks)
| Approach | Dataset | F1 (Cell Detection) | Processing Time | Source |
|---|---|---|---|---|
| Otsu + Watershed | Cellpose test set | 0.62 | CPU < 1s | Legacy baseline |
| StarDist | LIVECell | 0.71 | GPU 1~2s | Schmidt et al., MICCAI 2018 |
| Cellpose 2 | Cellpose test set | 0.86 | GPU 2~4s | Pachitariu & Stringer, Nat Methods 2022 |
| Cellpose 3 (with denoise) | Cellpose test set + low-SNR | 0.91 | GPU 3~5s | Stringer & Pachitariu, Nat Methods 2025 [1] |
| Cellpose-SAM | LIVECell + Cellpose test | 0.93~0.95 | GPU 5~8s | Cellpose team 2025 presentation (estimated) |
| SAM 2 stand-alone (bio application) | LIVECell | 0.84 (without prompt) | GPU 3~5s | Meta AI 2024 [2] |
### Estimated Cost for Learner Reproduction
- API cost: 0 (completely local).
- Based on a small consumer GPU (RTX 4060 8GB), processing approximately 100 images takes 5~10 minutes.
- With CPU fallback, processing 100 images takes 1~2 hours.
- Data download: Cellpose example images are free, and the LIVECell dataset is publicly available.
### Three Known Failure Cases (Collected from Community and Papers)
1. **Under-segmentation of adjacent cells in dense cell populations**
- Symptoms: When cells are close together, they are merged into a single large mask.
- Cause: Cellpose's flow field fails to capture the gradient signal at the cell boundaries. This is more severe in low SNR conditions.
- Mitigation: (a) Lower the `flow_threshold` (e.g., 0.2) to recognize more flow as part of a cell, (b) adjust the `cellprob_threshold`, (c) use SAM 2 to refine only the uncertain areas.
- Source: Numerous threads on Cellpose GitHub Issues β€” "over/under segmentation" [4].
2. **Failure with non-typical cell morphologies (e.g., neuronal axons/dendrites)**
- Symptoms: The cyto/nuclei model is trained to recognize round shapes, and therefore misses cells with long axons.
- Cause: Bias in the training data.
- Mitigation: (a) Use the `livecell` or `neurips` model, (b) fine-tune with your own data (very easy to do in Cellpose 3's GUI).
- Source: Cellpose official model zoo documentation and community reports [5].
3. **Channel assignment errors in multi-channel images**
- Symptoms: Setting `channels=[0,0]` (grayscale) when the nuclei channel should actually be specified separately for accurate results.
- Cause: Cellpose accepts channel assignments in the form of `[cyto_channel, nuclei_channel]`, and performance degrades significantly if this is done incorrectly.
- Mitigation: Verify the image channel order and specify it accurately. If DAPI is present, it must be specified as the nuclei channel.
- Source: Cellpose official documentation, "channels" section [6].
## Expansion Ideas
- **Time-series live imaging:** Utilize SAM 2's video mask propagation to enable cell tracking and automatic detection of division events.
- **3D confocal stack:** Use Cellpose 3's 3D mode + refine each slice of the z-stack with SAM 2.
- **Phenotypic screening:** Input morphological statistics into a downstream classifier (e.g., XGBoost) to automatically classify drug responses.
- **QuPath integration:** For large whole slide images, split into tiles with QuPath, process each tile with this pipeline, and then merge the results.
- **napari plugin packaging:** Package this pipeline as a napari plugin and deploy it to laboratory GUI users.
## Next Episode
- Episode 05 `pathology-image-embedding`: Embed cell crops extracted in this episode using a pathology foundation model (Virchow) for similar case retrieval.
- Episode 07 `drug-target-gnn`: Use cell morphology and intensity statistics as input for a drug response prediction GNN.
- Episode 12 `single-cell-perturbation`: Apply in silico gene perturbation predictions to automatically segmented cells.
- Episode 14 `bio-mcp-agent`: Expose this pipeline as an MCP tool to enable autonomous queries such as "count the cells in this image."
## References
1. Stringer C, Pachitariu M. "Cellpose3: one-click image restoration for improved cellular segmentation." Nature Methods 2025. `https://www.nature.com/articles/s41592-025-02595-5`
2. Ravi N, Gabeur V, Hu Y-T, et al. "SAM 2: Segment Anything in Images and Videos." Meta AI 2024. `https://ai.meta.com/research/publications/sam-2-segment-anything-in-images-and-videos/`
3. napari documentation: `https://napari.org/stable/`
4. Cellpose GitHub Issues (over/under segmentation): `https://github.com/MouseLand/cellpose/issues`
5. Cellpose Model Zoo: `https://cellpose.readthedocs.io/en/latest/models.html`
6. Cellpose channels documentation: `https://cellpose.readthedocs.io/en/latest/inputs.html`
7. Pachitariu M, Stringer C. "Cellpose 2.0: how to train your own model." Nature Methods 2022. `https://www.nature.com/articles/s41592-022-01663-4`
8. Schmidt U, Weigert M, Broaddus C, Myers G. "Cell Detection with Star-Convex Polygons (StarDist)." MICCAI 2018.
9. LIVECell dataset: `https://sartorius-research.github.io/LIVECell/`
10. Cellpose GitHub: `https://github.com/MouseLand/cellpose`
11. Segment Anything 2 GitHub: `https://github.com/facebookresearch/segment-anything-2`
12. scikit-image regionprops: `https://scikit-image.org/docs/stable/api/skimage.measure.html`
13. QuPath (whole slide image analysis): `https://qupath.github.io/`
14. Human Protein Atlas (example images): `https://www.proteinatlas.org/`
15. Broad Bioimage Benchmark Collection: `https://bbbc.broadinstitute.org/`

πŸ’¬ Questions & Comments

0 comments

You can post without signing in. Guest comments cannot be edited or deleted by their author.

0/2000

Loading...