Cellpose 3 + SAM Cell Segmentation Pipeline: A Practical Guide to Identifying Individual Cells in Fluorescence Microscopy Images
Anyone who has manually traced cell boundaries in fluorescence microscopy images will immediately understand why this guide is necessary. If you have 500 cells per slide and 100 slides per day, labeling 50,000 cells manually will take days. However, quantitative analysis of cell shape, area, and intensity is a fundamental input for downstream analysis (drug response, phenotypic screening, histopathology), so addressing this bottleneck will accelerate the entire experimental cycle. This guide presents a practical pipeline that combines the image restoration capabilities of Cellpose 3 and Meta's Segment Anything 2 to reduce this process to about 30 minutes.
π Recommended Prerequisite
This is an advanced AI x Biology topic. We strongly recommend that you first review the following DryBench tutorials before diving in.
Without the prerequisites, it will be difficult to follow the practical code, as this guide will proceed without re-explaining the principles of U-Net-based encoder-decoder architectures and PyTorch tensor/GPU transfer.
We Already Learned This in DryBench
In DryBench ai-native #2, we learned how neural networks learn representations by repeatedly applying weighted multiplication and non-linear activation functions to inputs, and in particular, why U-Net-based encoder-decoder architectures are strong for pixel-by-pixel prediction (segmentation) of images. In #12, we covered the basics of PyTorch tensor manipulation, GPU transfer, and model forward passes.
However, real-world experimental images present some additional challenges. These include uneven illumination (vignetting), out-of-focus blur, cell crowding, color shifts between channels, and low signal-to-noise ratio (SNR) in fluorescence conditions. We will examine in practice how robust pretrained CNNs are under these conditions, and how the latest foundation models (SAM 2) can further enhance this robustness.
Defining the Problem
Practical Requirements for Cell Segmentation
From a single fluorescence image (2048x2048), we need to automatically extract the following four pieces of information:
- Instance Segmentation: Unique ID + pixel mask for each cell. Distinguish overlapping cells.
- Morphological Statistics: Area (pxΒ²), circularity, and eccentricity for each cell.
- Intensity Statistics: Mean, maximum, and standard deviation of fluorescence channels for each cell.
- Spatial Relationships: Distance between neighboring cells, whether they form clusters.
Target metrics:
- Cell detection recall β₯ 0.92 (compared to human labeling).
- Cell detection precision β₯ 0.90.
- Processing speed: 3-5 seconds per GPU, 30-60 seconds per CPU for a 2048x2048 image.
Spectrum of Existing Approaches
- Classical rule-based (Otsu threshold + watershed): Recall of 0.7-0.8 for low-density, high-SNR images; often fails for overlapping cells.
- StarDist (star-convex polygon): Strong for segmenting round nuclei; weak for complex cell bodies.
- Cellpose 1/2 (U-Net + flow field): Covers a wide range of cell shapes; strong zero-shot performance with pre-trained weights. Cellpose 3 added an image restoration backbone in 2024.
- SAM (Segment Anything Model): Trained on natural images, but surprisingly effective for bioimages in a zero-shot manner; requires prompts.
- SAM 2: Integrates video and 2D images; strong mask propagation. The Cellpose team announced the integration of Cellpose-SAM (2025).
The combination of Cellpose 3 + SAM 2 is currently the most robust zero-shot approach.
Tool Stack and Infrastructure Requirements
| Tool | Role | License |
|---|---|---|
Cellpose 3 (cellpose>=3.0) | Image restoration + cell segmentation | BSD-3-Clause |
Segment Anything 2 (segment-anything-2) | Mask refinement, prompt-based precision segmentation | Apache 2.0 |
| napari | Interactive visualization and label verification | BSD-3-Clause |
| scikit-image | Calculation of morphological and intensity statistics | BSD-3-Clause |
| tifffile | Reading multi-channel TIFF microscopy images | BSD-3-Clause |
| numpy, pandas | Numerical and tabular data processing | BSD |
Infrastructure requirements:
- Cellpose 3 inference: Small consumer GPU (RTX 3060 6GB or higher recommended). CPU fallback is possible, but 10-20 times slower.
- SAM 2: Small to medium consumer GPU (RTX 4060 8GB or higher recommended). CPU inference is very slow.
- 16GB or more of RAM (for batch processing of 2048x2048 multi-channel images).
- Disk: Cellpose weights approximately 30MB, SAM 2 base weights approximately 160MB, large weights approximately 900MB.
Estimated cost for learners to reproduce: API cost 0 (completely local). GPU time for processing 100 images is 5-10 minutes (estimated for RTX 4060). Dataset is open (Cellpose official example images are free).
Practical Implementation of the Pipeline
Overall Flow:
Step 1. Loading and Preprocessing TIFF Images
Microscope images are typically multi-channel TIFFs (e.g., DAPI + FITC + TRITC). Normalization is important as the dynamic range of each channel can vary greatly.
from pathlib import Pathfrom typing import NamedTuple
import numpy as npimport tifffile
class MicroscopyImage(NamedTuple): """Microscope image container.""" data: np.ndarray # shape: (channels, height, width) or (height, width) channel_names: list[str] pixel_size_um: float # What is the physical size (in micrometers) of one pixel? filename: str
def load_microscopy_image( path: Path, channel_names: list[str] | None = None, pixel_size_um: float = 0.325,) -> MicroscopyImage: """Loads TIFF images. Pixel size can be automatically extracted from OME-TIFF metadata.""" data = tifffile.imread(path) if data.ndim == 2: data = data[np.newaxis, ...] # (H, W) β (1, H, W) if channel_names is None: channel_names = [f"ch{i}" for i in range(data.shape[0])] return MicroscopyImage( data=data, channel_names=channel_names, pixel_size_um=pixel_size_um, filename=path.name, )
def normalize_channel( channel_data: np.ndarray, lower_percentile: float = 1.0, upper_percentile: float = 99.5,) -> np.ndarray: """Percentile-based normalization. More robust to outliers than min-max.""" p_low = np.percentile(channel_data, lower_percentile) p_high = np.percentile(channel_data, upper_percentile) if p_high - p_low < 1e-6: return np.zeros_like(channel_data, dtype=np.float32) normalized = np.clip((channel_data - p_low) / (p_high - p_low), 0, 1) return normalized.astype(np.float32)Step 2. Cellpose 3 Image Restoration + Segmentation
Cellpose 3 integrates image restoration (denoise/deblur/upsample) as a front-end to the segmentation, significantly improving results for low-quality images [1].
from cellpose import models, io as cp_iofrom cellpose.denoise import DenoiseModel
class CellposeSegmenter: """Integrated Cellpose 3 restoration and segmentation."""
def __init__( self, model_type: str = "cyto3", # cyto3: latest cell body model, nuclei: dedicated to nuclei restore_type: str = "denoise", # denoise / deblur / upsample / None device: str = "cuda", ): self.model = models.CellposeModel( gpu=(device == "cuda"), model_type=model_type, ) self.restore_type = restore_type if restore_type: self.denoiser = DenoiseModel( model_type=f"{restore_type}_{model_type}", gpu=(device == "cuda"), ) else: self.denoiser = None
def segment( self, image: np.ndarray, diameter: float | None = None, channels: list[int] = [0, 0], cellprob_threshold: float = 0.0, flow_threshold: float = 0.4, ) -> dict: """ image: (H, W) grayscale or (C, H, W) multi-channel. diameter: Estimated cell diameter (in pixels). If None, it is automatically estimated. channels: [cell body channel, nucleus channel] indices. [0, 0] = grayscale. Returns: {masks, flows, styles, diams} """ if self.denoiser is not None: image = self.denoiser.eval(image, channels=channels)[0] masks, flows, styles = self.model.eval( image, diameter=diameter, channels=channels, cellprob_threshold=cellprob_threshold, flow_threshold=flow_threshold, ) return { "masks": masks, # (H, W) int, 0=background, 1..N=cell ID "flows": flows, # gradient flow visualization "diameter_used": diameter or self.model.diam_labels, }Step 3. SAM 2 Refinement (Optional)
Refine boundaries that Cellpose segments ambiguously or cells that overlap using SAM 2. SAM 2 returns a refined mask based on point/box/mask prompts [2].
from segment_anything_2 import SAM2ImagePredictor
class SAM2Refiner: """Refine Cellpose masks using SAM 2."""
def __init__(self, model_id: str = "facebook/sam2-hiera-base", device: str = "cuda"): self.predictor = SAM2ImagePredictor.from_pretrained(model_id) self.predictor.model.to(device)
def refine_mask( self, image: np.ndarray, cellpose_mask: np.ndarray, cell_id: int, ) -> np.ndarray: """Refine the Cellpose mask of a specific cell ID using SAM 2.""" self.predictor.set_image(image) # Use the bounding box and centroid of the Cellpose mask as the SAM prompt cell_mask = (cellpose_mask == cell_id) if not cell_mask.any(): return cell_mask ys, xs = np.where(cell_mask) bbox = np.array([xs.min(), ys.min(), xs.max(), ys.max()]) centroid = np.array([[xs.mean(), ys.mean()]]) refined_masks, scores, _ = self.predictor.predict( point_coords=centroid, point_labels=np.array([1]), # foreground box=bbox, multimask_output=True, ) # Select the mask with the highest score best_idx = scores.argmax() return refined_masks[best_idx]
def refine_uncertain_cells( self, image: np.ndarray, cellpose_result: dict, uncertainty_threshold: float = 0.5, ) -> np.ndarray: """Refine only cells with low confidence in the Cellpose flow using SAM.""" masks = cellpose_result["masks"].copy() # flows[2] is the cell probability map cell_probs = cellpose_result["flows"][2] if len(cellpose_result["flows"]) > 2 else None if cell_probs is None: return masks for cell_id in np.unique(masks): if cell_id == 0: continue cell_region = (masks == cell_id) mean_prob = cell_probs[cell_region].mean() if mean_prob < uncertainty_threshold: refined = self.refine_mask(image, masks, cell_id) masks[cell_region] = 0 masks[refined] = cell_id return masksStep 4. Morphological and Intensity Feature Quantification
Use skimage.measure.regionprops to extract standard metrics for each cell.
from dataclasses import dataclass, asdictfrom typing import Iterable
from skimage.measure import regionprops, regionprops_tableimport pandas as pd
@dataclassclass CellFeatures: """Morphological and intensity features of a single cell.""" cell_id: int area_px: int area_um2: float perimeter_px: float circularity: float # 4ΟΒ·area / perimeterΒ² (circular=1) eccentricity: float # eccentricity (0=circle, 1=line) solidity: float # area / convex_hull_area centroid_y: float centroid_x: float mean_intensity_per_channel: dict[str, float] max_intensity_per_channel: dict[str, float]
def extract_features( mask: np.ndarray, intensity_channels: dict[str, np.ndarray], pixel_size_um: float,) -> list[CellFeatures]: """ mask: (H, W) int, 0=background, 1..N=cell ID. intensity_channels: {channel_name: (H, W) array}. """ features = [] for prop in regionprops(mask): perimeter = prop.perimeter if prop.perimeter > 0 else 1e-6 circularity = (4 * np.pi * prop.area) / (perimeter ** 2) mean_intensity = { name: float(ch[prop.coords[:, 0], prop.coords[:, 1]].mean()) for name, ch in intensity_channels.items() } max_intensity = { name: float(ch[prop.coords[:, 0], prop.coords[:, 1]].max()) for name, ch in intensity_channels.items() } features.append(CellFeatures( cell_id=int(prop.label), area_px=int(prop.area), area_um2=float(prop.area * (pixel_size_um ** 2)), perimeter_px=float(prop.perimeter), circularity=float(circularity), eccentricity=float(prop.eccentricity), solidity=float(prop.solidity), centroid_y=float(prop.centroid[0]), centroid_x=float(prop.centroid[1]), mean_intensity_per_channel=mean_intensity, max_intensity_per_channel=max_intensity, )) return features
def features_to_dataframe(features: Iterable[CellFeatures]) -> pd.DataFrame: """Convert to a tabular format for downstream analysis.""" rows = [] for f in features: row = asdict(f) for ch_name, val in row.pop("mean_intensity_per_channel").items(): row[f"mean_{ch_name}"] = val for ch_name, val in row.pop("max_intensity_per_channel").items(): row[f"max_{ch_name}"] = val rows.append(row) return pd.DataFrame(rows)Step 5. napari Visualization + Manual Inspection
napari is a real-time 4D image viewer that allows users to quickly inspect and correct the automated results [3].
def visualize_with_napari( image: np.ndarray, masks: np.ndarray, channel_names: list[str],) -> None: """Visualize image and mask overlays in napari.
A GUI window will open upon execution. On a remote server, use X11 forwarding or the napari web viewer (napari-remote). """ import napari
viewer = napari.Viewer() for i, ch_name in enumerate(channel_names): viewer.add_image( image[i] if image.ndim == 3 else image, name=ch_name, colormap=["green", "red", "blue"][i % 3], blending="additive", ) viewer.add_labels(masks, name="cell masks", opacity=0.5) napari.run()Integrated Pipeline
def full_pipeline( image_path: Path, channel_names: list[str], pixel_size_um: float, output_csv: Path, use_sam_refinement: bool = False, device: str = "cuda",) -> pd.DataFrame: """Image (1 image) β Cell feature DataFrame.""" img = load_microscopy_image(image_path, channel_names, pixel_size_um)
# Normalize the cell body channel (e.g., FITC) cell_channel_idx = channel_names.index("FITC") if "FITC" in channel_names else 0 normalized = normalize_channel(img.data[cell_channel_idx])
segmenter = CellposeSegmenter(model_type="cyto3", restore_type="denoise", device=device) result = segmenter.segment(normalized) masks = result["masks"]
if use_sam_refinement: refiner = SAM2Refiner(device=device) masks = refiner.refine_uncertain_cells(normalized, result)
intensity_channels = { name: normalize_channel(img.data[i]) for i, name in enumerate(channel_names) } features = extract_features(masks, intensity_channels, pixel_size_um) df = features_to_dataframe(features) df["source_image"] = img.filename df.to_csv(output_csv, index=False) return df
## Performance, Cost, and Known Failure Cases
### Performance Reference (Using Public Benchmarks)
| Approach | Dataset | F1 (Cell Detection) | Processing Time | Source ||---|---|---|---|---|| Otsu + Watershed | Cellpose test set | 0.62 | CPU < 1s | Legacy baseline || StarDist | LIVECell | 0.71 | GPU 1~2s | Schmidt et al., MICCAI 2018 || Cellpose 2 | Cellpose test set | 0.86 | GPU 2~4s | Pachitariu & Stringer, Nat Methods 2022 || Cellpose 3 (with denoise) | Cellpose test set + low-SNR | 0.91 | GPU 3~5s | Stringer & Pachitariu, Nat Methods 2025 [1] || Cellpose-SAM | LIVECell + Cellpose test | 0.93~0.95 | GPU 5~8s | Cellpose team 2025 presentation (estimated) || SAM 2 stand-alone (bio application) | LIVECell | 0.84 (without prompt) | GPU 3~5s | Meta AI 2024 [2] |
### Estimated Cost for Learner Reproduction
- API cost: 0 (completely local).- Based on a small consumer GPU (RTX 4060 8GB), processing approximately 100 images takes 5~10 minutes.- With CPU fallback, processing 100 images takes 1~2 hours.- Data download: Cellpose example images are free, and the LIVECell dataset is publicly available.
### Three Known Failure Cases (Collected from Community and Papers)
1. **Under-segmentation of adjacent cells in dense cell populations** - Symptoms: When cells are close together, they are merged into a single large mask. - Cause: Cellpose's flow field fails to capture the gradient signal at the cell boundaries. This is more severe in low SNR conditions. - Mitigation: (a) Lower the `flow_threshold` (e.g., 0.2) to recognize more flow as part of a cell, (b) adjust the `cellprob_threshold`, (c) use SAM 2 to refine only the uncertain areas. - Source: Numerous threads on Cellpose GitHub Issues β "over/under segmentation" [4].
2. **Failure with non-typical cell morphologies (e.g., neuronal axons/dendrites)** - Symptoms: The cyto/nuclei model is trained to recognize round shapes, and therefore misses cells with long axons. - Cause: Bias in the training data. - Mitigation: (a) Use the `livecell` or `neurips` model, (b) fine-tune with your own data (very easy to do in Cellpose 3's GUI). - Source: Cellpose official model zoo documentation and community reports [5].
3. **Channel assignment errors in multi-channel images** - Symptoms: Setting `channels=[0,0]` (grayscale) when the nuclei channel should actually be specified separately for accurate results. - Cause: Cellpose accepts channel assignments in the form of `[cyto_channel, nuclei_channel]`, and performance degrades significantly if this is done incorrectly. - Mitigation: Verify the image channel order and specify it accurately. If DAPI is present, it must be specified as the nuclei channel. - Source: Cellpose official documentation, "channels" section [6].
## Expansion Ideas
- **Time-series live imaging:** Utilize SAM 2's video mask propagation to enable cell tracking and automatic detection of division events.- **3D confocal stack:** Use Cellpose 3's 3D mode + refine each slice of the z-stack with SAM 2.- **Phenotypic screening:** Input morphological statistics into a downstream classifier (e.g., XGBoost) to automatically classify drug responses.- **QuPath integration:** For large whole slide images, split into tiles with QuPath, process each tile with this pipeline, and then merge the results.- **napari plugin packaging:** Package this pipeline as a napari plugin and deploy it to laboratory GUI users.
## Next Episode
- Episode 05 `pathology-image-embedding`: Embed cell crops extracted in this episode using a pathology foundation model (Virchow) for similar case retrieval.- Episode 07 `drug-target-gnn`: Use cell morphology and intensity statistics as input for a drug response prediction GNN.- Episode 12 `single-cell-perturbation`: Apply in silico gene perturbation predictions to automatically segmented cells.- Episode 14 `bio-mcp-agent`: Expose this pipeline as an MCP tool to enable autonomous queries such as "count the cells in this image."
## References
1. Stringer C, Pachitariu M. "Cellpose3: one-click image restoration for improved cellular segmentation." Nature Methods 2025. `https://www.nature.com/articles/s41592-025-02595-5`2. Ravi N, Gabeur V, Hu Y-T, et al. "SAM 2: Segment Anything in Images and Videos." Meta AI 2024. `https://ai.meta.com/research/publications/sam-2-segment-anything-in-images-and-videos/`3. napari documentation: `https://napari.org/stable/`4. Cellpose GitHub Issues (over/under segmentation): `https://github.com/MouseLand/cellpose/issues`5. Cellpose Model Zoo: `https://cellpose.readthedocs.io/en/latest/models.html`6. Cellpose channels documentation: `https://cellpose.readthedocs.io/en/latest/inputs.html`7. Pachitariu M, Stringer C. "Cellpose 2.0: how to train your own model." Nature Methods 2022. `https://www.nature.com/articles/s41592-022-01663-4`8. Schmidt U, Weigert M, Broaddus C, Myers G. "Cell Detection with Star-Convex Polygons (StarDist)." MICCAI 2018.9. LIVECell dataset: `https://sartorius-research.github.io/LIVECell/`10. Cellpose GitHub: `https://github.com/MouseLand/cellpose`11. Segment Anything 2 GitHub: `https://github.com/facebookresearch/segment-anything-2`12. scikit-image regionprops: `https://scikit-image.org/docs/stable/api/skimage.measure.html`13. QuPath (whole slide image analysis): `https://qupath.github.io/`14. Human Protein Atlas (example images): `https://www.proteinatlas.org/`15. Broad Bioimage Benchmark Collection: `https://bbbc.broadinstitute.org/`