Multiple Testing and Shrinkage

Upon completing this topic

You will be able to define families and error criteria when testing many features simultaneously. You will understand shrinkage as a method to stabilize individual estimates, and you will not combine adjusted p-values, effect sizes, and uncertainty into a single ranking.

More features mean more questions

As the number of genes or features increases, a hypothesis can be formulated for each feature. The error structure differs between interpreting a single p-value and interpreting thousands together. First, record what constitutes a family, whether the approach is exploratory or confirmatory, and how follow-up validation will be conducted.

The roles of correction and shrinkage

Multiple testing correction manages error criteria across multiple hypotheses. Shrinkage is an approach that combines highly variable individual estimates with overall information to produce more stable estimates. Shrinkage is neither a mechanism that generates small p-values nor a guarantee of biological significance.

API and result manifest

from statsmodels.stats.multitest import multipletests

reject, adjusted_p, _, _ = multipletests(
    raw_p_values, method="fdr_bh", alpha=0.05
)
result = {
    "feature_id": feature_ids,
    "estimate": estimates,
    "raw_p": raw_p_values,
    "adjusted_p": adjusted_p,
    "family_id": ["F01"] * len(feature_ids),
}

Preserve the row correspondence of feature_id, estimate, and p-value, and record filtering, normalization, batch, and family definitions together. Do not mix raw counts with normalized values, and retain the results without losing the input matrix.

Ranking and interpretation

A list sorted by adjusted p-value can be used to identify discovery candidates. However, a high rank does not imply a large effect size or a validated mechanism. Examine the units and intervals of effect estimates, feature quality, and batch and replicate designs together.

Returning to research questions for interpretation

Multiple testing is a tool for managing the risk of false discovery when the number of features is large. Results should not end with a "list of significant features" but must include the scope of effects, uncertainty, reproducibility, and follow-up validation.

Common failures

• Do not confirm features based solely on raw p-value rankings.
• Do not write that FDR correction resolves batch and design issues.
• Do not describe shrinkage as biological amplification of effects.
• Do not lose the mapping between feature-specific results and family IDs.

Key takeaways

• Multiple testing requires defining the family and error criteria first.
• Correction and shrinkage serve different roles: error management and estimation stabilization, respectively.
• Feature rankings do not substitute for effect sizes or biological significance.

Next topic

The next module connects statistics to machine learning by linking prediction targets with baselines and metrics.

References

• statsmodels multiple testing API: https://www.statsmodels.org/stable/generated/statsmodels.stats.multitest.multipletests.html
• NCBI GEO: https://www.ncbi.nlm.nih.gov/geo/info/
• Bioconductor OSTA experimental design: https://bioconductor.org/books/release/OSTA/pages/bkg-exp-design.html

The feature manifest and result structure in this chapter were written independently by BioStatPy.