ANOVA · Multiple Comparisons · FDR

Upon Completing This Topic

You will be able to distinguish between the overall question and individual pairwise comparisons when comparing three or more groups. You will be able to define the family in multiple comparisons and explain that p-value correction does not substitute for issues related to effect size and study design.

As the Number of Comparisons Increases, So Do the Questions

When comparing conditions A, B, and C, the overall question "Are the means of all conditions equal?" is different from the pairwise question "Are A and B different?" Performing multiple pairwise comparisons accumulates the opportunity to obtain extreme results by chance.

ANOVA is a form of model-based question that compares multiple means simultaneously. A significant result from the overall test indicates the possibility that at least one difference exists, but it does not tell you which pairs differ or how. Subsequent comparisons and multiplicity management are required separately.

Defining the Family First

The starting point for multiple testing adjustment is not the correction function, but the family. You must pre-record what the bundle of hypotheses to be interpreted together in this study is, and how outcomes, time points, conditions, and gene sets are grouped. The choice to widen or narrow the scope of the family changes the scope of result interpretation.

FDR is a criterion aimed at managing the expected proportion of false discoveries across multiple tests. It is not a goal like family-wise error, and the choice of criterion depends on the research objective and the cost of errors.

The Role of the Python Correction API

The multiple testing API in statsmodels takes an array of p-values and a correction method, returning the corrected results. The API does not create the family of hypotheses, nor does it judge which comparisons are meaningful.

from statsmodels.stats.multitest import multipletests

raw_p = [0.001, 0.012, 0.031, 0.20]
reject, adjusted_p, _, _ = multipletests(
    raw_p, method="fdr_bh", alpha=0.05
)

The array in this example consists of educational numbers and is not actual research data. Each position in raw_p must be linked to a separately defined manifest indicating which pre-defined hypothesis it represents. A smaller corrected p-value does not imply a large effect size.

Separating Exploration and Confirmation

In the exploratory stage, where many conditions and outcomes are examined, multiple comparisons arise naturally. Patterns discovered here should be recorded as subsequent confirmation questions and not packaged as confirmatory evidence at the same level as the primary family defined from the start. Correction is not a procedure that prohibits discovery, but a procedure that transparently expresses the scope of discovery and the risk of error.

Each hypothesis should be assigned a comparison target, outcome, unit of analysis, direction, and family ID. Even with the same raw p-value array, if the family differs, the correction results and the scope of interpretation may vary. Therefore, do not copy only the results; preserve the hypothesis manifest along with the correction method and alpha.

Reading Results from a Table

The reject in the table represents the decision under that specific correction criterion, not the biological importance of the effect. If effect size and confidence intervals are absent, there is insufficient information for subsequent judgment.

Returning to Research Questions for Interpretation

Multiple testing adjustment is a tool for managing error criteria when viewing multiple hypotheses together. A small corrected p-value does not confirm biological importance or causation. Report the effect size, uncertainty, measurement unit, and family definition for each result.

The Relationship Between Overall Tests and Subsequent Comparisons

Even if a signal of difference is observed in the overall test, the number of subsequent pairwise comparisons and the family must be determined separately. Do not assume that the results of subsequent comparisons answer the same question as the overall test. The overall test asks about the existence of a general difference, while pairwise comparisons ask between which conditions and in what direction the difference lies.

Whether to apply the same correction criterion to all outcomes and time points, or to distinguish between primary and secondary outcomes, is decided in the planning stage. Changing the family after seeing the results makes it difficult to explain the meaning of overall error management, even if corrected numbers are present.

In the report, retain the original p-value, correction method, and family ID together. If only corrected results are provided, readers cannot verify which bundle of hypotheses they were calculated from. Correction must be used alongside records that enhance the transparency of the analysis.

Common Failures and Checks

• Do not report multiple pairwise comparisons as if they were a single test.
• Do not change the family definition favorably after seeing the results.
• Do not use FDR and family-wise error rate as the same criterion.
• Do not report only corrected p-values while omitting effect sizes.
• Do not claim that overall ANOVA results automatically indicate the direction and magnitude of specific pairwise comparisons.

Key Takeaways

• The overall comparison question and the pairwise comparison question are different.
• Managing multiplicity requires defining the family first.
• FDR is an error management criterion and does not indicate the magnitude or importance of effects.
• The API performs correction calculations but does not determine the meaning of hypotheses.

Next Topic

In the next section, we prospectively plan power and sample size, considering effect size, variability, and dropout.

References

• statsmodels multiple testing API: https://www.statsmodels.org/stable/generated/statsmodels.stats.multitest.multipletests.html
• ASA p-value statement: https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf
• SciPy statistics reference: https://docs.scipy.org/doc/scipy/reference/stats.html

The hypothesis list and values in this section are educational constructs written independently by BioStatPy.