Power and Sample Size Planning

Upon completing this topic

In power planning, you can separate and record effect size, variability, significance level, sample size, and design. Distinguish between post-hoc interpretation after observation and prospective design planning, and do not treat sample size calculations as numbers that resolve all uncertainties in the study.

Questions to be decided before calculating sample size

The question "How many are needed?" cannot be answered by the name of the analysis method alone. You must first decide what difference constitutes a meaningful effect, the extent of outcome variability, how to anticipate dropout and missingness, and what the independent unit is.

Power is the concept of considering the probability of rejecting the null hypothesis under specific effects and design assumptions. If the effect is small or variability is large, more independent units may be required for the same power. However, increasing only technical replicates is not equivalent to increasing the number of biological replicates.

Items to record in prospective planning

These items serve as inputs for entering numbers into a calculator and as a checklist for reviewing research questions and design.

The Python API calculates the plan

statsmodels provides power calculation APIs for specific tests and designs. Before running calculations, verify in the official documentation which assumptions and effect scales each function uses.

from statsmodels.stats.power import TTestIndPower

analysis = TTestIndPower()
required_n = analysis.solve_power(
    effect_size=0.5,
    alpha=0.05,
    power=0.80,
    ratio=1.0,
    alternative="two-sided",
)
print(required_n)

This number is an educational example calculated under specific effect scales for two independent groups and the assumptions of the corresponding API. Do not use it as-is, even if the actual outcome units, variability, replication structure, dropout, and multiple comparisons differ. When rounding calculation results, record them based on independent units and group assignments.

Distinguishing planning from post-observation analysis

Recalculating power using observed effects and sample sizes after a study ends, and presenting it as "the probability of this study," can lead to unstable interpretations. Prospective power planning is the process of designing based on effects and design assumptions that have not yet been observed, whereas after observation, the priority is to report the estimated values and uncertainties of the actual effects.

Increasing the sample size infinitely is not a solution if the plan fails. You must re-evaluate measurement quality, independent units, causes of dropout, and the importance of the question, and record the reasons for any changes.

Sensitivity planning for assumptions

Sample size calculations using only a single effect size and variability value can hide the uncertainty of input values. You can record in tables or graphs how the required independent units change by varying effect size, variability, and dropout rates within reasonable ranges defined in advance. This process is not about adjusting values until a desired number is reached, but about confirming which assumptions the plan is sensitive to.

For example, record the minimum important effect and a larger expected effect as separate scenarios, noting the basis for each scenario. When correcting for dropout, distinguish between the independent units that will actually be assigned to conditions and the units that will remain in the analysis, rather than simply increasing the number of measurement rows.

Documenting plan changes

If the design changes during the study, do not overwrite the original power plan. Record the date of change, reason for change, affected outcomes/families/analysis units, and new calculations as a separate version. The fact that the plan differs from the actual analysis is not an error in itself; the problem arises when changes cannot be tracked, hindering reproducibility and interpretation.

There are also elements not calculated in power planning. Selection bias, measurement accuracy, model assumption violations, and data provenance issues are not resolved by sample size alone. Record the limitations of the sample size output in the planning document.

Returning to research questions for interpretation

Power planning is a tool for explicitly stating in advance "what difference can be distinguished with what design" rather than "sample size needed to obtain significant results." Even if the calculated sample size appears sufficient, it does not correct for biased designs or incorrect units.

Questions to ask when reading sample size calculations

Upon receiving calculation results, do not immediately copy the numbers; instead, verify the following:

• On what scale of the original outcome was the effect size derived?
• What is the basis for the variance: pilot data, past data, literature, or assumption?
• Does the direction of the two-sided or one-sided test align with the plan?
• Do the group ratio and the actual allocation ratio match?
• Have independent units and technical replicates been distinguished?
• Is the minimum unit for analysis secured even after accounting for dropouts?

You must be able to answer these questions to transfer the calculated values to the design document. If you increase the sample size without knowing the answers, the precision of the numbers may be mistaken for the validity of the plan.

Sample size and ethical/resource judgment

More units do not always mean a better study. Research resources, measurement burden, sample limitations, dropout potential, and the value of the question must be considered together. This convenience statistical calculation does not make such judgments for you; it is used to make the assumptions and change history of the plan transparent.

Common pitfalls and checks

• Do not increase n to the number of technical measurements.
• Do not disguise post-hoc effect sizes as prospective plans.
• Do not declare power 0.80 as a universal standard for all studies.
• Do not omit multiple comparisons and dropout from the plan.
• Do not disconnect the API's effect scale from the study outcome's unit.

Key takeaways

• Power is a function of effect size, variability, sample size, error criteria, and design.
• Sample size is planned based on independent analysis units.
• Power APIs calculate under specific assumptions and do not replace study design.
• You must distinguish between prospective planning and post-hoc interpretation.

Next topic

In the next module, we proceed to randomization, blocking, and matching, aligning assignment and analysis units before determining sample size.

References

• statsmodels API: https://www.statsmodels.org/stable/api.html
• ASA p-value statement: https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf

The calculation inputs and values in this section are educational constructs written independently by BioStatPy.