Effect Size, Uncertainty, and p-value

Upon completing this topic

You will be able to explain that effect size, direction, estimation uncertainty, and p-value provide distinct information. You will be able to structure reporting so that research questions and results are presented together, avoiding interpretation of the p-value as the probability that the null hypothesis is true or as a measure of biological significance.

Three questions for one comparison

Suppose we are comparing the outcomes of two conditions. At a minimum, we must separate the following three questions.

1. What is the direction and magnitude of the observed difference?
2. Given sample variability, what is the uncertainty of that difference?
3. Under a specific null model, how rare are the current or more extreme data?

The first relates to effect size, the second to intervals and standard error, and the third to the p-value. All three may be necessary in the same study, but none can substitute for the other two.

Effect Size: Expressing Difference in Units of Inquiry

Effect size expresses the magnitude and direction of the observed difference. If it is a mean difference, it can be interpreted in the original units of the outcome; other scales such as ratio, risk ratio, or standardized difference may also be chosen. The appropriate scale depends on the research question and the measurement units.

A large effect size alone does not imply an important result. One must also consider the minimum difference required for biological judgment, measurement error, cost and risk, and the validity of the study design.

Uncertainty: How Much Does the Effect Fluctuate?

If we were to obtain samples again using the same procedure, the estimated effect size might differ. Standard error, confidence intervals, and bootstrap intervals are methods to express this variability. A wide interval may signal that the data do not precisely pin down the effect size, but this should not be attributed solely to sample size.

When reporting intervals, specify the effect scale, the unit of analysis, and the procedure used to construct the interval. Do not mix comparisons of differences in original units with standardized differences.

p-value: Extremity of Data Under the Null Model

The p-value quantifies how extreme the observed result, or a more extreme result, would be under a specific null hypothesis and analysis procedure. It is neither the probability that the null hypothesis is true nor the probability that the alternative hypothesis is true.

A small p-value provides information describing the relationship between the null model and the data. It does not automatically translate to meaning the effect is large, reproducible, biologically important, or that causation has been proven. Conversely, a large p-value does not confirm the absence of an effect.

Bundling Results in One Sentence

The basic structure for reporting results is as follows.

조건 B와 A의 추정 효과는 [척도와 방향]이었다.
불확실성은 [구간 또는 표준오차]로 표현했다.
사전 정의한 귀무모형 아래 p-value는 [값]이었다.
이 결과는 [설계·측정·외삽]의 범위에서만 해석한다.

When inserting actual numbers, verify the execution results and the unit of analysis. Leaving only the p-value prevents readers from knowing the magnitude and precision of the effect.

Different Results Can Occur Simultaneously

Even if the effect size is small, the p-value can be small if sample variability is low. Conversely, even if the effect appears substantial, the interval can be wide and the p-value large if data variability is high. These combinations are not contradictory because they answer different questions.

For example, a mean difference may be small in original units but still exceed the minimum important difference for the study's purpose. Conversely, a difference may be statistically distinguishable but have little meaning in actual measurement error or subsequent decision-making. To make such judgments, importance criteria and measurement units must be defined prior to the study.

Result tables should be designed to include at least columns for estimate, uncertainty, p_value, analysis_unit, method, and interpretation_limit. Do not create a hierarchy of results based solely on asterisks or colors in a single column.

A Priori Hypotheses and Analysis Scope

Interpretation of the p-value depends on which null hypothesis, direction, and analysis procedure were defined a priori. Do not change the outcome, subset, or comparison direction after viewing the data and then report the same p-value as a confirmatory result. If a comparison was discovered exploratorily, record that it is exploratory and treat it as a question requiring subsequent confirmation.

Returning to the Research Question for Interpretation

Researchers are usually more interested in "How different is it, and does this difference matter for the study's purpose?" than in "Is the difference 0?" Effect size and uncertainty connect more directly to this question. The p-value additionally shows the relationship with the null model but does not provide criteria for importance relative to the study's purpose.

Rows Researchers Should Verify in Result Tables

Read each comparison in the following four layers. First, verify the direction and unit in estimate. Next, observe the extent of the interval or standard error. Then, confirm under which null model and procedure the p-value was calculated. Finally, read the observational unit and the scope of extrapolation.

Adhering to this order prevents the single word "significant" from overshadowing the entire result. For example, even if the p-value is small, if the interval is mostly smaller than the minimum important difference in the study, the practical judgment may differ. Conversely, if the interval is wide, the direction and magnitude of the effect may need further observation.

Effect size criteria vary by field and measurement unit. Do not apply externally derived thresholds as universal standards of biological importance; instead, record the basis of the research question and the a priori plan.

Common failures and checks

• Do not use the p-value as the probability that the null hypothesis is true.
• Do not report only significance status while omitting effect size and intervals.
• Do not confirm biological importance or causation based on a small p-value.
• Do not use a large p-value as proof of no effect.
• Do not use post-hoc thresholds as if they were a priori hypotheses.

Key takeaways

• Effect size describes the magnitude and direction of the difference.
• Uncertainty expresses sample variability.
• The p-value describes the extremity of data under a specific null model.
• These three pieces of information are not interchangeable.
• Interpretation must be limited to the scope of study design and biological importance.

Next topic

In the next section, we will select designs and analysis units for two-group, paired, and non-parametric comparisons.

References

• ASA p-value statement: https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf
• SciPy statistics reference: https://docs.scipy.org/doc/scipy/reference/stats.html

The reporting sentences and examples in this section are educational compositions created independently by BioStatPy.