Existing process A had mean yield 84.2 and improved process B had mean 85.5, a difference of 1.3. Because B is larger, can we immediately declare improvement?

Not yet. Even if the two population means are equal, different samples can have different sample means. First ask how easily a difference like 1.3 could appear in a reference world with no effect.

A hypothesis test is not a machine that decides whether a cause exists. It defines a reference statistical model and measures how extreme the result is under that model, providing one part of the evidence.

At least two explanations lie behind an observed difference

When a sample shows a difference of 1.3:

• population means may be equal and sampling variation produced 1.3;
• population means may differ and that difference contributed to 1.3.

The first explanation becomes the reference world, the null hypothesis H0. For example, H0: μB−μA=0 states no population-mean difference.

The research alternative is H1. A non-directional question uses H1: μB−μA≠0; a prespecified improvement-only question might use H1: μB−μA>0.

H0 and H1 are not labels attached after seeing favorable data. Connect the effect definition, direction, design, and analysis model to the research question in advance.

A test statistic puts the difference on a common scale

A difference of 1.3 is difficult to judge alone. It carries different evidence when observations barely vary versus when they are widely dispersed.

A test statistic generally follows this intuition:

test statistic = difference between observed effect and H0 expectation / standard error of the effect

It expresses how far the effect lies from the H0 reference relative to sampling variation. The specific statistic may be t, F, χ², or a rank statistic depending on the question and structure. Later units choose among one, two, or many groups; continuous or categorical outcomes; and independent or paired designs.

A p-value is a tail probability under H0

Omitting its starting condition creates most p-value misunderstandings.

Assuming H0 and the test-model assumptions hold, the probability of a test statistic as extreme as or more extreme than the observed value in the direction defined by H1.

For a two-sided p=.03, under H0 and the model, about 3% of repeated results would have test statistics at least this extreme in either direction.

p=.03 does not mean:

• H0 has a 3% probability of being true;
• chance caused the result with 3% probability;
• H1 has a 97% probability of being true;
• 97% of repetitions will reproduce the result;
• the effect is large or biologically important.

Set significance level α before seeing the data

Significance level α sets the long-run willingness to make a Type I error: rejecting H0 when H0 is true. Choosing α=.05 uses a rule allowing about 5% false positives in repeated settings where H0 and all procedural assumptions hold.

If p<α, the result is sufficiently extreme under H0 to reject H0. If p≥α, there is insufficient evidence to reject H0. We have not accepted or proved H0.

With small samples and large spread, an important effect can still produce p≥α. Therefore “not significant = no difference = equivalent” is invalid. Equivalence requires prespecified equivalence margins and an analysis designed for that question.

One-sided and two-sided tests encode direction

A two-sided H1 treats both larger and smaller effects as differences. A right-sided H1 targets larger values; a left-sided H1 targets smaller values.

A one-sided test places all α in one tail and can more readily detect an effect in the prespecified direction. It does not count a large effect in the opposite direction as evidence for that H1. Therefore:

1. choose direction with the research question before seeing data;
2. justify why the opposite direction is scientifically irrelevant or handled by a separate safety decision;
3. report unexpected opposite-direction results.

Changing a two-sided test to a favorable one-sided test after seeing the mean can inflate the actual Type I error above the promised α.

The two error types are different failures

• Type I error: reject a true H0; a false positive claiming an effect that is absent.
• Type II error: fail to reject H0 when a specified H1 is true; a false negative missing a real effect.
• Power 1−β: probability of correctly rejecting H0 under a specified H1.

α is set before analysis, but β is not one universal constant. It depends on true effect size, spread, n, test direction, and method. Larger effects, smaller spread, and more independent units are generally easier to detect. Unit 13 develops sample size and power.

Repetition changes p-values too

The Lab repeats a one-sample z test in a simple normal model with reference mean μ0=10 and known σ=3. It is an educational model connecting sampling variation, p-values, and errors—not a replacement for choosing a real analysis.

1. At δ=0 with a two-sided test, inspect 80 p-values. Even with true H0, some may be below .05.
2. Press New batch of 80 and watch false-positive counts vary.
3. Increase δ through 0.5, 1.0, and 1.5; compare n=8 with n=32.
4. With positive δ, choose a “smaller” one-sided H1 and observe the conflict between data direction and question direction.

Exactly four of 80 need not have p<.05. α=.05 is not a quota for every batch; it is a long-run error rate under many repetitions of the same procedure.

Distinguish three routes to a smaller p-value

• A larger observed effect can increase the test statistic.
• Smaller spread makes the same effect clearer.
• Larger independent n reduces standard error, so even a tiny effect can yield a small p-value.

Thus p<.001 in a huge sample may accompany a trivial practical effect, while p=.08 in a small exploratory study does not rule out an important effect. Read effect size and CI in both cases.

JMP output separates parts of the evidence

JMP report names vary by analysis, but the core relationship joins the test statistic, p-value, H1, effect, and confidence interval.

Output such as Prob > |t| identifies the tails used in a two-sided t test. For a one-sided p-value, verify that the statistic sign matches the prespecified H1. Do not select whichever p-value appears smallest after analysis.

Statistical software does not guarantee that:

1. each row is the correct independent experimental unit;
2. assumptions and sample selection are valid;
3. only favorable analyses were not selected from many outcomes;
4. the effect is biologically, clinically, or qualitatively important;
5. a new experiment will reproduce the result.

Report more than reject or fail to reject

Bad:

The improved process works because p<.05.

Better:

In a prespecified two-sided analysis of independent batches, mean yield difference B−A was 1.3 percentage points, with 95% CI −1.8–4.4 and p=.39. This sample did not reject H0; it does not establish no improvement or equivalence of the processes.

Preserve the method, independent n, effect units, CI, exact p-value, test direction, and whether the analysis was prespecified. A p-value is part of an evidence sentence, not a one-character conclusion.

Seven statements to check before finishing

• H0 is a no-effect reference model; H1 carries the research direction.
• A test statistic scales the observed effect by its standard error under H0.
• A p-value is the probability, under H0 and the model, of an equally or more extreme result.
• A p-value is neither the probability H0 is true nor the effect size.
• At p≥α, fail to reject H0; do not accept it.
• α is the Type I error criterion under true H0 and is set before data.
• A rejection decision cannot replace effect size, CI, design, and practical meaning.

The next unit asks whether spread is equal before comparing two group means. We examine equal-variance assumptions and why one variance-test result should not mechanically choose the entire analysis.

Official supplementary resources

• NIST/SEMATECH · Critical values and p-values
• NIST/SEMATECH · Statistical tests
• ASA · Statement on Statistical Significance and P-Values
• JMP Statistics Knowledge Portal · Hypothesis Testing

The numbers, figures, and In-Silico Lab in this article are synthetic material for explaining statistics. They cannot support real research, clinical, quality, or regulatory decisions.