Confidence Intervals, Bootstrap, and Permutation

Upon Completing This Topic

You will be able to explain how confidence intervals express the uncertainty of an estimator. You will distinguish between bootstrap and permutation, avoiding the conflation of both as mere resampling, and clarify the specific questions and exchangeability assumptions each employs.

Not Hiding Uncertainty Behind a Single Value

The difference or mean calculated from a sample is a single number. However, if you were to obtain the sample again, you might not get the same value. An interval is one way to express this sampling variability. When stating the meaning of an interval, you must also specify the procedure, estimator, sample unit, and assumptions used.

Do not simply translate confidence intervals as "the probability that the parameter lies within this interval." You must separate sentences that describe the long-term inclusion property of repeated samples and procedures from those that interpret the current interval in practice.

Bootstrap: Resampling from the Observed Sample

Bootstrap constructs the variability of a statistic of interest by repeatedly drawing resampled data with replacement from the observed sample, treating it as an empirical population. It calculates the statistic relevant to the analytical purpose (e.g., mean, median, difference) for each iteration.

The critical factor is the unit of resampling. Even if there are multiple technical replicates, if the biological replicate is the independent unit, a bootstrap that does not preserve the structure of independent units may misrepresent uncertainty.

Permutation: Shuffling Labels Under the Null Structure

Permutation constructs the distribution of a statistic under a specific null structure by shuffling condition labels or swapping the roles of observations. This requires the assumption of exchangeability: the labels or roles must be exchangeable under the null structure. Shuffling labels arbitrarily in repeated-measures, clustered, or time-series data can disrupt the study design structure.

Bootstrap is primarily used to answer questions about estimating sample variability, while permutation is used to construct a baseline for null comparison. They are not the same method simply because both involve "shuffling many times via computer."

Creating a Bootstrap Distribution of Differences in Python

import numpy as np

def mean_difference(x, y):
    return np.mean(x) - np.mean(y)

rng = np.random.default_rng(20260806)
x = rng.normal(10, 1, size=20)
y = rng.normal(10.5, 1, size=20)
boot = []
for _ in range(2000):
    bx = rng.choice(x, size=len(x), replace=True)
    by = rng.choice(y, size=len(y), replace=True)
    boot.append(mean_difference(bx, by))

interval = np.quantile(boot, [0.025, 0.975])
print(mean_difference(x, y), interval)

This example is a synthetic illustration written under the assumption that the two arrays come from independent analytical units. If there is an actual replication structure, you must group the elements of x and y into units appropriate for the design rather than treating them as individual measurements for resampling. Do not automatically interpret the interval from the execution result as a universal truth; record the percentile procedure and seed used.

Checking for Exchangeability

Before resampling, ask the following:

• What is being resampled or shuffled?
• Is that unit independent?
• Is the exchange of conditional labels allowed by the design?
• Have the time, cluster, or paired structures been preserved?
• What procedure was used to summarize the uncertainty of the result?

If you increase the number of iterations without answering these questions, the numerical calculation may become more precise, but errors in the data structure will persist.

Interpreting Intervals Alongside Results

When presenting a bootstrap interval, record the point estimate along with the lower and upper bounds of the interval. If the interval is wide, examine whether the sample variability is large, the number of analytical units is small, or if the data tails and distribution influence the estimator. If the interval is narrow, it may indicate that the data is sufficiently stable, but you should also check for the possibility of resampling at an incorrectly low unit level or duplicating specific units.

Interpret permutation results in the same way. Compare the distribution of statistics generated under the null model with the position of the observed statistic. However, if the exchangeability assumption of the permutation does not match the design, the meaning of the p-value changes. Record the resampling unit, number of iterations, seed, method name, and assumption status in the results table.

In data with pairing or clustering, instead of shuffling entire rows, you may need a procedure that preserves pairs or clusters. Indicate the schema and code so that readers can reproduce the preservation rules used.

Returning to the Research Question for Interpretation

Intervals allow us to view both the magnitude of the difference and its uncertainty. However, do not declare biological importance or the absence of an effect solely on the basis of whether the interval includes 0. Report the direction and magnitude of the effect, study design, measurement unit, and analytical purpose together.

Recording the Choice of Bootstrap Interval

There may be several procedures for creating intervals, such as percentile, basic, and studentized. This section does not declare one procedure as the universal correct answer because the properties of each procedure can vary with the estimator and data structure. Record the method used, the number of iterations, the seed, and the resampling unit alongside the results.

Increasing the number of iterations may help reduce the Monte Carlo variability of the calculated quantiles, but it does not correct for sampling bias in the original data or incorrect unit selection. When checking whether the interval appears stable, re-examine the measurement and assignment structure of the original data.

Permutation and Prior Questions

Permutation starts not with the question "what difference arises if labels are shuffled?" but with the question "is it permissible to exchange labels under the null structure?" For paired data, a structure that flips the sign of paired differences may be more appropriate, and for clustered data, clusters may need to be preserved. A p-value without a description of the chosen structure is incompletely interpreted.

Common Failures and Checks

• Do not refer to bootstrap and permutation as the same procedure.
• Do not resample sub-measurements independently below the observational unit.
• Do not break pairs in paired data and shuffle labels.
• Do not express confidence intervals as posterior probabilities of the parameter.
• Do not write that increasing the number of iterations resolves incorrect exchangeability assumptions.

Key Takeaways

• Bootstrap constructs the distribution of an estimator from the variability of the observed sample.
• Permutation creates a comparison baseline under an exchangeable null structure.
• The resampling unit and exchangeability are determined by the study design.
• Intervals are tools for reading both the magnitude of the effect and its uncertainty.

Next Topic

In the next section, we separate effect size, uncertainty, and p-value as distinct pieces of information.

References

• SciPy statistics reference: https://docs.scipy.org/doc/scipy/reference/stats.html
• NIST/SEMATECH e-Handbook: https://www.itl.nist.gov/div898/handbook/

The samples and resampling examples in this section are synthetic educational examples independently created by BioStatPy.