Calibration, Class Imbalance, and Uncertainty

Upon completing this topic

You will be able to distinguish between discrimination and calibration, and explain the implications of metric selection in imbalanced data. You will report data splits and uncertainty alongside predicted probabilities, without converting a single predicted probability into a definitive judgment.

Discrimination vs. Calibration

Discrimination asks how well a model orders positive and negative cases. Calibration asks whether cases with a predicted probability of 0.7 actually exhibit the event at a rate of approximately 70% in the long run. Two models can have similar AUCs yet differ in calibration.

Imbalance

When the positive class rate is low, a model that simply predicts the majority class can still achieve high accuracy. Precision, recall, average precision, and specificity each emphasize different types of error. Pre-specify which errors are important and the intended use of the threshold.

Calibration API and Evaluation

The calibration tools in scikit-learn provide APIs to examine the relationship between predicted probabilities and observed outcomes. Calculate calibration curves and metrics within the train/test split, ensuring that the calibration process itself does not fit to test information.

Sources of Uncertainty

A single predicted probability may not reflect uncertainty from sample selection, model parameters, feature measurement, or site/time variations. Use purpose-appropriate repeated evaluations, such as per-fold performance in cross-validation, bootstrap assessments, and external validation, to record variability.

With severe class imbalance, the number of observations in each calibration bin may be small. Avoid over-interpreting the shape of the curve alone; instead, check the sample size per bin alongside confidence intervals or repeated variability.

Thresholds are Decision Rules

The threshold that converts probabilities into 0 and 1 classifications is not a fixed property of the model, but a rule based on error costs and usage purposes. Do not select the best threshold on the test set and then report that same test performance as the final result.

Interpreting via Research Questions

That probabilities are well-calibrated refers to the reliability of the predicted probabilities, not to the model explaining causality or performing identically in external populations. Consider site, time, and group shifts alongside uncertainty.

Common Failures

• Do not write that high AUC implies good calibration.
• Do not report only accuracy in imbalanced settings.
• Do not fit calibration to the entire dataset first.
• Do not interpret a predicted probability as certainty that an event will or will not occur.

Key Takeaways

• Discrimination and calibration address different evaluation questions.
• In imbalanced settings, the purpose of metrics and thresholds must be explicit.
• Calibration also adheres to split and leakage rules.

Next Topic

In the next section, we construct a reproducible analysis package with lock, seed, hash, and manifest.

References

• scikit-learn calibration: https://scikit-learn.org/stable/modules/calibration.html
• scikit-learn metrics: https://scikit-learn.org/stable/modules/model_evaluation.html
• GroupKFold: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html

The evaluation structure and examples in this section were written independently by BioStatPy.