Prediction Target, Baseline, and Metric

Upon completing this topic

You will be able to define the prediction target and the unit of analysis, and avoid confusing the baseline with the metric. You will select a metric aligned with your research objective and error costs, rather than evaluating the model solely by accuracy.

Translating the Prediction Question into a Data Contract

The question "Can we predict the outcome for this sample?" requires a target, a prediction time, available features, and an evaluation unit. If you do not verify whether the target is a future outcome or a currently observed value, leakage may occur.

The Baseline as a Starting Point

The baseline is the minimum standard for comparison against more complex models, such as the mean, the majority class, or existing rules. Performing better than the baseline indicates improvement within that specific evaluation design, not necessarily utility in all environments.

The Metric Reflects the Cost of the Question

In regression, MAE and RMSE may have different sensitivities to errors. In classification, precision, recall, ROC AUC, and average precision provide different perspectives. In cases of class imbalance, accuracy may be dominated by the majority class. Define the metric and the rationale for its selection in advance.

Creating a Metric Selection Table

If you want to know how large the prediction error is in the original units of the outcome, consider a metric that preserves those units, such as MAE. If you wish to weight larger errors more heavily, examine the properties of RMSE. In classification, the cost of missed positives and false positives may differ; link the priority of recall and precision to your research objective.

You may report multiple metrics, but the primary metric must be defined before viewing the model results. Use secondary metrics to show other failure modes of the model; do not cherry-pick the best numbers to represent performance.

Baseline and Model Improvement

The majority class baseline, the mean prediction baseline, and the existing rule-based baseline address different questions. If the model surpasses the baseline, report both the absolute difference and the uncertainty. Whether small performance gains persist in actual operational costs and external validation is a separate judgment.

Returning to the Research Question for Interpretation

Predictive performance is not a causal effect. Good metric results do not guarantee biological mechanisms or performance at external sites. Report the definition of the target, the evaluation unit, and the data split.

Common Failures

• Including the target in the features.
• Failing to distinguish between the baseline and actual operational criteria.
• Evaluating imbalanced classification using only accuracy.
• Selecting favorable metrics after viewing the results.

Key Takeaways

• Define the prediction target and evaluation unit first.
• The baseline is the starting point for comparison.
• Select the metric according to error costs and research objectives.

Next Topic

The next section covers Pipelines and group/time/site splits to prevent leakage.

References

• scikit-learn metrics: https://scikit-learn.org/stable/modules/model_evaluation.html
• scikit-learn calibration: https://scikit-learn.org/stable/modules/calibration.html

The target schema and examples in this chapter were written independently by BioStatPy.