Leakage, Pipeline, and Group/Time/Site Split

Upon completing this topic

You will be able to identify pathways through which information leaks from the training set to the test set before evaluation, isolate preprocessing within a Pipeline, and define splits aligned with group units such as patient, sample, site, or time.

The split unit comes first

When multiple rows originate from the same patient or independent sample, a row-level random split may cause information from the same entity to appear in both the training and test sets. For problems predicting the future, temporal order must also be preserved. Splits should be defined first based on the question and data structure, not after model selection.

The Pipeline fixes the preprocessing scope

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

model = Pipeline([
    ("scale", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1000)),
])
model.fit(X_train, y_train)
score = model.score(X_test, y_test)

The Pipeline bundles together the preprocessing steps that must be fit on the training fold and the model. The Pipeline itself does not automatically detect incorrect splits or target leakage.

Group split

Tools such as GroupKFold create a cross-validation structure that prevents the same group from appearing in both the training and test sets simultaneously. Do not assume that class balance is guaranteed in all folds; verify the number of groups and the target distribution.

Leakage path checklist

Preprocessing leakage can occur when scaling, imputation, or feature selection are fit on the entire dataset first. Target leakage occurs when features include columns derived directly from the target or from columns that are only known after the prediction timepoint. Group leakage occurs when related rows from the same patient, sample, or site are split between the training and test sets.

For each feature, record whether it would be available at the actual prediction time point, and verify that each preprocessing step is fitted within the appropriate training fold. After splitting, preprocessing statistics must be learned from the training data only.

Return to the research question for interpretation

The test score represents generalization performance relative to the defined split and target. To speak to performance on new sites, times, or patients, those units must be reflected in the evaluation.

Common failures

• Fitting preprocessing means and variances on the entire dataset first.
• Splitting the same patient/site between training and test.
• Including future rows in training for time prediction.
• Writing that the Pipeline resolves all leakage.

Key takeaways

• The split unit is determined by the research question and data hierarchy.
• The Pipeline aids in fold-local preprocessing.
• Generalization across groups, time, and sites requires different evaluation designs.

Next topic

In the next installment, we isolate selection information within folds using feature selection and nested tuning.

References

• scikit-learn common pitfalls: https://scikit-learn.org/stable/common_pitfalls.html
• GroupKFold: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html
• Bioconductor OSTA experimental design: https://bioconductor.org/books/release/OSTA/pages/bkg-exp-design.html

The pipeline example in this installment was written independently by BioStatPy.