Feature Selection and Nested Tuning

After completing this topic

You will be able to explain the leakage that occurs when feature selection and hyperparameter tuning are performed outside the evaluation folds. You will understand nested cross-validation as a separation between the selection process and the final evaluation process.

Selection is also learning

Choosing features with low variance, selecting columns highly correlated with the target, or determining a model's hyperparameters are all processes of learning information from the data. Using test or validation information for selection can make performance appear overly optimistic.

Nested structure

The outer folds evaluate the final generalization performance, while the inner folds perform feature selection and tuning. The outer test folds must remain unseen by the inner selection process. If group or time splits are required, both the inner and outer folds preserve these units.

Execution order of nested evaluation

1. Split train and test within the outer fold.
2. Perform inner cross-validation only within the outer train.
3. Refit preprocessing, feature selection, and model tuning for each inner fold.
4. Fit the selected configuration to the entire outer train.
5. Evaluate once on the outer test.

Repeat this sequence across all outer folds to obtain the performance distribution. If you change the inner search space based on the outer test results, the final evaluation becomes exposed to the selection process again.

Selection stability

In high-dimensional data, the selected features may vary across folds. Recording selection frequency and coefficient direction allows you to separate model performance from feature stability. Do not automatically conclude that features selected infrequently are meaningless; instead, examine sample variation and correlation structure together.

Once you have created the final feature list, verify its performance on separate data not used for selection, or clearly indicate which folds and tuning ranges were used to create that list.

Returning to the research question for interpretation

A selected feature list does not necessarily imply a stable biological marker. Selection may change if the sample or split changes, and selection stability and external validation must be assessed separately.

Common failures

• Do not select features from the entire dataset first.
• Do not change feature selection rules based on test scores.
• Do not lose group/time units in the nested structure.
• Do not interpret selected features directly as causal variables.

Key takeaways

• Feature selection and tuning are also learning processes.
• Selection must be isolated within the evaluation folds.
• Nested CV is a structure that separates selection from final evaluation.

Next topic

The next module separates calibration, imbalance, and uncertainty.

References

• scikit-learn common pitfalls: https://scikit-learn.org/stable/common_pitfalls.html
• GroupKFold: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html
• scikit-learn metrics: https://scikit-learn.org/stable/modules/model_evaluation.html

The nested structure and explanation in this module were written independently by BioStatPy.