FS methods benchmarking report

Author

COMPO

Published

July 16, 2026

/builds/compo/roofs/output/run_2026_07_16_13_50/optimism_correction/optimism.pkl

1 Dataset overview

Number of samples: 126
Number of features: 143
Outcome predicted: EARLY_PROG

Head of the features dataframe:

	AGE	HEIGHT	ALBU	SODIUM	POTASSIUM	CREAT	T3	SLD	TOTAL_SLD	PEAK1	...	N_N3	N_N3b	M_M0	M_M1	M_M1b	PATHOLOGY_HNSCC	PATHOLOGY_NSCLC	PATHOLOGY_bladder	CURRENT_LINE_1	CURRENT_LINE_2
ID
701-89	-1.196726	-0.779734	0.687236	0.849778	-1.207485	-0.645134	-2.460641	0.396588	0.830583	-1.578076	...	0.0	0.0	1.0	0.0	0.0	0.0	1.0	0.0	1.0	0.0
700-93	-2.139737	-0.984984	-0.055434	0.103632	-0.152767	-0.091747	0.178287	1.318167	0.864358	-0.170167	...	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	0.0
700-194	-0.159414	0.862269	-1.558455	0.103632	0.163649	1.046548	0.178287	-0.320196	-0.486662	-0.875661	...	0.0	0.0	1.0	0.0	0.0	1.0	0.0	0.0	0.0	1.0
700-1	0.877898	-0.437650	0.952475	0.103632	-1.471165	-0.925330	0.178287	-0.197319	-0.165795	0.524446	...	0.0	0.0	0.0	1.0	0.0	1.0	0.0	0.0	0.0	1.0
700-103	1.443705	0.451768	-0.090799	0.103632	-0.152767	0.125404	0.178287	-1.118897	-0.959520	-0.189764	...	0.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	1.0	0.0

5 rows × 143 columns

In this report, we show examples of tables and plots that can be used for summarizing performances obtained by benchmarking different FS methods by running run_pipeline.ipynb or run_pipeline.py. Here, some plots are generated specifically from the LogisticRegression classifier. This classifier can be easily changed by modifying the parameter clf_to_display.

2 Feature selection summary

2.1 Robustness of selected features

The following table summarizes the selection frequencies of individual features. The “Num methods” column indicates how many feature selection (FS) methods selected a feature in more than 50% of the resamples. The remaining columns show the selection frequencies for individual FS methods. The pink highlighting indicates that the feature was selected when applying the FS method to the full (apparent) set.

Code	Num methods	Lasso	mRMR	p t-test .05	p logistic .05
Loading ITables v2.1.2 from the internet... (need help?)

2.2 Size of selected feature subsets

3 Performances across different models

Two types of calculation were used here for the metric plotted on the y axis. The first one follows the approach described in the literature, where the mean \(\theta_{632+}\) is used to compute the final corrected score. The result of this calculation is a single value for each of the feature selection methods.

The second type of calculation involves computing the corrected score separately for each resample dataset using individual value of \(\theta_{632+}\), not the mean one. The result of this calculation is a set containing \(B\) optimism-corrected values, which allows to assess metric variability without running additional time-demanding experiments needed for confidence intervals calculation.

The point estimate and the mean of individual metrics are not identical, but they are generally very similar. It can therefore be said that these two ways of calculation yield equivalent results.

3.1 AUC vs stability

3.2 AUC vs PPV

3.3 All models

Several classifiers and FS methods were benchmarked, resulting in 15 FS method/classifier pairs. The interactive barplot below compares optimism-corrected metrics across all model pairs. You can click on the legend to show/hide specific pairs and hover over bars for detailed values.

In the table below, you can select a specific FS method or classifier using the search box and/or sort the models by a chosen metric.

Method	Classifier	Stability	Mean #	AUC app	AUC corr	AUC OOB	AUC OOB SD	PPV app	PPV corr	PPV OOB	PPV OOB SD	ACC app	ACC corr	ACC OOB	ACC OOB SD	SE app	SE corr	SE OOB	SE OOB SD	SP app	SP corr	SP OOB	SP OOB SD	NPV app	NPV corr	NPV OOB	NPV OOB SD
Loading ITables v2.1.2 from the internet... (need help?)
Method	Classifier	Stability	Mean #	AUC app	AUC corr	AUC OOB	AUC OOB SD	PPV app	PPV corr	PPV OOB	PPV OOB SD	ACC app	ACC corr	ACC OOB	ACC OOB SD	SE app	SE corr	SE OOB	SE OOB SD	SP app	SP corr	SP OOB	SP OOB SD	NPV app	NPV corr	NPV OOB	NPV OOB SD