FS methods benchmarking report

1 Dataset overview

• Number of samples: 126
• Number of features: 18
• Outcome predicted: EARLY_PROG

	NLR	ECOG	SEX	AGE	LDH	TPS	CPS	LINE	TREATMENT	PATHOLOGY	CONCENTRATION	HALF_WIDTH1	HEIGHT_PEAK1	PEAK1	PEAK2	PEAK_DIFF	LESS_75	SIZE_75_111	SIZE_111_240	SIZE_160_220	SIZE_240_370	SIZE_280_320	SIZE_370_580	SIZE_580_1650	GREATER_1650
ID
Loading ITables v2.1.2 from the internet... (need help?)

In this report, we show examples of tables and plots that can be used for summarizing performances obtained by benchmarking different FS methods by running run_pipeline.ipynb or run_pipeline.py. Here, some plots are generated specifically from the LogisticRegression classifier. This classifier can be easily changed by modifying the parameter clf_to_display.

2 Feature selection summary

2.1 Preprocessing flowchart

The flowchart below summarizes the preprocessing steps applied to the dataset before feature selection.

2.2 Robustness of selected features

The following table summarizes the selection frequencies of individual features. The “Num methods” column indicates how many feature selection (FS) methods selected a feature in more than 50% of the resamples. The remaining columns show the selection frequencies for individual FS methods. The pink highlighting indicates that the feature was selected when applying the FS method to the full (apparent) set. Below is a summary of the feature selection methods parameters.

Feature Selection Methods

Lasso

• scoring: roc_auc
• n_splits_cv: 5

mRMR

p t-test 0.05 adjust

• threshold: 0.05
• adjust: True

p logistic 0.05 adjust

• threshold: 0.05
• adjust: True

All features, no selection

Code	Num methods	Lasso	mRMR	p t-test 0.05 adjust	p logistic 0.05 adjust
Loading ITables v2.1.2 from the internet... (need help?)

2.3 Size of selected feature subsets

3 Performances across different models

Two types of calculation were used here for the metric plotted on the y axis. The first one follows the approach described in the literature, where the mean \(\theta_{632+}\) is used to compute the final corrected score. The result of this calculation is a single value for each of the feature selection methods.

The second type of calculation involves computing the corrected score separately for each resample dataset using individual value of \(\theta_{632+}\), not the mean one. The result of this calculation is a set containing \(B\) optimism-corrected values, which allows to assess metric variability without running additional time-demanding experiments needed for confidence intervals calculation.

The point estimate and the mean of individual metrics are not identical, but they are generally very similar. It can therefore be said that these two ways of calculation yield equivalent results.

3.1 AUC vs stability

3.2 AUC vs PPV

3.3 All models

Below is a summary of the classifiers parameters.

Classifiers

LogisticRegression

• C: 1.0
• penalty: l2
• solver: lbfgs
• max_iter: 100
• multi_class: auto
• l1_ratio: None

RandomForestClassifier

• n_estimators: 100
• max_depth: 2
• min_samples_leaf: 40
• max_features: 0.7
• min_samples_split: 2

GradientBoostingClassifier

• n_estimators: 100
• max_depth: 2
• min_samples_leaf: 40
• max_features: 0.7
• min_samples_split: 2
• subsample: 0.7
• learning_rate: 0.1

Several classifiers and FS methods were benchmarked, resulting in 15 FS method/classifier pairs. The interactive barplot below compares optimism-corrected metrics across all model pairs. You can click on the legend to show/hide specific pairs and hover over bars for detailed values.

In the table below, you can select a specific FS method or classifier using the search box and/or sort the models by a chosen metric.

Method	Classifier	Stability	Mean #	AUC app	AUC corr	AUC OOB	AUC OOB SD	PPV app	PPV corr	PPV OOB	PPV OOB SD	ACC app	ACC corr	ACC OOB	ACC OOB SD	SE app	SE corr	SE OOB	SE OOB SD	SP app	SP corr	SP OOB	SP OOB SD	NPV app	NPV corr	NPV OOB	NPV OOB SD
Loading ITables v2.1.2 from the internet... (need help?)
Method	Classifier	Stability	Mean #	AUC app	AUC corr	AUC OOB	AUC OOB SD	PPV app	PPV corr	PPV OOB	PPV OOB SD	ACC app	ACC corr	ACC OOB	ACC OOB SD	SE app	SE corr	SE OOB	SE OOB SD	SP app	SP corr	SP OOB	SP OOB SD	NPV app	NPV corr	NPV OOB	NPV OOB SD