Table 1 Performance metrics of different machine learning algorithms in nested, stratified k-fold CV. Mean AUC of the ROC and PR curves and F4 scores were calculated over 10 outer folds. The best performing model according to PR AUC is shown in bold. Feature selection was not performed for these models
Algorithm | Over- and/or Undersampling | Mean ROC AUC | Mean PR AUC | Mean F4 score |
|---|---|---|---|---|
Random Forest | Neither | 0.927 [0.907–0.946] | 0.571 [0.481–0.660] | 0.079 [0.000–0.178] |
Over | 0.928 [0.899–0.957] | 0.607 [0.514–0.701] | 0.370 [0.277–0.464] | |
Both | 0.935 [0.903–0.966] | 0.599 [0.558–0.640] | 0.648 [0.561–0.736] | |
XGBoost | Neither | 0.945 [0.933–0.958] | 0.631 [0.563–0.700] | 0.324 [0.219–0.429] |
Over | 0.951 [0.935–0.966] | 0.612 [0.521–0.704] | 0.477 [0.374–0.579] | |
Both | 0.929 [0.911–0.947] | 0.595 [0.539–0.652] | 0.659 [0.609–0.709] | |
CatBoost | Neither | 0.945 [0.897–0.968] | 0.612 [0.510–0.714] | 0.248 [0.113–0.382] |
Over | 0.950 [0.928–0.973] | 0.648 [0.603–0.694] | 0.416 [0.337–0.495] | |
Both | 0.934 [0.909–0.958] | 0.565 [0.497–0.633] | 0.639 [0.598–0.679] |