MLbias

MLbias corrects selection bias when comparing machine-learning classifiers by fitting learning curves with an inverse power law to adjust cross-validation error rates for more reliable genomic classifier evaluation.

Key Features:

• Implementation: Implemented as an R package for analysis and correction of classifier error-rate estimates.
• Bias correction methodology: Uses learning curve fitting by inverse power law (IPL) to mitigate selection bias from choosing the classifier with the smallest cross-validation error rate.
• Comparative methods: Explicitly compares IPL-based correction against nested cross-validation, weighted mean correction, and the Tibshirani-Tibshirani procedure.
• Performance characterization: Demonstrated via extensive simulations and real-world dataset analyses that IPL yields superior bias correction with smaller variance.
• Error extrapolation: Extrapolates error estimates to larger sample sizes to inform whether additional samples would improve classifier accuracy.
• Targeted domain: Intended for genomic studies with moderate sample sizes (e.g., n = 30–60) where reported cross-validation error rates may be optimistic.

Scientific Applications:

• Genomic classifier development: Adjusts cross-validation error rates to support development of classifiers that generalize to independent testing datasets.
• Clinical study validation: Reduces the risk of reporting artificially deflated error rates in clinical studies by correcting selection bias.
• Sample size recommendations: Provides guidance on whether additional data collection is warranted by extrapolating expected error at larger sample sizes.

Methodology:

Fits learning curves using an inverse power law (IPL), compares IPL-corrected error estimates with nested cross-validation, weighted mean correction, and the Tibshirani-Tibshirani procedure, evaluates performance via simulations and real-world dataset analyses, and extrapolates error estimates to larger sample sizes.