AUC comparison p-value

AUC comparison p-value calculates the statistical p-value for comparing two Area Under the Curve (AUC) values from Receiver Operating Characteristic (ROC) analyses to assess whether predictive models differ in discrimination ability.

Key Features:

• Statistical comparison: Calculates the p-value to determine whether two ROC-derived AUCs differ significantly in discrimination performance.
• Net Reclassification Improvement (NRI) and Integrated Discrimination Improvement (IDI): Computes NRI and IDI indices to quantify improvements in risk prediction between models.
• Clinical model comparison: Applicable to comparing prognostic models such as APACHE IV for hospital mortality in sepsis cohorts.
• Data context: Demonstrated on data from the Multiparameter Intelligent Monitoring in Intensive Care II (MIMIC II) database using variables around hypotension episodes in septic patients.
• Predictor selection and modeling (example): In the described study, a genetic algorithm selected 30 predictors from 189 candidate variables and models were validated using logistic regression.
• Reported metrics: Outputs include AUC (example 82.0%), Hosmer-Lemeshow C statistic (example 10.43, p=0.06), NRI (example 0.19, p<0.001), and IDI (example 0.09, p<0.001).

Scientific Applications:

• Clinical prediction development and validation: Comparing new mortality prediction algorithms against established models such as APACHE IV in sepsis research.
• Retrospective cohort model comparison: Assessing whether alternative predictive models significantly improve discrimination and reclassification in datasets like MIMIC II.
• Risk stratification around clinical events: Evaluating model performance using dynamic variables collected around hypotensive episodes in septic patients.

Methodology:

Computes p-values for differences between two ROC AUCs and calculates Net Reclassification Improvement (NRI) and Integrated Discrimination Improvement (IDI); example workflow included genetic-algorithm-based predictor selection and logistic regression validation on separate training (n=1500) and validation (n=613) sets.