EAPP - Extended a Priori Probability tool

EAPP - Extended a Priori Probability tool evaluates binary classification task difficulty and dataset bias using a semi-supervised metric that extends a priori probability.

Key Features:

• Beyond Baseline: Extends the a priori probability baseline (e.g., ZeroR) by factoring dataset biases and additional characteristics that affect class separability.
• Semi-Supervised Approach: Employs a semi-supervised method to quantify task complexity and data bias for binary classification.
• AUC ROC Integration: Grounds the metric in area under the ROC curve (AUC ROC) to maintain robustness to class imbalance.
• Multiobjective Feature Extraction: Uses autoencoders for feature extraction to identify relevant representations from the data.
• Clustering and Assignation: Performs clustering in the input space followed by a combinatory weighted assignation of clusters to classes that considers distance to nearest clusters and optimizes assignment to maximize AUC ROC.
• Cross-Validation: Applies cross-validation during feature extraction and clustering to reduce overfitting and improve generalizability.
• Cluster-Based Evaluation: Defines the metric for various numbers of clusters, starting from the inverse of the minority class proportion, to enable fair comparisons across imbalanced datasets.

Scientific Applications:

• Baseline Assessment: Provides a comprehensive baseline metric for evaluating binary classifiers beyond simple a priori probability.
• Bias Detection: Helps determine whether high classifier performance arises from inherent task ease or from dataset biases.
• Imbalanced Dataset Evaluation: Facilitates comparison of task difficulty across datasets with varying class imbalance using cluster-based definitions tied to minority class proportion.

Methodology:

EAPP computes a semi-supervised metric by extracting features with autoencoders, clustering the input space for varying cluster counts (starting from the inverse of the minority class proportion), assigning clusters to classes via a combinatory weighted scheme based on distances to nearest clusters and selecting assignments that maximize AUC ROC, with cross-validation applied during feature extraction and clustering.