ADS

ADS evaluates Gene Ontology (GO) classifiers by generating controlled dilution series of annotation signal and measuring the discriminative performance of evaluation metrics.

Key Features:

• Signal dilution series: Generates artificial datasets with systematically varied levels of correctness ("signal levels") by diluting existing GO annotation data.
• False positive annotation sets: Constructs deliberate false positive annotation sets to expose systematic weaknesses in evaluation metrics.
• Metric comparison: Computes and compares the performance of multiple evaluation metrics to assess discriminative power and susceptibility to misleading scores.
• Comprehensive benchmarking: Includes a comparative study involving 37 different evaluation metrics.
• Metric refinement: Identifies poorly performing metrics and proposes improvements to several well-known evaluation measures for GO annotation.

Scientific Applications:

• GO classifier evaluation: Benchmarking and validating Gene Ontology classifier performance across controlled signal levels.
• Metric selection and development: Guiding selection and refinement of evaluation metrics for GO-based automated protein annotation.
• Robustness testing: Assessing metric robustness to false positive annotations and systematic annotation errors.
• Generalized metric assessment: Providing a framework for evaluating evaluation metrics in other prediction contexts where signal discrimination and false positives matter.

Methodology:

Generate artificial datasets by diluting existing GO annotations to create signal levels, construct false positive annotation sets, and compute and compare 37 evaluation metrics to assess their discriminative power and vulnerability to false positives.