REFUTE

REFUTE benchmarks scientific critique honesty and epistemic calibration in language models by testing whether models keep claims within what evidence allows and whether stated confidence is properly calibrated.

Key Features:

• Overclaim Detection: Tests whether models avoid making claims that exceed what the evidence supports.
• Planted-Flaw Identification: Evaluates whether models can detect deliberately introduced errors or weaknesses in scientific studies.
• Falsifier Selection: Assesses whether models can identify what evidence would overturn a study's conclusions.
• Faithful Summary Evaluation: Tests whether models can determine which summary accurately reflects the original study findings.
• Confidence Calibration: Measures whether models' stated confidence levels match their actual accuracy.
• Open-Ended Critique Scoring: Evaluates models' ability to produce unstructured scientific critiques.
• Judge-Free Multiple-Choice Evaluation: Uses auto-graded multiple-choice questions to avoid subjective scoring.
• Truth Score Metric: Provides a composite score ranging from 46.3 to 74.5 across tested models.

Scientific Applications:

• Language Model Evaluation: Benchmarks 20+ models on their ability to honestly interpret recent life-science and biomedical literature.
• Scientific Literature Critique: Tests AI models' capacity for honest reading and critical analysis of published biomedical research.
• Epistemic Calibration Research: Investigates the dissociation between critique skill and confidence calibration in language models.

Methodology:

REFUTE constructs 320 auto-graded questions across multiple-choice and open-ended formats, with evaluation through judge-free MCQ axes and open-ended critique scoring to produce Truth Scores.