statcheck

statcheck detects inconsistencies in null-hypothesis significance testing (NHST) reporting by extracting reported test statistics and degrees of freedom and recomputing p-values to compare with author-reported values.

Key Features:

• Automated extraction: Automatically extracts NHST results, including reported test statistics and degrees of freedom, from scientific articles.
• Recomputation of p-values: Recalculates p-values from the extracted test statistics and degrees of freedom.
• Discrepancy detection: Compares recalculated p-values with author-reported p-values to identify reporting inconsistencies.
• Automated screening: Applies the extraction, recomputation, and comparison steps in an automated manner for efficient processing of large volumes of literature.

Scientific Applications:

• Meta-analysis screening: Identifies statistical inconsistencies in primary studies to inform inclusion and data correction in meta-analyses.
• Reproducibility assessment: Detects reporting errors that affect the evidential value and reproducibility of published findings.
• Reporting quality evaluation: Supports assessment of the accuracy of NHST reporting across scientific literature.

Methodology:

Extract reported test statistics and degrees of freedom from articles; recalculate p-values from those values; compare recalculated p-values with author-reported p-values; perform these steps automatically to enable efficient screening of large numbers of studies.