SMAFIRA

SMAFIRA provides a curated corpus of 300 annotated biomedical abstracts to evaluate relevance-ranking algorithms and identify methodologically equivalent alternatives to animal experiments.

Key Features:

• Corpus composition: The corpus comprises 300 annotated abstracts drawn from three experimental case studies focused on finding research papers with equivalent scientific purposes but different methodologies as potential alternatives to animal experiments.
• Annotation model: Annotations are based on a novel model that delineates stages in experimental biomedicine linked to generic scientific purposes, akin to topic modeling approaches.
• Annotation provenance: Annotations were performed by a researcher with extensive domain knowledge, with one instance supported by an additional expert opinion.
• Equivalence concept: Introduces the concept of "equivalence," a measure combining scientific purpose and alternative methodology—inspired by techniques from intellectual property domains—to enable transparent, reproducible comparisons of biomedical texts.
• Evaluation benchmark: Serves as a benchmark for evaluating the "similar articles" algorithm used in PubMed/Medline by comparing algorithmic rankings against expert-annotated relevance.
• Growing resource: Designed to expand over time as a benchmark resource for computer science research on biomedical knowledge discovery.

Scientific Applications:

• Relevance-ranking evaluation: Assess the precision and ranking performance of relevance-ranking algorithms in biomedical databases such as PubMed/Medline by comparison to expert annotations.
• Alternative-method discovery: Identify research articles that achieve equivalent scientific purposes via different methodologies to support identification of non-animal-based experimental alternatives.
• Text-mining and IR development: Support development and validation of text-mining, retrieval, and similarity measures, including topic modeling–style approaches, for biomedical knowledge discovery.

Methodology:

Annotations apply a novel model that delineates experimental stages linked to generic scientific purposes (akin to topic modeling); "equivalence" is operationalized as a measure combining scientific purpose and alternative methodology; evaluation is performed by comparing PubMed/Medline "similar articles" algorithm rankings against expert-annotated relevance.