Auto-CORPus

Auto-CORPus converts HTML and table image files from biomedical publications into standardized, machine-interpretable formats to support biomedical text analytics and natural language processing.

Key Features:

• Standardization with BioC Format: Converts HTML content from publications into the BioC format for consistent representation of text and annotations used in biomedical NLP and machine learning.
• Information Artifact Ontology Annotation: Annotates publication sections in the BioC output using the Information Artifact Ontology to standardize section semantics.
• Table Data Conversion: Processes inline and linked HTML tables and converts them into a custom JSON format that captures table content and metadata to represent table data absent from the BioC specification.
• Abbreviation Extraction: Extracts abbreviations declared in publication texts and generates JSON mappings of each abbreviation to its full definition for incorporation into text-mining workflows.

Scientific Applications:

• Machine learning model training: Provides standardized, annotated corpora in BioC to enhance training and evaluation of biomedical NLP and machine learning models.
• Interoperability of text analysis systems: Enables exchange and annotation of table data across text analytics systems via the custom table JSON format.
• Named entity recognition augmentation: Improves named entity recognition by supplying publication-specific abbreviations and their definitions.

Methodology:

The pipeline parses HTML files (including inline and linked HTML tables) to extract sections and tables, annotates sections with the Information Artifact Ontology, converts table data into a custom JSON format with metadata, and extracts abbreviations mapping them to definitions in structured JSON.