BioALBERT

BioALBERT provides domain-adapted ALBERT language models for biomedical natural language processing to improve performance across BioNLP tasks.

Key Features:

• Architecture: ALBERT (A Lite BERT) transformer-based architecture adapted for biomedical text.
• Model variants: Eight pretrained variants were developed and evaluated.
• Pretraining corpora: Models were pretrained on PubMed, PubMed Central, and MIMIC-III corpora.
• Fine-tuning and benchmarks: Variants were fine-tuned for six BioNLP tasks across 20 benchmark datasets.
• Performance improvements: A large variant trained on PubMed achieved an 11.09% increase in BLURB score for named-entity recognition, 0.80% improvement for relation extraction, 1.05% improvement in sentence similarity, a 0.62% rise in F1-score for document classification, and a 2.83% boost in question answering.
• Benchmark dominance: Variants attained state-of-the-art results in five of six benchmark tasks and outperformed previous models on 17 of the 20 benchmark datasets.
• Baseline utility: Provides reliable pretrained baselines that reduce the computational cost of training new models for BioNLP evaluation.

Scientific Applications:

• Named-entity recognition: Improves BLURB-measured performance on biomedical NER tasks.
• Relation extraction: Enhances extraction of relationships between biomedical entities.
• Sentence similarity: Improves assessment of semantic similarity between biomedical sentences.
• Document classification: Increases F1-score for classification of biomedical documents.
• Question answering: Boosts performance on biomedical question answering benchmarks.
• Biomedical and clinical text analysis: Enables improved modeling of both PubMed/PubMed Central literature and MIMIC-III clinical notes.

Methodology:

Eight ALBERT variants were pretrained on PubMed, PubMed Central, and MIMIC-III corpora and then fine-tuned on six BioNLP tasks across 20 benchmark datasets.