ProteinPretraining

ProteinPretraining introduces advanced tokenization methods for protein language models using byte-pair-encoding and unigram to improve performance on downstream tasks, and provides PETA as a benchmark for systematic evaluation of protein language models.

Key Features:

• Advanced Tokenization: Employs byte-pair-encoding algorithm and unigram to construct residue vocabularies that recognize frequently occurring combinations of amino acids as single tokens.
• PETA Benchmark: Provides a comprehensive benchmark for systematically evaluating protein language models on downstream tasks.
• Optimized Vocabulary Sizes: Identifies vocabularies comprising 50 and 200 elements as achieving optimal performance.

Scientific Applications:

• Protein Representation Learning: Enhances protein language model pre-training by moving beyond simple 20 natural amino acid tokenization to capture co-occurring residue patterns.
• Protein Language Model Evaluation: Enables systematic comparison of protein language models through standardized downstream task assessment.

Methodology:

Uses byte-pair-encoding algorithm and unigram to construct advanced residue vocabularies for protein sequence tokenization.