DistilProtBert

DistilProtBert applies distilled deep learning language models adapted from natural language processing to analyze protein sequences and discriminate native proteins from singlet-, doublet-, and triplet-shuffled sequences.

Key Features:

• Distillation Approach: Implements model distillation by training a smaller student network to replicate a larger teacher network derived from ProtBert.
• Efficiency Improvements: Reduces network size and runtime by 50% and decreases pretraining computational resources by 98% relative to ProtBert.
• Performance: Maintains accuracy close to the full ProtBert model across evaluated tasks.
• Application in Protein Sequence Analysis: Discriminates real protein sequences from randomly shuffled sequences that preserve singlet, doublet, and triplet amino-acid composition, achieving AUCs of 0.92, 0.91, and 0.87 respectively.
• Potential for Novel Discoveries: Identifies false-positive shuffled sequences classified as real proteins, enabling exploration of potential natural-like proteins.

Scientific Applications:

• Protein Function Prediction: Supports prediction of functions for newly discovered proteins by differentiating genuine sequences from artificial ones.
• Structural Biology: Assists studies of protein folding and stability by identifying potential natural-like sequences from shuffled data.
• Evolutionary Studies: Provides insights into sequence features and evolutionary processes by analyzing misclassified shuffled sequences.

Methodology:

Uses model distillation (student network trained to mimic a teacher network derived from ProtBert), adapts deep learning models from NLP for protein sequences, performs pretraining with reduced computational resources, and evaluates classification of real versus singlet/doublet/triplet-shuffled sequences using AUC metrics.