Pool PaRTI

Pool PaRTI leverages PageRank and internal transformer attention matrices to assign token importance weights, producing fixed-length protein embedding vectors while identifying critical residues for biological interpretability.

Key Features:

• PageRank-Based Pooling: The method applies PageRank to internal transformer attention matrices to compute token importance weights for pooling variable-length protein sequence embeddings into fixed-length vectors.
• Critical Residue Identification: The unsupervised, parameter-free approach prioritizes residues experimentally annotated as critical for function, assigning them higher importance scores without relying on explicit structural data or annotated training.
• Cross-Model Generalizability: Pool PaRTI has been evaluated with two encoder-only protein language models, ESM2 and protBERT, demonstrating robustness across different architectures.
• Pre-computed Embeddings: Sequence embeddings and residue importance values for all human proteins on UniProt are available for ESM2 and protBERT.

Scientific Applications:

• Protein Representation Learning: Pool PaRTI compresses token-level protein language model outputs into single vector representations of entire proteins for downstream machine learning tasks.
• Functional Site Prediction: The method identifies biologically relevant regions and critical residues for protein function without requiring structural data or functional annotations.
• Downstream Protein Machine Learning: Pool PaRTI enables significant performance gains across diverse protein machine learning tasks requiring fixed-length input vectors.

Methodology:

Pool PaRTI extracts internal transformer attention matrices, applies PageRank to assign token importance weights, and uses these weights to pool token-level embeddings into a fixed-length protein representation. The method is implemented in Python with PyTorch.