Genomic Tokenizer

Genomic Tokenizer incorporates the biological process flow of DNA transcription and translation into a standard tokenizer interface for transformer models to improve DNA sequence analysis by aligning tokenization with the central dogma of molecular biology.

Key Features:

• Biology-driven tokenization: Tokenization aligns with DNA transcription into RNA and translation into proteins, including three-letter codons, synonymous codons, start codons, and stop codons.
• Standard tokenizer interface: Implements the HuggingFace transformer package tokenizer interface for compatibility with existing transformer model training pipelines.
• Pre-training capability: Can be used to pre-train foundational transformer models on DNA sequences.
• Comparative evaluation: Performance can be compared with alternate tokenization strategies.

Scientific Applications:

• Foundational model pre-training: Pre-training transformer models on DNA sequences using biologically informed tokenization.
• DNA sequence analysis: Improved analysis of long DNA sequences through tokenization that reflects the central dogma of molecular biology.

Methodology:

The tokenizer incorporates the biological process flow of DNA transcription and translation into a standard tokenizer interface within the HuggingFace transformer package.