polars-bio

polars-bio enables fast, parallel, and out-of-core operations on large genomic interval datasets in Python, using Rust-based components for efficient manipulation of tabular genomic coordinate data.

Key Features:

• High-performance interval operations: Implements count overlaps, coverage, and overlap operations on genomic interval dataframes with single-thread speedups of 38x for count overlaps and 15x for coverage compared to bioframe.
• Memory-efficient streaming: Overlap operation consumes 90x less memory in streaming mode for out-of-core processing of large datasets.
• Multi-thread scalability: Achieves up to 282x speedup for count overlaps with 8 CPU cores through parallel execution.
• DataFrame compatibility: Compatible with both Polars and Pandas DataFrame formats for integration with existing Python data analysis workflows.
• Rust-based backend: Core components implemented in Rust using Apache DataFusion query engine and Apache Arrow for efficient data representation.

Scientific Applications:

• Genomic interval analysis: Computational analysis of relationships between genomic features represented as intervals along chromosome coordinate systems.
• Large-scale overlap detection: Identification of overlapping genomic regions between feature sets, such as gene annotations and sequencing peaks.
• Genomic coverage calculation: Quantification of feature coverage across genomic intervals for enrichment and accessibility studies.
• High-throughput genomic data processing: Scalable analysis of large genomic datasets that exceed available memory through out-of-core operations.

Methodology:

polars-bio implements genomic interval operations in Rust using the Apache DataFusion query engine and Apache Arrow for efficient columnar data representation, with vectorized processing and multi-threading support for parallel execution.