ArrowSAM

ArrowSAM leverages Apache Arrow and the Plasma shared-memory object store to represent Sequence Alignment/Map (SAM) columnar data in-memory for high-performance, cross-language genomic data processing.

Key Features:

• In-Memory Data Representation: ArrowSAM employs an in-memory SAM format using Apache Arrow's language-independent columnar layout to avoid disk I/O and (de)-serialization between applications.
• Shared Memory Object Store: ArrowSAM uses the Plasma shared-memory object store to enable direct shared-memory access and parallel processing of tabular genomic data.
• Integration with Genomics Workflows: ArrowSAM integrates with BWA-MEM, Picard, and GATK to enable in-memory communication among these tools within variant calling workflows.
• Performance Optimization: ArrowSAM targets GATK best practices for germline analysis on whole genome sequencing (WGS) and whole exome sequencing (WES) datasets and reports overall execution speedups of 4.85x for WGS and 4.76x for WES.
• Comparative Advantage: ArrowSAM's columnar in-memory representation outperforms ramDisk and Unix pipes, achieving speedups of 1.45x for WGS and 1.27x for WES over the second fastest workflow.
• Scalability and Efficiency: ArrowSAM exploits cache locality and shared-memory parallel scalability to reduce memory accesses and accelerate tasks such as sorting, duplicate removal, and base quality score recalibration.

Scientific Applications:

• Variant Calling: ArrowSAM accelerates high-throughput variant calling workflows by enabling rapid in-memory processing of SAM-formatted alignment data.
• Germline Analysis on WGS and WES: ArrowSAM supports GATK best-practice germline analysis on whole genome sequencing and whole exome sequencing datasets with reported execution speedups.
• Large-Scale NGS Data Processing: ArrowSAM is applicable to processing large next-generation sequencing (NGS) datasets where reduced I/O and inter-process copying improve throughput.

Methodology:

ArrowSAM stores SAM columnar data in Apache Arrow's in-memory format and uses the Plasma shared-memory object store to minimize (de)-serialization and copying between processes while enabling parallel shared-memory access.