SparkGA2

SparkGA2 provides a scalable, memory-efficient Apache Spark-based framework for processing Next Generation Sequencing (NGS) DNA data for large-scale genomic analysis.

Key Features:

• Scalability: Leverages Apache Spark distributed computing to scale across clusters, demonstrated on configurations from 6-node to 67-node setups.
• Memory Efficiency: Implements on-the-fly compression of intermediate data to reduce memory footprint by up to three times, with a modest performance trade-off.
• Performance Optimization: Uses Apache Spark in-memory caching to accelerate processing, reporting up to 22% faster performance on large clusters and 9% on smaller clusters, and up to 51% speed increase with streaming pre-processing on a 6-node cluster.
• Streaming Approach: Processes input data incrementally as it is received to improve processing speed and adapt resource utilization under varying memory constraints.

Scientific Applications:

• Genetic disease diagnosis: Enables rapid analysis of NGS DNA datasets to identify genetic variations associated with disease for research and clinical investigations.

Methodology:

Builds upon the Broad Institute's DNA analysis pipeline best practices and combines on-the-fly compression, Apache Spark distributed computing and in-memory caching, and streaming data pre-processing.