PipeMEM

PipeMEM accelerates BWA-MEM-based DNA sequence alignment by integrating BWA-MEM with Apache Spark to enable multi-node, low-overhead, high-throughput genome alignment.

Key Features:

• BWA-MEM integration: Optimizes the single-node BWA-MEM aligner for distributed execution while retaining its speed and accuracy.
• Apache Spark integration: Leverages Apache Spark as the big data platform to run distributed alignment tasks.
• Spark pipe operation: Uses the pipe operation in Spark to facilitate efficient data processing and communication between nodes.
• Pipeline structure: Employs a pipeline architecture combined with in-memory computation techniques to accelerate processing.
• Reduced disk I/O: Minimizes disk I/O operations to enhance throughput and lower computational overhead.
• Latency and resource reduction: Lowers latency and resource consumption typical of distributed computing environments.
• Paired-end alignment optimization: Demonstrates improved performance specifically for paired-end alignment tasks.
• Empirical performance gains: Achieves reported speedups of 2.27× over BWASpark (Genome Analysis Toolkit) and 2.33× over SparkBWA in experiments.

Scientific Applications:

• Multi-node genome alignment: Scales BWA-MEM to multi-node environments for large-scale genome alignment.
• Paired-end sequencing analysis: Accelerates paired-end alignment workflows used in genome analysis.
• High-throughput genomic datasets: Processes large-scale genomic datasets with reduced computational burden and increased throughput.
• Genome analysis pipelines: Integrates into genome analysis applications requiring accurate, high-throughput alignment in distributed settings.

Methodology:

Integrates BWA-MEM with Apache Spark and applies the Spark pipe operation, a pipeline structure, and in-memory computation to reduce disk I/O, latency, and resource consumption.