MetaSpark:

MetaSpark performs distributed recruitment of metagenomic reads to reference genomes using Apache Spark to enable scalable analysis of large next-generation sequencing (NGS) datasets.

Key Features:

• Spark-based distributed processing: Implements computation on Apache Spark to distribute workloads across cluster nodes.
• RDD in-memory caching: Utilizes the resilient distributed dataset (RDD) framework to cache datasets in memory across cluster nodes.
• Metagenomic read recruitment: Recruits metagenomic reads to reference genomes for reference-based analysis of NGS data.
• Scalability: Designed to manage large-scale metagenomic datasets and scale across cluster and cloud computing environments.
• Performance improvement: Empirically outperforms SOAP2, BWA, LAST, and FR-HIT, recruiting more reads and achieving approximately a 4% increase over FR-HIT on a 1 million read dataset against 0.75 GB of reference genomes.
• Benchmarked across dataset sizes: Demonstrated robust performance across varying dataset sizes and complexities.

Scientific Applications:

• Large-scale metagenomic analysis: Processing and analysis of extensive NGS metagenomic datasets via reference recruitment.
• Reference-based read mapping: Mapping metagenomic reads to reference genomes for downstream comparative analyses.
• Cloud and cluster-based workflows: High-throughput metagenomic processing in clustered or cloud computing environments.

Methodology:

Uses Apache Spark's distributed processing with the resilient distributed dataset (RDD) framework and in-memory caching across cluster nodes to recruit metagenomic reads to reference genomes.