SpaRC

SpaRC: Scalable Apache Spark-Based Metagenomic Read Clustering

SpaRC performs distributed metagenomic read clustering using Apache Spark to process large-scale sequencing datasets with high accuracy and low false positive rates.

Key Features:

• Distributed Scalability: Utilizes Apache Spark distributed computing to handle large metagenomic sequencing datasets efficiently.
• Low False Positive Clustering: Implements a read clustering method with minimal false positive rates to accurately distinguish distinct genetic sequences.
• Secondary Cluster Refinement: Re-clusters small clusters using statistical data across multiple samples to reduce under-clustering of reads from the same genome.
• Parameter Optimization: Supports exploration of clustering parameters to optimize performance across genomes with varying sequencing coverage levels.

Scientific Applications:

• Metagenomic Genome Reconstruction: Clusters reads prior to assembly to enable parallel genome assembly and improve resolution of microbial community structure in complex environmental and microbiome samples, including synthetic mouse gut microbiome datasets.

Methodology:

SpaRC applies an initial scalable read clustering algorithm implemented on Apache Spark, followed by a secondary clustering step that leverages inter-sample statistical information to refine small clusters. This two-stage approach minimizes under-clustering and over-clustering while maintaining high accuracy across variable sequencing coverage conditions.