ViraPipe

ViraPipe analyzes metagenomic next-generation sequencing (NGS) datasets to identify and characterize viral sequences using an Apache Spark–based distributed computing framework.

Key Features:

• Scalability: Built on Apache Spark to process large numbers of samples, enabling handling of thousands of human microbiome samples across a cluster.
• Parallel Processing: Integrates parallelized tools including BWA-MEM for read alignment, MegaHit for de novo assembly, and BLAST and HMMER3 for sequence searching to run concurrently across computing cores.
• Efficiency: Demonstrated to process 768 human samples in approximately 210 minutes on a Spark cluster with 23 nodes and 1,288 cores, achieving an ~11-fold speedup versus sequential execution on a single node.
• Versatility: Tuned for viral metagenome analysis but adaptable to other metagenomic study types.

Scientific Applications:

• Human microbiome research: Rapid analysis of large NGS cohorts to detect and characterize viral components of the human microbiome.
• Viral genome identification: Detection and annotation of viral genomes within complex biological samples.
• Microbial diversity assessment: Characterization of viral contributions to microbial community composition.
• Pathogen detection: Identification of viral pathogens in metagenomic datasets.
• Viral ecology studies: Investigation of the role of viruses in various ecosystems using large-scale metagenomic data.

Methodology:

Parallel decompression of NGS data; read interleaving and alignment using BWA-MEM; filtering and normalization to isolate non-human reads; de novo assembly with MegaHit; sequence searching and annotation with BLAST and HMMER3.