Flint

Flint performs scalable metagenomic profiling by aligning Illumina paired-end sequencing reads to large bacterial reference collections using Apache Spark to enable high-throughput metagenomic whole-genome sequencing (mWGS) analyses.

Key Features:

• Scalability: Built on the Apache Spark framework, Flint processes large-scale datasets and has been demonstrated on a 170 GB reference collection comprising 43,552 bacterial genomes from Ensembl.
• Speed and Efficiency: Using Spark's parallelism and streaming engine architecture, Flint profiles 1 million Illumina paired-end reads against over 40,000 genomes on 64 machines in 67 seconds.
• Cost-Effectiveness: When deployed on Amazon Elastic MapReduce (EMR), Flint sustains mapping rates up to 55 million reads per hour with an example cluster hourly cost of $8.00 USD.
• Real-Time Processing: Flint's streaming capability enables real-time processing of sequencing data and avoids storing large volumes of intermediate alignments.

Scientific Applications:

• Metagenomic whole-genome sequencing (mWGS): Enables comprehensive profiling of bacterial populations using extensive reference genome collections.
• Environmental microbiology: Supports large-scale community profiling in environmental samples.
• Human health studies: Enables high-resolution characterization of microbial communities in clinical or human-associated samples.
• Biotechnology: Facilitates microbial composition analysis for biotechnological applications.

Methodology:

Flint aligns sequencing reads to a large collection of bacterial genomes using Spark's distributed computing, leveraging parallel processing and streaming data techniques to map reads in real time.