SeQual-Stream

SeQual-Stream performs streaming quality control of next-generation sequencing (NGS) datasets to enable scalable, on-the-fly preprocessing and validation of DNA sequence reads.

Key Features:

• Streaming paradigm: Utilizes a streaming model to perform real-time quality control operations as sequencing data become available.
• Scalability and performance: Built on Apache Spark and HDFS to exploit stream processing, with experimental results reporting up to 2.7× speedup versus batch processing on datasets exceeding 250 million DNA sequences.
• Format support: Supports single-end and paired-end reads in FASTQ and FASTA formats.
• Distributed processing: Integrates with HDFS to enable distributed processing across multiple compute nodes.

Scientific Applications:

• Real-time QC during data acquisition: Performs quality control on-the-fly during data transfer or download to reduce time before downstream analysis.
• Preprocessing for large-scale genomic studies: Accelerates preprocessing of very large NGS datasets (for example, datasets >250 million sequences) used in population genomics and large-scale sequencing projects.
• QC for distributed storage workflows: Applies quality control in workflows that use remote repositories or distributed file systems such as HDFS.

Methodology:

Implements stream processing on Apache Spark over HDFS to perform on-the-fly quality control of single-end and paired-end FASTQ/FASTA reads, eliminating the need for complete dataset availability before initiating QC.