Quip

Quip performs lossless compression of next-generation sequencing (NGS) data in FASTQ and SAM/BAM formats by combining reference-based and assembly-based compression with statistical compression of sequencing metadata.

Key Features:

• Lossless Compression: Preserves all original sequencing information without loss during compression.
• Reference-Based Compression: Aligns reads to existing genome references to represent reads relative to known sequences for reduced storage.
• Assembly-Based Compression: Uses a de novo assembly algorithm to assemble reads into contigs and store reads as positions within assembled contigs.
• Probabilistic Data Structure for Assembly: Employs a probabilistic data structure to reduce memory requirements compared to traditional de Bruijn graph assemblers, enabling assembly of millions of reads.
• Statistical Compression: Applies statistical compression to read identifiers, quality scores, alignment information, and sequences.
• High Compression Ratio: Combines methods to reduce datasets to less than 15% of their original size in reported cases.

Scientific Applications:

• High-Throughput Sequencing Data Storage: Reduces storage requirements for large-scale NGS projects while maintaining data integrity.
• Data Transfer and Archival: Lowers data volume for transmission and long-term archival of FASTQ and SAM/BAM files.
• Integration into Bioinformatics Pipelines: Preserves raw data fidelity for downstream genomic analyses that require original read and quality information.

Methodology:

Quip combines reference-based compression, a de novo assembly algorithm that uses a probabilistic data structure to assemble reads into contigs and store read positions, and statistical compression applied to read identifiers, quality scores, alignment information, and sequences.