MQF

MQF implements a Mixed-Counters Quotient Filter that extends the Counting Quotient Filter (CQF) to store k-mers, counts, and labels with improved space and time efficiency for scalable k-mer counting and metadata management in large sequencing datasets.

Key Features:

• Novel Counting System: Integrates a counting mechanism that adapts to varying data distributions to improve space efficiency and achieves faster insertions and queries than the Counting Quotient Filter (CQF) in tested scenarios.
• Buffered Storage Option: Provides a buffered mode that offloads storage to disk to substantially reduce memory usage while trading some insertion and query speed for lower RAM requirements.
• Advanced Labeling System: Assigns labels to items with high data locality and compact representation, functioning as a minimal perfect hash approximately ten times faster than BBhash and not requiring re-analysis of the original data for additional insertions or deletions.

Scientific Applications:

• K‑mer counting: Stores k-mers with their counts and metadata for large-scale k-mer counting from high-throughput sequencing data.
• Genome assembly: Supplies compact k-mer counts and labels to inform assembly algorithms and reduce memory footprint during assembly-related analyses.
• Metagenomics: Handles large k-mer datasets and counts to support community profiling and comparative analyses in metagenomic studies.
• Variant calling: Manages k-mer counts and associated metadata that can be used in variant detection and validation workflows.

Methodology:

MQF extends the CQF by integrating a dynamic counting mechanism that adapts to data distributions, offers a buffered disk-backed storage mode to reduce memory usage, and implements a compact labeling system that serves as a minimal perfect hash and enables rapid access without reprocessing original data.