DSK

DSK performs efficient k-mer counting using a disk-streaming partitioning algorithm to reduce memory and disk requirements for genomic and transcriptomic analyses.

Key Features:

• Memory Efficiency: Operates with a fixed user-defined amount of memory by partitioning the multiset of k-mers into manageable parts processed individually.
• Disk Space Utilization: Temporarily saves partitions to disk and reloads each partition into memory for counting to minimize peak RAM usage.
• Scalability and Flexibility: Supports large values of k, extending beyond tools such as Jellyfish which are constrained to k=32 on high-memory servers.
• Performance Optimization: Balances memory, processing time, and disk space and can count all 27-mers in a human genome using 4.0 GB of memory and ~160 GB of disk space in 17.9 hours.
• Low-abundance k-mer Filtering: Offers an optional filter to remove low-abundance k-mers to focus on higher-abundance signals and reduce computational load.

Scientific Applications:

• Genome Assembly: Provides k-mer frequency data used as a foundational step in genome assembly workflows.
• Error Correction: Supplies k-mer counts that support sequencing error correction methods.
• Metagenomics Analysis: Enables k-mer–based profiling and comparison in metagenomic datasets.
• Comparative Genomics: Facilitates comparative analyses that rely on k-mer frequency comparisons across genomes.

Methodology:

Partition the dataset into smaller segments, stream partitions to disk, load each partition into a temporary hash table in memory, count k-mers per partition, and aggregate counts across partitions.