SWAPCounter

SWAPCounter performs distributed k-mer counting of high-throughput sequencing data to support large-scale analyses such as de novo genome assembly and error correction.

Key Features:

• MPI Streaming I/O Module: Facilitates rapid loading of massive datasets by leveraging MPI streaming capabilities to maximize data throughput.
• Counting Bloom Filter Module: Uses a counting Bloom filter to reduce memory and communication overhead for large-scale k-mer counting.
• Scalable distributed architecture: Overlaps counting steps across processes to achieve high parallel efficiency in high-performance computing environments.
• Comparative performance: Performs competitively against KMC2 and MSPKmerCounter in shared memory environments.
• Strong-scaling demonstration: Achieved 79% parallel efficiency when scaled to 32,768 cores on the Cetus supercomputer processing 4 TB of sequence data from the 1000 Genomes project.

Scientific Applications:

• De novo genome assembly: Provides k-mer counts used in assembly workflows for reconstructing genomes from sequencing reads.
• Sequencing error correction: Supplies k-mer frequency information for identifying and correcting sequencing errors.
• Population-scale sequence analysis: Enables processing of large datasets such as the 1000 Genomes project for population genomics studies.

Methodology:

Distributed k-mer counting using MPI streaming I/O and a counting Bloom filter with overlapped counting steps and strong-scaling experiments to evaluate parallel efficiency up to 32,768 cores on the Cetus supercomputer.