HULK

HULK generates compact, similarity-preserving histosketches from streaming k-mer spectra to enable rapid metagenomic comparison and classification of microbiome sequencing data.

Key Features:

• Compact Representation: Generates small, fixed-size histosketches as compressed representations of microbiome k-mer spectra for dimensionality reduction and storage efficiency.
• Rapid Dissimilarity Analysis: Enables fast pairwise Jaccard similarity estimation and sample clustering using histosketch comparisons.
• Efficient Streaming Processing: Implements streaming histogram sketching to process large microbiome datasets quickly (example: ~2 GB processed in ~50 seconds on a four-core laptop) with histosketches of ~3000 bytes.
• Locality Sensitive Hashing Indexing: Incorporates a locality sensitive hashing (LSH) indexing scheme to accelerate similarity searches across large microbiome collections.
• Machine Learning Integration: Trains machine learning classifiers, including random forest models, on histosketches and supports rapid real-time classification (example: 108 neonatal samples, 97% accuracy, 96% precision; classification in <3 seconds).

Scientific Applications:

• Clinical metagenomics: Rapidly compare and classify microbiome samples for clinical studies and diagnostics.
• Microbial diversity analysis: Quantify and compare k-mer–based diversity across samples using compact sketches.
• Host–microbe interaction studies: Enable large-scale comparisons of microbiome profiles in host-associated research.
• Diagnostic development: Support development of microbiome-based diagnostic classifiers using sketch-derived features.
• Dataset augmentation and reanalysis: Facilitate adding to and reanalyzing published microbiome datasets via compact, searchable representations.
• Study design and sample clustering: Assist in clustering samples by type and informing study design through rapid dissimilarity estimation.

Methodology:

Creates similarity-preserving sketches from streaming k-mer spectra (histosketching) via streaming histogram sketching, estimates pairwise Jaccard similarity, indexes sketches with locality sensitive hashing, and trains classifiers such as random forest models on histosketch features for real-time classification.