vcztools

vcztools leverages the Zarr data format to encode and process Variant Call Format (VCF) data, enabling field- and sample-oriented storage and efficient computation on large genomic datasets.

Key Features:

• Zarr-Based Encoding: Encodes VCF data using the Zarr format to enable multi-dimensional, field- and sample-oriented storage.
• Partial bcftools Reimplementation: Provides a partial reimplementation of bcftools focused on Zarr-backed VCF handling.
• Field-Separated Chunk-Compressed Binary Encoding: Stores VCF fields separately in chunk-compressed binary arrays to improve access patterns and compression.
• Scalability and Performance: Optimizes for biobank-scale datasets and reports improved compression ratios and single-threaded calculation performance versus traditional VCF encodings.
• Cloud and GPU Integration: Facilitates cloud computing and GPU utilization for high-performance genomic computations.
• Compatibility with File-Oriented Workflows: Maintains compatibility with existing file-oriented workflows despite switching to field-oriented encoding.

Scientific Applications:

• Large-scale human genomics: Enables storage and analysis of biobank-scale whole-genome datasets including Genomics England, Our Future Health, and All of Us.
• Population-scale variant analysis: Suited for hundreds of thousands of genomes and terabytes of compressed VCF data.
• Plant genomics: Applied to whole-genome datasets such as Norway Spruce.
• Viral genomics: Applied to SARS-CoV-2 whole-genome datasets.

Methodology:

Implements the VCF Zarr specification and a partial reimplementation of bcftools, encoding VCF fields separately as chunk-compressed binary Zarr arrays and supporting massively parallel processing with cloud and GPU-enabled execution.