bio2zarr

bio2zarr converts Variant Call Format (VCF) files into the Zarr format to provide chunk-compressed, per-field storage and parallel access for efficient handling and analysis of large-scale genomic datasets.

Key Features:

• Efficient Data Encoding: Encodes each VCF field separately into chunk-compressed binary arrays according to the VCF Zarr specification.
• Scalability and Performance: Supports massively parallel processing and reports competitive compression ratios and single-threaded performance versus specialized genotype storage methods.
• Random and Field-Specific Access: Facilitates rapid retrieval of specific fields or samples without reading entire VCF rows.
• Cloud and GPU-Accelerated Processing: Enables high-performance applications through cloud computing and GPU utilization.
• Cost Reduction via Efficient Access Patterns: Reduces storage and processing costs by enabling more efficient data access patterns and leveraging cloud resources.
• Compatibility with File-Oriented Workflows: Maintains interoperability with existing file-oriented bioinformatics pipelines.

Scientific Applications:

• Biobank-scale whole-genome analysis: Applied to biobank-scale datasets encompassing hundreds of thousands of whole genomes, including Genomics England, Our Future Health, and All of Us.
• Non-human and pathogen genomics: Applied to whole-genome datasets for species such as Norway Spruce and the SARS-CoV-2 virus.
• High-performance cloud-based analyses: Enables cloud- and GPU-accelerated analyses and demonstrations on subsets of large human and other genomic projects.

Methodology:

Implements the VCF Zarr specification by restructuring the VCF data model to encode each field separately as chunk-compressed binary arrays optimized for parallel processing and efficient retrieval, enabling access to specific fields or samples without processing entire rows.