bio2zarr

bio2zarr converts VCF files to the Zarr format, enabling efficient storage and processing of large-scale genetic variation data.

Key Features:

• VCF to Zarr Conversion: Converts VCF files to the Zarr format for optimized data access.
• Biobank-Scale Optimization: Leverages Zarr's multidimensional data storage capabilities for massively parallel processing of large datasets.
• Field and Sample-Based Access: Enables efficient retrieval of data by field or sample, addressing the inefficiency of row-wise VCF encoding.
• Cloud and GPU Compatibility: Supports high-performance applications using cloud computing and GPUs.
• Cross-Species Applicability: Handles human genome datasets as well as whole genome datasets for non-model organisms such as Norway Spruce and SARS-CoV-2.

Scientific Applications:

• Human Population Genomics: Enables analysis of large human cohorts including Genomics England (n=78,195), Our Future Health (n=651,050), and All of Us (n=245,394).
• Non-Model Organism Genomics: Supports whole genome analysis of Norway Spruce (n=1,063) and large-scale viral datasets such as SARS-CoV-2 (n=4,484,157).
• Cloud-Based Variant Analysis: Facilitates cost-effective, high-performance genetic variation analysis directly from cloud-based object stores.

Methodology:

bio2zarr implements the VCF Zarr specification, an encoding of the VCF data model using the Zarr format for storing multidimensional data. The conversion process transforms row-wise VCF data into a columnar Zarr structure optimized for field and sample-based access patterns.