Kipper

Kipper provides versioning and efficient storage management for large sequence databases, enabling precise tracking of insertions, deletions, and updates to support reproducible bioinformatics analyses.

Key Features:

• Version Control: Implements a key-value large-file versioning system that records insertions, deletions, and updates and enables recall of specific versions by date or version ID.
• Efficient Storage Management: Stores multiple versions of large databases in a file marginally larger than the latest version alone, reducing storage requirements for databases such as NCBI nr (which can exceed 50 GB per version).
• Integration with Bioinformatics Pipelines: Integrates with BioMAJ for downloading sequence databases and with Galaxy (including the Galaxy Versioned Data tool) to manage database versions within workflows.
• Implementation: Implemented in Python (v1.0.0) and built on a key-value data store mechanism for item-level change tracking.

Scientific Applications:

• Reproducibility: Preserves exact sequence database versions used in analyses to enable precise reproduction of results.
• Validation and Testing: Allows validation of new tools or workflows against historical database versions.
• Impact Analysis: Enables assessment of how database updates affect sequencing data analyses by tracking changes across versions.

Methodology:

Kipper uses a key-value data store that records item insertions and deletions, stores version metadata for recall by date or version ID, and compresses multiple versions into a single file; it integrates with BioMAJ and Galaxy.