SparkGC

SparkGC compresses large collections of genomes using Apache Spark to provide scalable, in-memory genome compression that reduces storage and transmission requirements for large-scale genomic datasets.

Key Features:

• Scalability: Implements a distributed computing architecture on Apache Spark to handle vast genomic datasets across clustered nodes.
• Efficiency in Compression: Uses a two-stage (first-order and second-order) compression approach with in-memory processing to improve compression ratio, reported to exceed state-of-the-art methods by at least 30%.
• Speed: Leverages Spark's in-memory computation to reduce runtime, achieving reported speeds at least 3.8 times faster than leading methods on a single worker node and scaling with additional nodes.
• Cost-Effectiveness: Employs distributed and in-memory computation on Apache Spark to lower compute and storage resources required for large-scale genome compression.

Scientific Applications:

• Genomic Data Storage: Reduces storage footprint of sequencing-derived and assembled genomes for large collections and repositories.
• Data Transmission: Produces high-compression outputs suitable for transmitting large genomic datasets between sites or institutions.
• Bioinformatics Research: Enables handling of massive genomic datasets in comparative genomics, population genetics, and evolutionary biology analyses.

Methodology:

SparkGC applies a two-stage compression process—an initial data reduction stage followed by a second-order refinement—while maintaining data activity in memory between the first-order and second-order compression stages and leveraging Apache Spark's distributed, in-memory computation.