BioPig

BioPig provides scalable, parallel processing of large-scale genomic sequence data using the Pig data-flow language on Apache Hadoop's MapReduce framework.

Key Features:

• Scalability: Scales to large genomic datasets and has been tested on datasets up to 500 gigabases (Gb), leveraging Hadoop MapReduce to adjust computation to input size.
• Programmability and Development Efficiency: Uses the Pig language to express complex data transformations in a high-level scripting environment, reducing development time compared to serial or Message Passing Interface (MPI)-based algorithms.
• Portability: Runs across Hadoop infrastructures without code modification and has been deployed on systems such as the Magellan system at the National Energy Research Scientific Computing Center and on Amazon Elastic Compute Cloud (EC2).

Scientific Applications:

• Sequence alignment: Enables high-throughput processing of sequence alignment workloads on large genomic datasets.
• Variant calling: Provides scalable data handling and preprocessing support for variant calling workflows.
• Large-scale genomic analyses: Facilitates computationally intensive analyses in genomics, personalized medicine, and evolutionary biology that require processing hundreds of gigabases.

Methodology:

Implements Hadoop's distributed MapReduce framework and the Pig data-flow language to parallelize data processing across multiple nodes, distributing computations and minimizing data-transfer overheads.