BigMPI4py

BigMPI4py enables parallel processing of very large Python objects (exceeding 2^31 bytes) to accelerate large-scale bioinformatics analyses such as whole genome bisulfite sequencing (WGBS) CpG methylation studies.

Key Features:

• Support for Large Objects: Overcomes the mpi4py 2^31-byte object size limit to enable communication of objects larger than 2^31 bytes.
• Automatic Distribution Strategy: Automatically determines and applies an optimal strategy to distribute large objects across multiple processors.
• Vectorized Methods: Employs vectorized operations to improve parallelization efficiency.
• MPI and mpi4py Integration: Leverages the Message Passing Interface (MPI) and integrates with mpi4py for parallel execution on multicore workstations and high-performance computing systems.
• Parallel Statistical Tests: Supports parallel implementation of statistical tests such as the Kruskal-Wallis test for differential methylation evaluation.

Scientific Applications:

• WGBS CpG methylation analysis: Accelerated analysis of whole genome bisulfite sequencing (WGBS) data from the ENCODE project across 59 human tissue samples.
• Discovery of methylation motifs: Enabled the search for germ line-specific de novo DNA methylated and unmethylated motifs.
• Differential methylation testing: Facilitated rapid evaluation of differential methylation in CpG sites across germ layers using parallel Kruskal-Wallis tests.
• Large-scale CpG analysis: Demonstrated analysis of 55 million CpGs with a 22-fold speedup using 25 cores.
• Identification of tissue-associated methylation: Contributed to identification of hypermethylated genes associated with ectoderm, mesoderm, and endoderm-related tissues.

Methodology:

Uses MPI via mpi4py to distribute and communicate objects larger than 2^31 bytes by automatically selecting distribution strategies and applying vectorized operations; implements parallel statistical tests such as the Kruskal-Wallis test across multicore and high-performance computing systems.