PyBDA

PyBDA enables automated, distributed analysis of large-scale biological datasets for statistical and machine learning investigations in high-performance computing (HPC) environments.

Key Features:

• Automated distributed analysis: Performs automated, distributed analysis of large-scale biological datasets.
• Apache Spark backend: Uses Apache Spark as its backend framework to enable scalable distributed computation.
• Snakemake integration: Integrates Snakemake to schedule and manage computational jobs across distributed HPC clusters.
• HPC support: Tailored for high-performance computing environments to leverage cluster resources.
• Scalability: Scales to handle hundreds of millions of data points and high-dimensional datasets beyond conventional applications.
• Statistical and machine learning methods: Executes common statistical methods and machine learning algorithms on large datasets.
• Proven at scale: Demonstrated on image-based RNA interference (RNAi) data comprising 150 million single cells.

Scientific Applications:

• Image-based RNAi single-cell analysis: Analysis of image-based RNA interference (RNAi) datasets at single-cell scale, demonstrated on 150 million single cells.
• High-dimensional data analysis: Analysis of high-dimensional bioinformatics datasets comprising hundreds of millions of data points.
• Large-scale statistical and machine learning studies: Application of statistical methods and machine learning algorithms to large biological datasets in HPC settings.

Methodology:

Uses Apache Spark as the backend and integrates Snakemake to schedule and manage computational jobs across distributed HPC clusters to perform automated distributed analysis of large-scale datasets.