SciLuigi

SciLuigi extends Luigi to manage and automate complex bioinformatics and predictive modeling workflows for drug discovery and related scientific applications.

Key Features:

• Flow-Based Programming Paradigm: Incorporates flow-based programming principles to enable modular design and composition of interdependent workflow tasks.
• Enhanced Workflow Automation: Extends Luigi to automate multi-step predictive modeling workflows including cross-validation and parameter tuning across dependent tasks.
• Fault-Tolerant Execution: Provides mechanisms for fault-tolerant execution to handle large-scale data and computationally intensive modeling methods.
• Integration with High-Performance and Cloud Computing: Supports execution on high-performance computing clusters and cloud environments to address computational resource demands.

Scientific Applications:

• Predictive modeling for drug discovery: Orchestrates complex predictive-modeling pipelines used in drug discovery workflows.
• Modeling biochemical interactions: Manages large-scale modeling of biochemical interactions through composed, interdependent tasks.
• Large-scale predictive analyses on shared resources: Enables extensive predictive analyses using shared computer cluster resources and cloud infrastructures.

Methodology:

Extends Spotify's Luigi workflow system with flow-based programming elements to manage task dependencies, enable cross-validation and parameter tuning workflows, and support fault-tolerant execution on HPC and cloud infrastructures.