FlowCron

FlowCron wraps Globus into a function-as-a-service to enable automated data transfer, analysis, and return on High-Performance Computing (HPC) clusters with minimal user training.

Key Features:

• Automated HPC Pipeline Execution: Users start analysis through the Globus website, after which data is encrypted, uploaded to the HPC, analysed, and returned to the originating machine along with a record of the analysis.
• Reduced Administrative Burden: Transfers responsibilities of installation and upkeep of data processing pipelines from users to HPC project PIs and/or HPC project managers.
• Reproducibility and Logging: All actions performed are documented and logged, encouraging reproducibility of results and reducing effort associated with troubleshooting, maintenance, and support.
• Ubiquitous Software Dependencies: Relies on common HPC software dependencies, specifically linking Globus and cron, making it simple to install and use.
• User Authentication: Requires appropriate user authentication for Globus to transfer data to and from the HPC, ensuring only authorised users can send data to the HPC.
• Optional Data Cleanup: After returning processed data to the user, the system optionally deletes any data or files left in the cluster.

Scientific Applications:

• Cryogenic Electron Microscopy (cryo-EM): Accelerates scientific throughput in computationally intensive cryo-EM workflows by enabling timely response to experimental results and protocol correction.

Methodology:

FlowCron links Globus and cron to produce an open-source system that requires little administrative support; data transfer requires appropriate user authentication through Globus.