Apache Airflow is an open-source platform to programmatically author, schedule, and monitor workflows as Python DAGs — providing a scalable scheduler, rich UI, operators for cloud/services, and an extensible provider ecosystem for ETL, MLOps, and infra automation.
Argo Workflows is an open-source, Kubernetes-native workflow engine (CRD) that orchestrates containerized, parallel jobs using DAG or step-based specs—ideal for CI/CD, ML training, data processing and large-scale batch workloads.
Common Workflow Language (CWL) is an open, community-driven standard for describing command-line tools and dataflow workflows to make analyses portable, reproducible, and scalable across laptops, clusters, clouds and HPC systems.
Cromwell is an open‑source workflow engine for running WDL scientific pipelines at scale across local, HPC and cloud environments, enabling reproducible, portable bioinformatics workflows and multi‑backend execution.
Dagster is an open-source data and AI orchestration platform that models data as code, providing asset-aware pipelines with lineage, observability, built-in data quality, cost insights, and a managed Dagster+ option for production.
Dockstore is an open-source ‘app store’ for bioinformatics: a registry that indexes, shares, and runs containerized tools and workflows (CWL, WDL, Nextflow, Galaxy) to enable reproducible, portable analyses across platforms.
Flyte is an open-source, Kubernetes-native workflow orchestration platform for building scalable, reproducible data, ML and analytics pipelines. It provides strong typing, dynamic DAGs, caching, lineage and SDKs to run workflows in Python or any language via containers.
Galaxy is a free, open-source, web-based platform for accessible, reproducible, and collaborative data analysis—providing thousands of bioinformatics tools, workflow authoring, training materials, and scalable execution on HPC and cloud resources.
Luigi is a Python library for building and managing complex batch workflows. It handles dependency resolution, scheduling, failure recovery and provides a web visualiser to orchestrate Hadoop, Spark, DB and local tasks reliably.
Nextflow is a dataflow DSL and workflow engine for building scalable, portable, and reproducible computational pipelines that run locally, on HPC schedulers, cloud providers or Kubernetes with container and Git integration.
nf-core is a community-curated collection of production-grade Nextflow pipelines, templates and tooling that deliver portable, reproducible bioinformatics workflows with container/Conda support, CI-tested releases and broad HPC/cloud compatibility.
Prefect is an open-source, Python-native orchestration engine that turns Python functions into resilient, production-grade workflows with built-in state tracking, retries, dynamic execution, and real-time observability.
