Cloud Workflows¶
Google Cloud Workflows is a serverless orchestration service that coordinates multi-step processes. For how the pipeline's workflow is structured (steps, polling, variables, error handling), see Workflows Orchestration.
Workflows documentation | Google CloudWorkflows and executions¶
A workflow is a series of steps that orchestrate Google Cloud services and events. The pipeline's workflow is defined in terraform/workflow.yaml and deployed via Terraform.
An execution is a single run of a deployed workflow. Google Cloud assigns a unique execution ID to each run. When triggered (e.g., via epycloud run workflow), the execution receives input parameters such as the experiment ID, bucket name, and parallelism settings. Multiple executions can run concurrently.
Execution states: ACTIVE → SUCCEEDED, FAILED, or CANCELLED.
For more details on the concepts:
Core concepts | Cloud Workflows | Google CloudConnectors¶
Cloud Workflows communicates with other Google Cloud services through built-in connectors. The pipeline workflow uses:
| Connector | Purpose |
|---|---|
| Cloud Batch | Submit and monitor Batch jobs for each stage |
| Cloud Storage | List and count builder artifacts between stages |
Connectors handle authentication and retries automatically.
Batch API Connector Overview | Cloud Workflows | Google CloudQuotas and limits¶
| Limit | Value |
|---|---|
| Max concurrent executions per workflow | 10,000 |
| Max execution duration | 1 year |
| Max steps per execution | 100,000 |
| Max workflow definition size | 1 MB |
See Workflows quotas and limits for the full list.