Configuration Variables¶

Complete reference for all configuration keys, their types, and default values. These keys can be set in any configuration file (base config.yaml, environments, profiles, or project config).

For an introduction to the configuration system, file locations, and resolution order, see Configuring epycloud.

When each config is used¶

Configuration keys are consumed at three different points: infrastructure deployment, image building, and workflow execution. Understanding this helps you know which changes require redeploying infrastructure versus just re-running a workflow.

Infrastructure deployment¶

These keys are read when you run epycloud terraform apply. Their resolved values are baked into the deployed Cloud Workflows definition and Cloud infrastructure. Changing them in your config (whether in base, an environment, or a profile) has no effect until you run terraform apply again.

Image building¶

These keys are read when you run epycloud build. They determine what goes into the Docker image. Changing them requires rebuilding.

Workflow execution¶

These keys are read each time you run epycloud run workflow. Changes take effect immediately on the next run without redeploying infrastructure. Some of these can also override the terraform-baked defaults.

Environment Variable Overrides¶

Any configuration key can be overridden using environment variables with the EPYCLOUD_ prefix. Use double underscores (__) to separate nested paths. This works for every key listed on this page.

Examples:

storage¶

Directory prefix for organizing pipeline data in GCS (or local filesystem).

Example paths after interpolation:

• pipeline/prod/flu/ (environment=prod, profile=flu)
• pipeline/dev/covid/ (environment=dev, profile=covid)

execution¶

Selects the backend used for cloud pipeline and stage execution. Configurations created before this setting was introduced continue to use GCP.

google_cloud¶

Google Cloud Platform project and region settings.

Each data-plane region needs a non-overlapping subnet range:

google_cloud:
  region: us-central1
  batch_regions:
    us-central1:
      subnet_cidr: 10.0.0.0/20
    us-east5:
      subnet_cidr: 10.1.0.0/20

google_cloud.batch¶

Cloud Batch job configuration controlling parallelism and per-stage compute resources.

google_cloud.batch.stage_a¶

Compute resources for Stage A (Builder). Single-task job that generates input files for Stage B.

google_cloud.batch.stage_b¶

Compute resources for Stage B (Runner). Parallel tasks, each processing one input file.

google_cloud.batch.stage_c¶

Compute resources for Stage C (Output). Runs as a single task that loads all Stage B results into memory, so it typically needs more memory than the other stages.

Sizing guidelines¶

Stage B (Runner):

Stage C (Output):

CPU per task and tasks per VM¶

cpu_milli and memory_mib are what one task requests. For each machine candidate, epycloud computes how many Stage B tasks fit on one VM:

tasks per VM = min(machine vCPUs / cpu_milli, (machine memory - 1024 MiB) / memory_mib)

The 1024 MiB is left for the OS and container runtime when tasks share a VM (one task may use the whole VM). The workflow caps the result at the number of Stage B tasks, and task_count_per_node (or --task-count-per-node) caps it further when set.

Even packing. Batch provisions floor(tasks / tasks per VM) VMs, so a remainder waits until a slot frees up (51 tasks at 4 per VM get 12 VMs and 3 waiting tasks). Once Stage A has produced the tasks, the workflow lowers each candidate to the largest divisor of the Stage B parallelism that keeps at least ¾ of its VM busy (51 tasks run at 3 per VM on 17 VMs). It then tries the candidates that pack evenly first, keeping machine_types order within each group. A one-task-per-VM candidate always packs evenly, so including a small machine such as c4d-highcpu-2 in the chain removes the wait for any task count.

Request one physical core per single-threaded task. On Compute Engine a vCPU is one hardware thread, and two vCPUs share each physical core. Stage B calibration runs in one thread, so cpu_milli: 1000 puts two tasks on each physical core. Measured on c3d-standard-4 with the flu model:

Sharing a core halves the price per task-hour but nearly doubles the run time, so it saves nothing and makes runs about twice as long. Keep cpu_milli: 2000 (one physical core) unless the task is I/O-bound or multithreaded.

Packing only pays off when a run has at least as many tasks as fit on one VM. For one or two tasks, a 2-vCPU machine (one physical core) costs less than a mostly idle 4-vCPU VM; order machine_types accordingly.

Machine type selection¶

When machine_types contains multiple values:

• The workflow tries candidates in order
• Every candidate gets the same per-task request; a smaller machine only fits fewer tasks
• An unsuccessful candidate is cancelled and drained before replacement

When a --stage-*-machine-type CLI option is supplied:

• The stage is pinned to that one candidate
• The per-task cpu_milli and memory_mib still come from the config, and the machine must fit at least one task

The legacy singular machine_type key remains a one-candidate compatibility path. Run epycloud config migrate to replace legacy keys with requirement-based chains.

For available machine types, pricing, and sizing recommendations, see Machine Types.

docker¶

Docker image configuration for the pipeline container.

The full image URI is constructed as:

{registry}/{google_cloud.project_id}/{repo_name}/{image_name}:{image_tag}

github¶

GitHub repository references for the modeling suite package and experiment data.

logging¶

Logging configuration for pipeline scripts and the CLI.

workflow¶

Cloud Workflows orchestration settings.

Complete Template¶

For reference, here is the full default config.yaml template:

# Storage configuration
storage:
  dir_prefix: "pipeline/{environment}/{profile}"

# Cloud execution backend
execution:
  provider: gcp

# Google Cloud Platform configuration
google_cloud:
  project_id: your-gcp-project-id
  region: us-central1
  bucket_name: your-bucket-name
  billing_project: ""

  batch:
    max_parallelism: 100
    task_count_per_node: 1

    stage_a:
      cpu_milli: 2000
      memory_mib: 7168
      machine_types: [c4d-standard-2, c3-highcpu-4, n4d-standard-2, n4-standard-2]
      max_run_duration: 3600

    stage_b:
      cpu_milli: 2000
      memory_mib: 7168
      machine_types: [c4d-standard-2, c3-highcpu-4, n4d-standard-2, n4-standard-2]
      max_run_duration: 36000

    stage_c:
      cpu_milli: 4000
      memory_mib: 15360
      machine_types: [c4d-standard-4, c3-standard-4, n4d-standard-4, n4-standard-4]
      max_run_duration: 7200
      run_output_stage: true

# Docker image configuration
docker:
  registry: "us-central1-docker.pkg.dev"
  repo_name: epymodelingsuite-repo
  image_name: epymodelingsuite
  image_tag: latest

# GitHub repositories
github:
  modeling_suite_repo: mobs-lab/epymodelingsuite
  modeling_suite_ref: main
  forecast_repo_ref: ""

# Logging configuration
logging:
  level: INFO
  storage_verbose: true

# Workflow configuration
workflow:
  retry_policy:
    max_attempts: 3
    backoff_seconds: 60
  notification:
    enabled: false
    email: null