Configuration Variables¶

Complete reference for all configuration keys, their types, and default values. These keys can be set in any configuration file (base config.yaml, environments, profiles, or project config).

For an introduction to the configuration system, file locations, and resolution order, see Configuring epycloud.

When each config is used¶

Configuration keys are consumed at three different points: infrastructure deployment, image building, and workflow execution. Understanding this helps you know which changes require redeploying infrastructure versus just re-running a workflow.

Infrastructure deployment¶

These keys are read when you run epycloud terraform apply. Their resolved values are baked into the deployed Cloud Workflows definition and Cloud infrastructure. Changing them in your config (whether in base, an environment, or a profile) has no effect until you run terraform apply again.

Image building¶

These keys are read when you run epycloud build. They determine what goes into the Docker image. Changing them requires rebuilding.

Workflow execution¶

These keys are read each time you run epycloud run workflow. Changes take effect immediately on the next run without redeploying infrastructure. Some of these can also override the terraform-baked defaults.

Environment Variable Overrides¶

Any configuration key can be overridden using environment variables with the EPYCLOUD_ prefix. Use double underscores (__) to separate nested paths. This works for every key listed on this page.

Examples:

storage¶

Directory prefix for organizing pipeline data in GCS (or local filesystem).

Example paths after interpolation:

• pipeline/prod/flu/ (environment=prod, profile=flu)
• pipeline/dev/covid/ (environment=dev, profile=covid)

google_cloud¶

Google Cloud Platform project and region settings.

google_cloud.batch¶

Cloud Batch job configuration controlling parallelism and per-stage compute resources.

google_cloud.batch.stage_a¶

Compute resources for Stage A (Builder). Single-task job that generates input files for Stage B.

google_cloud.batch.stage_b¶

Compute resources for Stage B (Runner). Parallel tasks, each processing one input file.

google_cloud.batch.stage_c¶

Compute resources for Stage C (Output). Runs as a single task that loads all Stage B results into memory, so it typically needs more memory than the other stages.

Sizing guidelines¶

Stage B (Runner):

Stage C (Output):

Machine type selection¶

When machine_type is set to a specific value (e.g., "c4d-standard-2"):

• Cloud Batch provisions that exact machine type
• cpu_milli and memory_mib act as task-level constraints (must fit within the machine)

When machine_type is empty (""):

• Cloud Batch auto-selects a VM based on cpu_milli and memory_mib
• Recommended when you don't need a specific machine family

For available machine types, pricing, and sizing recommendations, see Machine Types.

docker¶

Docker image configuration for the pipeline container.

The full image URI is constructed as:

{registry}/{google_cloud.project_id}/{repo_name}/{image_name}:{image_tag}

github¶

GitHub repository references for the modeling suite package and experiment data.

logging¶

Logging configuration for pipeline scripts and the CLI.

workflow¶

Cloud Workflows orchestration settings.

Complete Template¶

For reference, here is the full default config.yaml template:

# Storage configuration
storage:
  dir_prefix: "pipeline/{environment}/{profile}"

# Google Cloud Platform configuration
google_cloud:
  project_id: your-gcp-project-id
  region: us-central1
  bucket_name: your-bucket-name
  billing_project: ""

  batch:
    max_parallelism: 100
    task_count_per_node: 1

    stage_a:
      cpu_milli: 2000
      memory_mib: 8192
      machine_type: "c4d-standard-2"
      max_run_duration: 3600

    stage_b:
      cpu_milli: 2000
      memory_mib: 8192
      machine_type: ""
      max_run_duration: 36000

    stage_c:
      cpu_milli: 4000
      memory_mib: 15360
      machine_type: "c4d-standard-4"
      max_run_duration: 7200
      run_output_stage: true

# Docker image configuration
docker:
  registry: "us-central1-docker.pkg.dev"
  repo_name: epymodelingsuite-repo
  image_name: epymodelingsuite
  image_tag: latest

# GitHub repositories
github:
  modeling_suite_repo: mobs-lab/epymodelingsuite
  modeling_suite_ref: main
  forecast_repo_ref: ""

# Logging configuration
logging:
  level: INFO
  storage_verbose: true

# Workflow configuration
workflow:
  retry_policy:
    max_attempts: 3
    backoff_seconds: 60
  notification:
    enabled: false
    email: null