Machine Types¶

Google Cloud offers different machine types that vary in CPU architecture, memory, and price. Cloud Batch provisions VMs of the chosen machine type to run each pipeline task.

For how cpuMilli and memoryMib map to machine resources, see Cloud Batch: Resource units. For recommended configurations per stage, see Sizing Recommendations.

Machine family overview¶

A machine family groups machine types that share the same CPU platform and pricing model. The pipeline primarily uses C4D (AMD compute-optimized) for its strong single-thread performance at a competitive price, but other families are available depending on your workload.

Here are some examples of machine families that can fit your needs:

For details on each machine families and pricing, see:

Machine families resource and comparison guide | Google Clouddocs.cloud.google.com

VM instance pricing | Google Cloudcloud.google.com

Machine type selection¶

How it works¶

When Cloud Batch runs a job, it provisions VMs to execute tasks. There are two approaches to control which VMs get provisioned:

Auto-select: Specify only CPU and memory requirements. Cloud Batch picks a machine type that satisfies them. This is simpler but less predictable, as Google may choose slower machine families (e.g., E2) when requirements are low.

Explicit machine type: Specify the exact machine type (e.g., c4d-standard-2). Cloud Batch provisions that specific VM. The CPU and memory values become task-level constraints and must not exceed the machine's capacity. If the values are smaller than the machine's capacity, Cloud Batch may pack multiple tasks onto the same VM (controlled by task_count_per_node). This gives predictable provisioning and consistent performance.

For details on how Batch automatically creates and deltes resources that meet specification, see:

Job resources | Cloud Batch | Google Clouddocs.cloud.google.com

Pipeline configuration¶

In the pipeline, machine type selection is configured per stage:

google_cloud:
  batch:
    stage_b:
      cpu_milli: 2000
      memory_mib: 4096
      machine_type: ""  # Cloud Batch picks the VM

google_cloud:
  batch:
    stage_b:
      cpu_milli: 2000
      memory_mib: 7168
      machine_type: "c4d-standard-2"

See Configuration Variables for all per-stage resource settings.

Naming convention¶

Google Cloud machine types follow the pattern: {family}-{type}-{vcpus}

• Family: e2, n2, c4, c4d, etc.
• Type: standard (balanced), highmem (more memory per vCPU), highcpu (more vCPUs per memory)
• vCPUs: Number of virtual CPUs (2, 4, 8, 16, ...)

Examples:

• c4d-standard-2: C4D family, standard memory, 2 vCPUs
• c4d-highmem-4: C4D family, high memory, 4 vCPUs
• e2-standard-4: E2 family, standard memory, 4 vCPUs

For full list of machine types, see:

VM instance pricing | Google Cloudcloud.google.com

Common machine types¶

Machine types commonly used by the pipeline. A typical calibration task in Stage B for a single state uses approximately 4 GB of memory. For per-stage defaults and guidance, see Sizing Recommendations.

2-vCPU machine types¶

4-vCPU machine types¶

Use these for memory-intensive workloads such as Stage C (output aggregation), which loads all Stage B results into memory.

For Cloud Build machine types, see Cloud Build.