Machine Runner Orchestrator performance benchmarks

Resource class: my-org/runner-large — 8 vCPU / 32 GiB, maxReplicas=9

Steps 0 and 1 (up to 2 jobs/min) kept queue p90 at or below 106 seconds. At step 2 (4 jobs/min), queue p90 jumped from 82 s to 518 s. At step 3 (6 jobs/min), queue p90 reached 1574 s (26 minutes). All 130 jobs completed with zero errors, but wait times at steps 2 and 3 make these rates unsuitable for latency-sensitive workloads.

maxReplicas is a capacity ceiling you set explicitly. When sustained arrival rate exceeds what that ceiling can drain, MRO queues incoming jobs until running VMs finish and free capacity. The key result: MRO completes every job — the workload waits, but nothing drops.

Resource class: my-org/runner-small — 2 vCPU / 8 GiB, maxReplicas=34

Queue p90 stayed at or below 86 seconds across every step, even at 60 jobs and 6 jobs/min. maxReplicas=34 (versus 9 in the large-class run) lets the pool sustain a high enough concurrent VM count to drain 5-minute jobs without queue buildup. Queue times stay low when maxReplicas exceeds the peak arrival rate — the pool has enough concurrent capacity to drain jobs as they arrive. All runs completed with zero errors.

The test randomly assigned jobs to either the large or small resource class. Step totals vary due to random assignment but land near 50/50.

Large-class jobs in the mixed run:

Small-class jobs in the mixed run:

Mixing resource classes reduced queue times for large-class jobs. At step 3 (6 jobs/min), large-class queue p90 dropped from 1574 s in the large-only run to 417 s in the mixed run — a 73% reduction. In the large-only run, all 60 jobs at step 3 competed for the same nine replica slots. In the mixed run, only around half the jobs targeted the large class, so the nine-replica pool had lower concurrent demand and cleared the queue faster.

Small-class jobs queued little in all cases (queue p90 ≤ 57 s at every step), consistent with the small-only results. All jobs in the mixed run completed with zero errors.

All three experiments completed with zero job errors.

Both resource classes shared the same node group (ASG 2–6, c8i.12xlarge).

Resource class: my-org/runner-large — 8 vCPU / 32 GiB, maxReplicas=12 Burst rate: 12 jobs in 60 seconds.

The burst saturated the pool: queue p50 jumped from five seconds at warmup to 153 seconds during the burst. After the burst window ended, the system continued draining the backlog through the cooldown phase. Cooldown queue p90 reached 181 seconds and p99 reached 305 seconds. The large class takes significant time to clear a backlogged queue after a sudden spike, even at a low ongoing trigger rate. All 28 jobs completed with zero errors.

Resource class: my-org/runner-small — 2 vCPU / 8 GiB, maxReplicas=48 Burst rate: 49 jobs in 60 seconds.

A burst of 49 jobs in 60 seconds exceeded the pool’s immediate capacity: queue p50 rose from 6 to 171 seconds. Despite the larger burst size compared to the large-class run, the small class recovered faster in cooldown — queue p90 dropped to 34 seconds and p99 to 128 seconds. The higher maxReplicas ceiling (48 vs. 12) allowed more VMs to run concurrently, draining the backlog faster. All 65 jobs completed with zero errors.

The burst mixed test randomly assigned jobs to either the large or small resource class. The burst rate was 20 jobs per minute for 60 seconds, producing 12 large-class and eight small-class jobs during the burst phase. Random assignment sent the single warmup job to the small class.

Large-class jobs in the mixed burst run:

Small-class jobs in the mixed burst run:

In the mixed run, large-class burst queue p90 reached 156 seconds — close to the 157 seconds in the large-only run. The pool received a similar number of large-class jobs during the burst window. Large-class cooldown recovered far faster in the mixed run: cooldown queue p90 dropped to 31 seconds. That is down from 181 seconds in the large-only run, where more large-class jobs remained in the queue at the burst’s end. Small-class jobs queued little at the burst peak (queue p90 = 63 seconds) despite competing for the same node group. All jobs completed with zero errors.

All three burst experiments completed with zero job errors. The large class shows its highest queue p90 during cooldown (181 seconds), not during the burst itself. A low ongoing arrival rate in cooldown limits throughput, so the backlog clears across the full cooldown window. The small class peaks during the burst (198 seconds at 49/min) and recovers in cooldown (queue p90 = 34 seconds), reflecting its larger replica pool.

The maxReplicas limit is the most direct lever for queue time. Once arrival rate exceeds what the running VM pool can turn over, the queue grows. Set maxReplicas based on your node group’s available memory divided by the VM size for that resource class, minus a buffer for OS and Kubernetes system pods. For example, with 96 GiB nodes and 32 GiB VMs, a single node can host floor(96 / 32) = 3 VMs. For a six-node group, the theoretical ceiling is 18 — subtract a buffer for system pods to arrive at a safe maxReplicas.

When multiple resource classes share a node group, each class competes for the same physical memory. The sum of maxReplicas × VM memory across all resource classes must not exceed the node group’s total allocatable memory. If one class has a generous maxReplicas, it may starve the other.

Routing traffic across resource classes of different sizes reduces queue times for the larger class. If your workload includes jobs with different resource requirements, define appropriately sized resource classes for each job type rather than routing everything through a single large class.

MRO supports a headroom configuration that provisions extra VMs beyond the current task count. Setting headroom ensures idle VMs are available when a burst of jobs arrives, reducing the delay from VM cold-start. Headroom is most effective for large resource classes where VM provisioning time adds noticeably to queue latency. The burst experiments show that after a sudden spike, the large class can take several minutes to drain the backlog — even at a low ongoing trigger rate. Headroom reduces this by keeping pre-warmed VMs available at the start of the burst.