How We Run 250,000 Automated Tests a Day on Kubernetes

[lokeshsuri]

Key Takeaways​

• Centralizing test execution only pays off when it is run as a product with admission control, quotas, and priorities — otherwise it turns into one shared queue where the noisiest team sets everyone's wait time.
• Kubernetes which has indexed jobs paired with a quota aware queue like Kueue present a clean and standard model for sharding test suites and determining which work to put forward when we have less capacity.
• Flakiness is a platform concern, not a per-team annoyance: at a quarter-million executions a day, even a low flake rate yields thousands of misleading signals unless retries, classification, and quarantine are policy.
• We see value in spreading out execution across many Kubernetes clusters which in turn reduces blast radius and control plane pressure, but it also shifts the complex issues into routing, capacity planning and consistent config.
• The report layer which takes raw execution data and turns it into information that we can trust release certification, flake trends, capacity planning etc. that is what this layer does and it requires the same level of design rigor as the scheduler.

Executive Summary​

Duplicated runners, inconsistent environments, and also no shared view of quality. This article describes the architecture of a centralized test orchestration and certification platform that executes roughly 250,000 automated tests per day for every engineering repository in a large financial-services organization, running on four Amazon EKS clusters. We cover admission, sharding, prioritization and retries; flakiness as a platform policy; the reporting and insights layer; and we look at the tradeoffs of a shared, multi tenant exec plane a reusable model which we also present with the pros and cons of where it does and does not work.

Introduction​

Test automation is a framework issue which we discuss. As thousands of repos turn to auto-run suites the complex issues shift to infrastructure. At 9 a.m., who is going to get the capacity when all teams are pushing? And what do we do when a nightly regression suite is in play at the same time as a pre-merge check which is blocking a developer? How does anyone decide whether a release is certified when a meaningful share of failures are not real?

Google's published experience is instructive: In their continuous testing framework, what we see is that resources, instead of test authoring became the issue, and also very few tests fail [4]. Enterprises hit the same shape of problem much earlier, because their capacity is smaller and their suites less curated.

The platform discussed here grew from that pressure. It is the shared execution infrastructure that all engineering repositories in the organization schedule against, and it serves as a certification engine: a release is considered ready when its required suites pass on the platform, not on a developer's machine or a team's private runner.

Background​

Three Kubernetes primitives carry most of the design. A Job runs pods to completion, and Indexed completion mode gives each pod a stable index, which maps naturally onto test shards [6]. PriorityClasses let the scheduler preempt lower-priority pods when higher-priority pods cannot be placed [8]. ResourceQuotas and LimitRanges cap what a namespace can consume and set default requests, preventing unbounded containers on a shared cluster [9], [10].

Kueue, a SIG Scheduling project, adds queue-level quotas and decides when a job waits, is admitted, or is preempted [7]; an autoscaler such as Karpenter adds nodes for unschedulable pods [11], [12].

Problem Analysis​

The workload has characteristics that break naive designs:

• Extreme burstiness. Demand goes up during work hours and at release drops; we see either that we have over capacity or not enough.
• Mixed urgency. Pre merge checks which tie up an engineer and also the nightly full regression have different latency requirements but use the same resources.
• Heterogeneous tenants. Different teams bring in different runtimes, browser stacks and data dependencies; one suite without resource constraints can starve out the others.
• Untrustworthy signals. We see from empirical studies that flaky tests are common which come from asynchronous waits, concurrency, and test order dependencies [1], [2].Google reported that about 1.5% of all test runs produced a flaky result and almost 16% of tests showed some flakiness [3]. Applied illustratively to 250,000 daily executions, a 1.5% rate would mean roughly 3,750 misleading results every day.
• Audit expectations. In a regulated environment, "this release was certified" must be reproducible from records, not memory.

Architecture and Design​

The platform separates a control plane that decides what runs from an execution plane that runs it, with a results plane that turns executions into decisions (Figure 1).

Submission. CI systems, we have set up that teams use to trigger a run which includes the suite, target environment, and a classification for urgency. At no time do teams interface with Kubernetes; we have defined the API as the contract which means that we may change the execution details that teams are not a part.

Admission and queueing. Per tenant, we have a namespace which also has a quota and a local queue.Kueue holds workloads until quota is available and applies preemption and fair sharing across a cohort, so idle capacity borrowed by one team can be reclaimed when another needs its share [7].

Shard planning. Large suites are split into shards executed as an Indexed Job. The shard index is exposed to the runner, following the same contract Bazel uses with TEST_TOTAL_SHARDS and TEST_SHARD_INDEX [5].

Execution plane. Four EKS clusters host the platform's 40 applications and the ephemeral test pods. Multiple clusters trade higher control-plane and tooling cost for isolation and smaller blast radius, a trade-off AWS documents explicitly for multi-tenant designs [13]. Placement needs a deliberate policy. A we can use is to present each tenant with a home cluster that has image caches and test data which are kept warm, also at the same time we will have the control plane spill over to another cluster as a whole run not just separate shards when the home cluster’s queue is over a certain wait time. Breaking up one run between clusters also causes issues in terms of results collection and also in the identification of what went wrong with the infrastructure.

Results and certification. Each shard sends a formatted result message to a reporting and insight service that determines a pass/fail status for each test suite, identifies flakes, and registers a certification outcome on the release.The event schema is the most important contract in the platform:

{
  "runId": "checkout-regression-7f3a",
  "shardIndex": 17,
  "attempt": 2,
  "test": "CheckoutFlowTest.appliesPromoCode",
  "outcome": "passed",
  "failureCause": null,
  "previousAttemptCause": "environment",
  "node": "ip-10-0-42-7",
  "durationMs": 48210
}

failureCause (test, environment, or platform) and attempt are what allow the results layer to tell a flaky test from a lost node, and to charge infrastructure failures to the platform rather than to the team.

Implementation​

A tenant queue and its quota are declared once by the platform team:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata: { name: team-payments }
spec:
  cohort: shared-test-capacity
  preemption:
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority
  resourceGroups:
  - coveredResources: ["cpu", "memory"]
    flavors:
    - name: default
      resources:
      - { name: cpu,    nominalQuota: 200, borrowingLimit: 100 }
      - { name: memory, nominalQuota: 800Gi }

The cohort lets teams borrow idle capacity, while reclaimWithinCohort lets the owner take it back. The borrowingLimit stops any single tenant from absorbing the whole cohort during a release crunch.

A sharded suite is then an ordinary Indexed Job with a priority class:

apiVersion: batch/v1
kind: Job
metadata:
  name: checkout-regression-7f3a
  labels: { kueue.x-k8s.io/queue-name: team-payments }
spec:
  completionMode: Indexed
  completions: 40
  parallelism: 40
  backoffLimitPerIndex: 1
  template:
    spec:
      priorityClassName: premerge-high
      restartPolicy: Never
      containers:
      - name: runner
        image: registry.internal/test-runner:2.14
        env:
        - { name: TEST_TOTAL_SHARDS, value: "40" }
        - name: TEST_SHARD_INDEX
          valueFrom:
            fieldRef:
              fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']
        resources:
          requests: { cpu: "2", memory: 4Gi }
          limits:   { memory: 4Gi }

backoffLimitPerIndex retries a failed shard once without rerunning the other 39, which matters when one shard hits an infrastructure fault. Memory limits are set because an out-of-memory runner on a shared node degrades its neighbours; CPU is left unbounded above the request to absorb short bursts.

Shard balancing should be driven by historical duration, not test count. The planner groups tests so each shard's expected runtime is similar, because a suite finishes only when its slowest shard does.

Performance and Scalability​

Run latency is dominated by three waits, each with its own lever:

Wait	Lever	Cost
Queueing	Larger nominal quota, cohort borrowing	Idle reserved capacity
Node provisioning	Warm buffer nodes, pre-pulled runner images	Paying for idle nodes
Slowest shard	Duration-based sharding, more shards	Per-pod startup overhead

Karpenter is a good fit for spiky, diverse demand because it provisions nodes directly from pending pod requirements [12]; Cluster Autoscaler remains viable with well-designed node groups [11]. Either way, keep runner images small and cached — at this volume, image pull time is paid tens of thousands of times a day.

Security Considerations​

Test code is semi-trusted code: it runs whatever a repository commits. As much as possible, pod implementations should be conducted while respecting the Restricted Pod Security Standard set in [18], while ensuring RBAC is enforced via the use of namespaces so that service account employed by a team will not read any work orders issued by another team [16].Cloud access should use IAM Roles for Service Accounts or EKS Pod Identity rather than long-lived keys [14], [15]. Test credentials belong in a secret store with least-privilege access, following Kubernetes secrets guidance [17]. Runner images should be scanned and pinned, consistent with NIST's container security guidance [19]. In a regulated setting, test environments should also be kept out of production data which is to do via network policy as opposed to what we have been doing.

Operational Excellence​

Instrument the platform as a whole not just the tests: trace each run from submission through admission, scheduling and result ingestion with OpenTelemetry [20] for example why was my run slow?" has a per-phase answer. The core service-level indicators are time-to-admission, time-to-first-result, infrastructure-failure rate (distinct from test-failure rate), and flake rate per suite. Delivery outcomes can be tracked with DORA metrics to show whether the platform actually shortens lead time and reduces change-failure rate [21].

A Failure Scenario: The Release-Morning Crunch​

Consider an illustrative but common situation. Three teams deploy full regression suites before the release cutoff which at the same time see dozens of other engineers perform pre-merge checks. Without quotas in place, the regression suites that go in first will fill all the available nodes, leaving pre-merge checks to queue up and in turn stall development across the org. With the design above, each regression suite runs within its team's nominal quota plus a bounded borrowing limit; pre-merge pods carry a higher priority class and preempt nightly-class work when nodes are short [8]; and the autoscaler adds capacity for the remaining backlog. The regression suites finish their runs after they would on an empty cluster, but no one team can put the brakes on the whole process. That is the trade-off which a shared platform puts out there.

Trade-offs and Limitations​

Centralization creates a single team that every other team depends on. Its roadmap becomes a bottleneck, and a platform outage stops every release at once. Multi-cluster execution reduces blast radius but multiplies upgrades, add-on versions, and configuration drift. Somebody has to determine each team’s share which is an organizational issue that current schedulers don’t address. Also automatic retries mask what are, in fact, intermittent issues; race conditions in production code play out the same as flaky tests.

Why don’t we use what CI systems have for parallelism? Per pipeline runners which do one repo at a time; they don’t do org wide fair play, don’t do consistent attribution of infra failures and don’t produce a single certification report. A central platform is for when these issues are what you care most about, not team autonomy. It is the wrong play for a small shop with few repos, and for jobs which need special hardware a shared pool can’t practically keep warm.

Production Lessons Learned​

Lesson: Separate infrastructure failures from test failures at the source. If a node eviction and an assertion failure produce the same red result, teams lose trust in the platform. Runners should report failure cause — test, environment, or platform — as a first-class field.

Lesson: Retry once, on a fresh pod, and record it. A single retry on different infrastructure separates environment noise from deterministic failures. Silent, repeated retries inflate cost and bury genuine concurrency defects, which research identifies as a leading root cause of flakiness [1].

Lesson: Quarantine is a workflow, not a flag. A quarantined test needs an owner, a ticket, and an expiry; otherwise the quarantine list grows until certification means little.

Lesson: The reporting layer is the product. Teams judge the platform by the results view, not the scheduler. The reporting and insights service deserves a dedicated engineering team rather than being bolted on, because it is the part of a shared platform every repository's engineers see every day.

Lesson: Onboarding is part of the platform. When every repository depends on one execution platform, misuse — unbounded suites, missing resource requests, retry loops in test code — costs everyone. Monthly continuous-delivery onboarding sessions for 200+ new hires are, in effect, a platform control as important as any quota.

Best Practices​

• Expose a stable submission API; never let tenants depend on cluster internals.
• Give every tenant a quota, a queue, and default resource limits from day one.
• Shard by historical duration; retry per shard, never per suite.
• Publish flake rate per suite and trace each run end to end.

Future Outlook​

Two directions look durable. Change-based test selection is the largest remaining cost lever, since Google's data shows most tests rarely fail [4]. AI-assisted triage can group failures by probable cause, but it should assist owners rather than auto-quarantine tests, because misclassification directly weakens certification. Treating such capabilities as self-service internal products, as the CNCF platforms white paper describes, is becoming the default funding and governance model [22].

Conclusion​

A centralized test orchestration platform does more than schedule work. It also separates tenants securely and provides the authoritative record of quality. Kubernetes offers useful primitives—Indexed Jobs, priorities, quotas, and Kueue—but policy determines their value: which teams receive capacity, how failures are categorized, and what qualifies as “certified.” Build the queue, failure taxonomy, and reporting layer as one system; none can be trusted if the other two are unreliable.

References​

[1] Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, "An empirical analysis of flaky tests," in Proc. 22nd ACM SIGSOFT Int. Symp. Foundations of Software Engineering (FSE), Hong Kong, China, 2014, pp. 643–653, doi: 10.1145/2635868.2635920. [Online]. Available: https://doi.org/10.1145/2635868.2635920

[2] O. Parry, G. M. Kapfhammer, M. Hilton, and P. McMinn, "A survey of flaky tests," ACM Trans. Softw. Eng. Methodol., vol. 31, no. 1, Art. no. 17, pp. 1–74, Jan. 2022, doi: 10.1145/3476105. [Online]. Available: https://doi.org/10.1145/3476105

[3] J. Micco, "Flaky tests at Google and how we mitigate them," Google Testing Blog, May 27, 2016. [Online]. Available: https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html. Accessed: Oct. 5, 2026.

[4] A. Memon et al., "Taming Google-scale continuous testing," in Proc. IEEE/ACM 39th Int. Conf. Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), Buenos Aires, Argentina, 2017, pp. 233–242, doi: 10.1109/ICSE-SEIP.2017.16. [Online]. Available: https://research.google/pubs/taming-google-scale-continuous-testing/

[5] The Bazel Authors, "Test encyclopedia," Bazel Documentation. [Online]. Available: https://bazel.build/reference/test-encyclopedia. Accessed: Oct. 5, 2026.

[6] The Kubernetes Authors, "Jobs," Kubernetes Documentation. [Online]. Available: https://kubernetes.io/docs/concepts/workloads/controllers/job/. Accessed: Oct. 5, 2026.

[7] Kubernetes SIG Scheduling, "Overview," Kueue Documentation. [Online]. Available: https://kueue.sigs.k8s.io/docs/overview/. Accessed: Oct. 5, 2026.

[8] The Kubernetes Authors, "Pod priority and preemption," Kubernetes Documentation. [Online]. Available: https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/. Accessed: Oct. 5, 2026.

[9] The Kubernetes Authors, "Resource quotas," Kubernetes Documentation. [Online]. Available: https://kubernetes.io/docs/concepts/policy/resource-quotas/. Accessed: Oct. 5, 2026.

[10] The Kubernetes Authors, "Limit ranges," Kubernetes Documentation. [Online]. Available: https://kubernetes.io/docs/concepts/policy/limit-range/. Accessed: Oct. 5, 2026.

[11] Amazon Web Services, "Cluster Autoscaler," Amazon EKS Best Practices Guide. [Online]. Available: https://docs.aws.amazon.com/eks/latest/best-practices/cas.html. Accessed: Oct. 5, 2026.

[12] Amazon Web Services, "Karpenter," Amazon EKS Best Practices Guide. [Online]. Available: https://docs.aws.amazon.com/eks/latest/best-practices/karpenter.html. Accessed: Oct. 5, 2026.

[13] Amazon Web Services, "Tenant isolation," Amazon EKS Best Practices Guide. [Online]. Available: https://docs.aws.amazon.com/eks/latest/best-practices/tenant-isolation.html. Accessed: Oct. 5, 2026.

[14] Amazon Web Services, "IAM roles for service accounts," Amazon EKS User Guide. [Online]. Available: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html. Accessed: Oct. 5, 2026.

[15] Amazon Web Services, "Learn how EKS Pod Identity grants pods access to AWS services," Amazon EKS User Guide. [Online]. Available: https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html. Accessed: Oct. 5, 2026.

[16] The Kubernetes Authors, "Using RBAC authorization," Kubernetes Documentation. [Online]. Available: https://kubernetes.io/docs/reference/access-authn-authz/rbac/. Accessed: Oct. 5, 2026.

[17] The Kubernetes Authors, "Good practices for Kubernetes Secrets," Kubernetes Documentation. [Online]. Available: https://kubernetes.io/docs/concepts/security/secrets-good-practices/. Accessed: Oct. 5, 2026.

[18] The Kubernetes Authors, "Pod Security Standards," Kubernetes Documentation. [Online]. Available: https://kubernetes.io/docs/concepts/security/pod-security-standards/. Accessed: Oct. 5, 2026.

[19] M. Souppaya, J. Morello, and K. Scarfone, "Application container security guide," NIST, Gaithersburg, MD, USA, NIST SP 800-190, Sep. 2017, doi: 10.6028/NIST.SP.800-190. [Online]. Available: https://csrc.nist.gov/pubs/sp/800/190/final

[20] OpenTelemetry Authors, "What is OpenTelemetry?," OpenTelemetry Documentation. [Online]. Available: https://opentelemetry.io/docs/what-is-opentelemetry/. Accessed: Oct. 5, 2026.

[21] N. Harvey, "DORA's software delivery performance metrics," DORA, 2026. [Online]. Available: https://dora.dev/guides/dora-metrics-four-keys/. Accessed: Oct. 5, 2026.

[22] CNCF TAG App Delivery Platforms Working Group, "CNCF platforms white paper," Cloud Native Computing Foundation, v1.0, Apr. 2023. [Online]. Available: https://tag-app-delivery.cncf.io/whitepapers/platforms/. Accessed: Oct. 5, 2026.

About the Author​

Lokesh Suri is a Senior VP and engineering lead at a global financial services company which he has been with for over 20 years during which he built enterprise systems in Java and Python in investment banking, retail banking, anti money laundering, and regulatory risk. He is the lead behind over 40 applications which include a central test orchestration platform that is on Amazon EKS and also heads a team which is into report and insight development. Also he has obtained AWS Cloud Practitioner and AWS Architect certifications.