Using Cypress Cloud CLI and Test Replay for AI-Assisted CI Debugging

[Raju Dandigam]

A checkout test fails in CI with this message:

expected '[data-cy="cart-total"]' to have text '$45.00'

The screenshot shows $50.00. An AI assistant can invent several plausible explanations: the discount request failed, React rendered stale state, the selector found the wrong cart, or the assertion ran early.

Plausible is not the same as supported.



The quality of AI-assisted debugging depends less on how confidently the assistant explains the screenshot and more on whether it can inspect the evidence that produced it.

Recorded CI and live local debugging are different surfaces​

Cypress now exposes two useful but distinct agent workflows:

• cypress tap: Evidence: A live, local open-mode Cypress session; Best use: Run and verify a change during development
• Cloud CLI: Evidence: Completed Cloud runs and Test Replay data; Best use: Reconstruct an existing CI failure

I have written separately about giving an agent access to a live Cypress session. This workflow starts at the opposite end: the failure already happened, perhaps only in CI, and the recorded attempt is the evidence.



The Cypress Cloud CLI returns JSON for runs, specs, tests, commands, network requests, and console events. The local Test Replay viewer is available through cy-cloud replay view in CLI version 0.4.0 or newer.

Start with a deliberately small evidence packet​

The current documentation requires Node.js 22.21.0 or newer and recommends using the latest CLI. Install it, record the resolved version, and pin that exact version in repeatable CI automation after validation:

npm install --global @cypress/cloud@latest
cy-cloud version
cy-cloud status

For a failed run, the shortest useful path is:

cy-cloud run list \
  --projectId <projectId> \
  --branch main \
  --status failed \
  --limit 1

cy-cloud test list \
  --projectId <projectId> \
  --runNumber <runNumber> \
  --status failed

cy-cloud replay timeline \
  --testId <testId> \
  --commands \
  --aroundFailure 5 \
  --network \
  --logs

That packet answers four questions:

1. Which command failed?
2. Which commands immediately preceded it?
3. Which XHR or fetch activity overlapped the failure?
4. Did the application log a related error?

Do not begin by dumping an entire organization's run history into an agent. Scope the query to one project, run, test, attempt, and timeline window.

A passing retry is the best comparison case​

Suppose the test failed on attempt one and passed on attempt two. A screenshot from the failed attempt tells us the symptom. Comparing the attempts can expose the first causal difference.

cy-cloud replay timeline \
  --testId <testId> \
  --attempt 1 \
  --commands --network --logs \
  > attempt-1.json

cy-cloud replay timeline \
  --testId <testId> \
  --attempt 2 \
  --commands --network --logs \
  > attempt-2.json

Imagine the relevant entries reduce to:

• POST /discounts/apply: Failed attempt: 200, response at 1,240 ms; Passing attempt: 200, response at 310 ms
• Assertion starts: Failed attempt: 820 ms; Passing attempt: 760 ms
• UI total: Failed attempt: $50.00; Passing attempt: $45.00
• Console: Failed attempt: none; Passing attempt: none

The evidence does not yet prove whether the product or test is wrong, but it eliminates several guesses. The assertion can run before the successful response. The next review question is whether the UI should block until the response arrives, or whether the test failed to wait for an application event that defines completion.

Pin the state before and after a suspect command​

The timeline gives structure. The local replay viewer lets you inspect the recorded DOM and styles around one command:

cy-cloud replay view open --testId <testId>

cy-cloud replay view move \
  --testId <testId> \
  --attempt 1 \
  --pin '<commandId>|before'

cy-cloud replay view inspect \
  --testId <testId> \
  --selector '[data-cy="cart-total"]'

cy-cloud replay view move \
  --testId <testId> \
  --attempt 1 \
  --pin '<commandId>|after'

This is more diagnostic than asking an agent to infer causality from the final screenshot. It can see whether an earlier command changed an attribute, collapsed a container, or left the DOM stale.



replay view requires a Chromium-family browser and Cloud CLI 0.4.0 or newer. Because Cloud CLI is Beta, scripts should pin a compatible version and treat output schemas as versioned dependencies.

Give the agent a diagnostic contract​

An unrestricted prompt such as “fix this test” encourages the assistant to optimize for a green run. It may add a hard wait, broaden a selector, remove an assertion, or mask a product regression.



A better task is bounded:

Analyze attempt-1.json and attempt-2.json for test <testId>.

1. Identify the first material difference before the failed assertion.
2. Cite the relevant command, network, or console event IDs.
3. Classify the likely cause as product, test synchronization,
   test data, environment, or unknown.
4. Propose the smallest code change, but do not edit files.
5. Do not add fixed waits or weaken the business assertion.
6. List the local verification commands and one negative control.

The request separates observation, diagnosis, proposal, and execution. A human can reject the classification before any code changes.

Credential locality is not the whole security model​

Cloud CLI access follows the organizations, teams, projects, and roles available to the authenticated Cypress Cloud user. OAuth is recommended locally; a personal access token can be supplied through CYPRESS_CLOUD_TOKEN in CI. Stored credentials are owner-readable on the local machine.



Those are useful controls, but they do not mean every AI environment is automatically safe. If an agent can read arbitrary environment variables, home-directory files, or shell output, it may reach more than intended.



Use the ordinary least-privilege questions:

• Which Cypress Cloud user or service identity is authenticating?
• Which projects can it read?
• Can the agent execute arbitrary shell commands?
• Where are downloaded replay data and screenshots cached?
• Can captured network bodies contain secrets or personal data?
• When are local artifacts deleted?

Cypress supports masking and redaction for Test Replay network data; configure those boundaries before making replay evidence broadly available.

Verification remains a separate step​

After reviewing a proposed fix, run the affected spec locally. A useful sequence is:

1. reproduce with the same fixture or a sanitized equivalent;
2. apply the smallest reviewed change;
3. run the original failure case;
4. run a negative control that should still fail;
5. inspect the behavior, not just the exit code;
6. push only after the evidence agrees.

The negative control matters. If changing the selector makes the test pass even when the discount is not applied, the “fix” destroyed the test.

Remove runner drift with a pinned container​

Before blaming the test or product, rerun the candidate change in the same declared Node and browser environment.



An official cypress/browsers or cypress/included image can make that boundary repeatable, but only when you pin a full version tag or digest:

export CYPRESS_IMAGE='cypress/browsers@sha256:<approved-digest>'
docker compose run --rm \
  -e CYPRESS_SPEC='cypress/e2e/checkout.cy.ts' \
  cypress

The Compose service should run the same repository revision and application fixture used in the investigation. Record docker image inspect "$CYPRESS_IMAGE", node --version, npx cypress version, and the selected browser with the evidence packet. A green result in a different image is not a reproduction; it is a new experiment.

What this workflow does not prove​

Recorded Test Replay evidence describes one captured run. It does not prove that a failure is statistically representative or that the proposed patch is correct. A passing retry can differ because of data, timing, infrastructure, or an actual product race.

Cloud CLI is also not an autonomous remediation system. It is a read-oriented evidence interface. Code changes, approval, and verification remain responsibilities of your development workflow.

The reusable rule is:

Ask AI to explain bounded evidence before asking it to change code.

That turns the assistant from a confident guesser into a supervised participant in a debugging process whose claims can be reviewed