How to Shadow-Test an AI Agent Without Letting It Act Twice

[Raju Dandigam]

You have a new prompt, model, or orchestration policy. Offline tests look good, but the change will see vocabulary, timing, and context combinations that your fixture set never captured.



So you shadow-test it: send production traffic to both the current agent and the candidate, compare their decisions, and keep the current agent in control.



Then the candidate calls send_email too.



Now one user request has produced two real messages. The “safe” experiment has become a duplicate-action incident.

This synthetic failure happens when shadowing is implemented as “run the whole application twice.” For agents that can change the world, shadow testing must duplicate observation and decision—not authority and effects.

The Architecture Boundary That Makes Shadowing Safe​

An action-taking agent contains at least four separable stages:

input -> decision proposal -> policy/approval -> tool execution -> outcome

The shadow should stop after the proposal and policy simulation:

                         +--> primary decision --> real policy --> execute
sanitized input/context -|
                         +--> shadow decision  --> dry policy  --> record only

This boundary is useful beyond experiments. It makes approvals testable, prevents a model response from directly becoming an external write, and gives you structured evidence to compare.

Represent Intent Before Execution​

Instead of letting the model own an executable tool object, normalize its choice into a proposal:

type ActionProposal = {
  action: string;
  schemaVersion: number;
  args: Record<string, unknown>;
  reasonCode: string;
  confidence?: {
    value: number;
    source: "model" | "calibrator";
  };
};

type PolicyDecision = {
  result: "allow" | "deny" | "require_approval";
  policyVersion: string;
  reasonCodes: string[];
};

Do not collect hidden chain-of-thought. A bounded reason code such as invoice_overdue or user_requested_change is reviewable. If you store a short explanation, treat it as untrusted, redact it, and never require private reasoning from a model.



The executor accepts only a proposal that has passed real policy:

async function executeAuthorized(
  proposal: ActionProposal,
  decision: PolicyDecision,
  context: ExecutionContext,
) {
  if (context.mode !== "primary") {
    throw new Error("Effects are disabled outside primary mode");
  }
  if (decision.result !== "allow") {
    throw new Error(`Action is not authorized: ${decision.result}`);
  }

  return toolRegistry.execute(proposal, context);
}

That mode check is defense in depth. The stronger control is that the shadow environment never receives write-capable credentials.

Do Not Give the Shadow Real Tools​

A dryRun: true flag passed into a real tool is fragile. One tool may ignore it, a refactor may omit it, or a newly registered tool may default to execution.



Build a distinct registry:

type ToolDescriptor = {
  name: string;
  inputSchema: object;
  sideEffect: "read" | "write";
};

const shadowTools = realTools.map(({ name, inputSchema, sideEffect }) => ({
  name,
  inputSchema,
  sideEffect,
} satisfies ToolDescriptor));

The candidate sees the same tool names and schemas needed to make a decision, but the objects have no execute method. If it proposes send_email, the harness records the arguments and predicted policy result.



At the infrastructure layer, use separate identities and network policy:

• no production write tokens;
• no direct access to message brokers used for effects;
• read-only or synthetic dependencies where possible;
• a separate telemetry namespace;
• hard spend and concurrency budgets; and
• a kill switch independent of model behavior.

Application checks can fail. Missing credentials fail closed.

Decide What Context the Shadow Is Allowed to See​

Copying an entire production prompt into an experiment can expose personal, contractual, or security-sensitive data. Define a context projection:

type ShadowContext = {
  requestClass: string;
  locale: string;
  accountTier: string;
  recentEventCodes: string[];
  policyFacts: Record<string, boolean | number | string>;
};

function toShadowContext(input: LiveContext): ShadowContext {
  return {
    requestClass: input.requestClass,
    locale: input.locale,
    accountTier: input.accountTier,
    recentEventCodes: input.events.map(event => event.code).slice(-10),
    policyFacts: selectApprovedPolicyFacts(input),
  };
}

Run this projection through privacy and threat review. The candidate often needs less context than the primary runtime accumulated.

Compare Decisions, Not Sentences​

Two good agents can explain the same decision differently. Text similarity therefore creates noisy promotion gates.

Compare structured behavior:

type DecisionDiff = {
  sameAction: boolean;
  changedArgumentPaths: string[];
  primaryPolicy: PolicyDecision["result"];
  shadowPolicy: PolicyDecision["result"];
  riskClass: "equivalent" | "benign_change" | "review" | "unsafe";
};

The highest-signal mismatches are usually:

• primary says “stay silent,” candidate proposes a write;
• primary requests approval, candidate predicts allow;
• both choose the same tool but differ on amount, recipient, or scope;
• candidate calls a forbidden or nonexistent tool;
• candidate exceeds latency, token, or tool-call budgets; or
• candidate makes a user-facing claim unsupported by the proposed evidence.

Google's Agents CLI evaluation guide reflects this broader view of agent quality: it separates tool-use quality, trajectory quality, task success, final-response quality, hallucination, grounding, and safety. A shadow score should likewise be a vector, not a single “similarity” percentage.

Use Policy Simulation Carefully​

The shadow can run the current policy engine in a non-authoritative mode. That reveals whether the candidate's proposal would be allowed.

const simulated = await policy.evaluate({
  proposal: shadowProposal,
  principal: redactedPrincipalFacts,
  mode: "simulate",
  policyVersion: productionPolicyVersion,
});

Simulation must not mint approval tokens, reserve inventory, mutate rate-limit counters, or write audit decisions into the authoritative ledger. If the policy system cannot evaluate without side effects, build a pure policy core and keep recording outside it.

Also record the policy version. A candidate comparison is uninterpretable if primary and shadow were evaluated under different rules without saying so.

Avoid Selection Bias​

It is easy to shadow only the simple requests that survive a convenient filter. The resulting dashboard says the candidate is safe because risky traffic never entered the experiment.



Define inclusion explicitly:

type ShadowEligibility = {
  eligible: boolean;
  reason:
    | "selected"
    | "privacy_excluded"
    | "unsupported_tool"
    | "budget_excluded"
    | "control_group";
};

Report both the evaluated population and exclusions. Stratify results by action type, risk level, language, account segment, workflow length, and whether the primary required recovery. Otherwise a large aggregate can hide the exact slice that matters.

Nondeterminism Needs Repeated Comparisons​

One primary output versus one candidate output does not tell you the candidate's variance. For a bounded sample, run multiple candidate trials with fixed configuration and preserve the random seed where the provider supports it.

Do not mistake a seed for full determinism. Provider updates, retrieval changes, timestamps, tool data, and concurrency can still change behavior.



Useful summaries include:

action agreement distribution
forbidden-action proposal rate
approval-bypass proposal rate
argument-diff severity
p50/p95 decision latency
token and cost distribution
variance across repeated candidate runs
coverage and exclusion rate

Never invent thresholds because they sound mature. Set them from product risk and baseline behavior, then document the decision.

A Safer Promotion Ladder​

Shadowing is one stage, not a release strategy by itself.

1. Static checks: schemas, tool allowlists, secret scanning, policy tests.
2. Offline replay: sanitized, versioned fixtures with known expected behavior.
3. Repeated model evaluation: measure variance and grader stability.
4. Production shadow: real context projection, no effect authority.
5. Internal or consented canary: narrow population, real effects, rollback controls.
6. Progressive rollout: risk-aware expansion with outcome monitoring.

Promotion should consider downstream outcomes, not only proposal agreement. The current agent may be consistently wrong. A candidate that disagrees could be an improvement, so route meaningful disagreements to labeled review rather than automatically treating the primary as truth.

What a Shadow Record Should Contain​

A reviewable record needs enough context to reproduce the comparison without becoming a privacy archive:

type ShadowRecord = {
  experimentId: string;
  requestFingerprint: string;
  cohort: string;
  primary: { proposal: ActionProposal; policy: PolicyDecision };
  candidate: { proposal: ActionProposal; policy: PolicyDecision };
  diff: DecisionDiff;
  primaryVersion: string;
  candidateVersion: string;
  policyVersion: string;
  evaluatedAt: string;
};

Store raw prompts and outputs only when authorized and necessary. Prefer schema-approved fields, stable fingerprints, and short retention. Observability should reduce uncertainty without expanding the blast radius of sensitive data.

The Non-Negotiable Invariant​

I would encode this as an automated contract:

For every shadow run:
  external_write_count == 0
  authoritative_approval_count == 0
  production_message_publish_count == 0

If the telemetry needed to verify those invariants is missing, the experiment should fail closed or pause. “We did not see a write” is weaker than “the shadow identity could not write.”



The point of shadow testing is to learn from realistic inputs before granting realistic authority. Duplicate the decision surface. Keep the effect surface singular.

References​

• Google Agents CLI: Evaluating Agents
• OpenTelemetry: Context Propagation