When Old Workflows Meet New Code: Preventing Temporal Replay Failures in Production

[Akhil Madineni]

Temporal changes the meaning of deployment because a Workflow Execution can outlive the process and code revision that started it. The Temporal Service persists Event History, while Workers reconstruct Workflow state by replaying that history through Workflow code. A Worker released today may therefore execute against decisions recorded by an older revision. A change can compile, pass ordinary tests, and still become incompatible with existing history. Safe deployment requires treating Workflow code as a durable protocol whose command sequence must remain understandable across releases.

Replay Turns Code Into a Compatibility Contract​

During execution, SDK calls that schedule Activities, start timers, or perform other durable operations produce Commands. The Temporal Service converts those Commands into Events and appends them to Event History. During replay, Workflow code runs again and the SDK checks whether newly produced Commands agree with recorded history. Activities are not re-executed merely because replay occurs as recorded results reconstruct progress. This mechanism permits recovery after Worker failure, but it also makes deterministic orchestration a deployment constraint.

Determinism does not prohibit every source-code edit. Temporal documents several replay-compatible changes, including many changes to Activity inputs, return values, and timeouts, plus some timer-duration changes. The dangerous category is a change that alters Command-producing behavior. Adding, removing, or reordering an Activity or timer can make new code emit a different Command sequence from the one already represented in history. Changing an Activity or Child Workflow type or identifier can create the same incompatibility.

Consider an order Workflow whose original sequence reserves inventory and captures payment:

public String placeOrder(Order order) {
    activities.reserveInventory(order);
    return activities.capturePayment(order);
}

A later revision inserts fraud screening:

public String placeOrder(Order order) {
    activities.reserveInventory(order);
    activities.runFraudCheck(order);
    return activities.capturePayment(order);
}

For a new execution, the sequence is valid. For an execution whose history already records reserveInventory followed by capturePayment, replay can attempt to schedule runFraudCheck where history expects the payment command. Temporal documents this failure pattern for newly inserted Activities, code and Event History diverge, producing nondeterminism.

Patching Preserves Both Histories​

The Java SDK provides Workflow.getVersion for changes that must coexist with histories created by older code. The call records a version marker for new executions and returns a stable value when that execution is replayed. Existing histories that predate the patch can follow the old branch, while newer histories follow the new branch.

public String placeOrder(Order order) {
    activities.reserveInventory(order);

    int version = Workflow.getVersion(
        "fraud-check", Workflow.DEFAULT_VERSION, 1);

    if (version >= 1) {
        activities.runFraudCheck(order);
    }

    return activities.capturePayment(order);
}

The important property is that the decision becomes part of durable history. An old execution continues through the pre-change path, while a new execution records the marker and consistently takes the fraud-check path on later replays. Temporal recommends retaining compatibility code until executions created before the change can no longer require it. Java can also expose TemporalChangeVersion as a Search Attribute, which helps identify executions associated with version markers.

Patching is best kept around orchestration changes that affect Commands. HTTP calls, database operations, and other external interactions belong in Activities rather than Workflow code because those operations are nondeterministic and execute outside the Workflow replay path.

Worker Versioning Moves Compatibility Into Deployment​

Current Temporal guidance recommends Worker Versioning as the default approach when versioned Worker deployments are practical. A Worker Deployment groups related Workers, while a Worker Deployment Version combines a deployment name with a Build ID. Versioned Workers report that identity to Temporal, allowing Workflow Tasks to be routed according to deployment version.

The key behavior is pinning. A Workflow Type configured as PINNED remains on the Worker Deployment Version where its execution started. New releases can therefore contain incompatible orchestration changes without forcing already-running pinned executions to replay against new code. Java supports declaring this behavior on the Workflow implementation method.

@Override
@WorkflowVersioningBehavior(VersioningBehavior.PINNED)
public String placeOrder(Order order) {
    activities.reserveInventory(order);
    activities.runFraudCheck(order);
    return activities.capturePayment(order);
}

Worker configuration identifies the deployment and build:

WorkerDeploymentOptions deployment = WorkerDeploymentOptions.newBuilder()
    .setVersion(new WorkerDeploymentVersion("checkout", buildId))
    .setUseVersioning(true)
    .build();

Temporal also supports AUTO_UPGRADE. That mode allows an execution to move to newer Worker Deployment Versions as rollout state changes, so replay compatibility remains an application responsibility and patching is still required for incompatible Workflow changes. Pinned and Auto-Upgrade are therefore different lifecycle choices rather than equivalent safety modes.

Versioned rollout also supports a Ramping Version for a percentage of eligible Workflow traffic before promotion to Current. Older pinned versions enter a draining state after routing moves forward and can be retired after their pinned executions close. Temporal tracks draining and drained deployment-version states for this lifecycle.

Replay Testing Belongs in the Release Gate​

Worker Versioning reduces exposure to incompatible code, but replay testing remains important. Temporal recommends taking representative Event Histories from affected Workflow Types or Task Queues, replaying them with candidate code, and failing CI when replay reports an error. The Java SDK exposes WorkflowReplayer for this purpose.

javaCopyvoid verifyHistory(File history) throws Exception {
    WorkflowReplayer.replayWorkflowExecution(
        history, OrderWorkflowImpl.class);
}

A strong replay corpus represents real execution shapes rather than only a happy path. Histories containing retries, Signals, timers, failures, and different branches provide broader compatibility coverage because replay validates recorded Command sequences. Temporal’s safe-deployment guidance places replay testing during development, pre-deployment validation, and deployment-time verification, while noting that encrypted payloads and personally identifiable information require careful handling.

Long-Lived Workflows Need an Upgrade Policy​

Pinning preserves compatibility by preserving the code version, but indefinite pinning can preserve old Worker versions as well. Temporal exposes drainage because old versions may need to remain available while pinned executions stay open. For Workflows that use Continue-As-New, Temporal also provides an upgrade-on-Continue-As-New mechanism so a new run can move to a newer deployment version at a run boundary while each individual run remains pinned. The capability is currently documented as a Public Preview SDK-level option.

Shorter business processes can often let old pinned deployments drain naturally. Very long-lived entity-style Workflows need an explicit code-retirement strategy through Continue-As-New boundaries, carefully tested moves, or replay-compatible patching. Without such a policy, safe deployment can turn into permanent support for historical binaries.

Conclusion​

Temporal deployment safety depends on recognizing that Workflow history is durable even when application code changes. Replay makes the sequence of Command-producing decisions a compatibility contract across releases. Patching preserves that contract when old and new logic must coexist, while Worker Versioning can isolate pinned executions on the code revision that created them and provide controlled ramping and drainage. Replay testing verifies compatibility before production histories encounter a mistake. A safe Temporal release process therefore treats Workflow changes as protocol evolution as deterministic behavior remains deliberate, incompatible revisions are versioned explicitly, and historical executions stay executable for as long as persisted history requires.