Orchestrations that never complete because replay no longer matches history

Overview

Durable Functions orchestrators work by replaying their code against an event history. That only works if the orchestrator is deterministic — no direct clock reads, no random values, no I/O, no unordered async work outside the durable APIs. Break determinism and the replay diverges from history, and the orchestration wedges rather than fails cleanly.

Challenge

A batch of billing orchestrations stopped progressing after a deployment. The orchestrator called the system clock directly to compute a cutoff, generated a correlation GUID inline, and made an HTTP call from the orchestrator rather than from an activity. Existing in-flight instances had histories that the new code could not reproduce, so they stalled — and because they were not marked failed, no failure alert fired.

Environment

Durable Functions on Azure with fan-out/fan-in billing orchestrations spanning multiple activity functions and external calls. AceMQ works on the customer's orchestrator and activity code; Microsoft operates the platform.

Approach

AceMQ inspected the instance histories of stuck orchestrations to find where replay diverged, which pointed directly at the non-deterministic calls. We corrected the orchestrator, then handled the existing in-flight instances deliberately — some could be resumed, some had to be terminated and re-run from a known-safe point.

Solution

• Inspected instance event histories to identify exactly where replay diverged from recorded history
• Replaced direct clock reads with the durable context's deterministic current time
• Moved GUID generation and all I/O out of the orchestrator and into activity functions
• Triaged in-flight instances into resumable and terminate-and-replay groups with idempotent re-run
• Adopted versioned orchestrators so future changes do not break instances already in flight
• Added alerting on orchestration age and stuck-instance count, not just on failures

Outcome

Technologies