Orchestrations that never complete because replay no longer matches history
Azure Service Bus
PostgreSQL
DockerThe stalled billing orchestrations were cleared without duplicate charges, and orchestrator versioning means a deployment no longer risks wedging every instance already running.
Overview
Durable Functions orchestrators work by replaying their code against an event history. That only works if the orchestrator is deterministic — no direct clock reads, no random values, no I/O, no unordered async work outside the durable APIs. Break determinism and the replay diverges from history, and the orchestration wedges rather than fails cleanly.
Challenge
A batch of billing orchestrations stopped progressing after a deployment. The orchestrator called the system clock directly to compute a cutoff, generated a correlation GUID inline, and made an HTTP call from the orchestrator rather than from an activity. Existing in-flight instances had histories that the new code could not reproduce, so they stalled — and because they were not marked failed, no failure alert fired.
Environment
Durable Functions on Azure with fan-out/fan-in billing orchestrations spanning multiple activity functions and external calls. AceMQ works on the customer's orchestrator and activity code; Microsoft operates the platform.
Approach
AceMQ inspected the instance histories of stuck orchestrations to find where replay diverged, which pointed directly at the non-deterministic calls. We corrected the orchestrator, then handled the existing in-flight instances deliberately — some could be resumed, some had to be terminated and re-run from a known-safe point.
Solution
- 1Inspected instance event histories to identify exactly where replay diverged from recorded history
- 2Replaced direct clock reads with the durable context's deterministic current time
- 3Moved GUID generation and all I/O out of the orchestrator and into activity functions
- 4Triaged in-flight instances into resumable and terminate-and-replay groups with idempotent re-run
- 5Adopted versioned orchestrators so future changes do not break instances already in flight
- 6Added alerting on orchestration age and stuck-instance count, not just on failures
Outcome
The stalled billing orchestrations were cleared without duplicate charges, and orchestrator versioning means a deployment no longer risks wedging every instance already running.
Technologies
Azure Service BusPostgreSQLDockerRelated Use Cases
Azure Functions Storage Account Trigger Failure Remediation
Diagnosing functions that silently stopped triggering after a storage account connectivity change — a dependency the runtime has but the application code never mentions.
Azure Functions Event Architecture Consulting
Designing trigger selection, concurrency control, and failure handling for an Azure Functions estate integrating messaging and event streams.
Need Expert Azure Functions Support?
AceMQ's senior Azure Functions engineers have handled this exact type of engagement before. Whether you need architectural guidance, hands-on remediation, or an ongoing managed partnership, we're ready to help.