Overview
Durable Functions orchestrators work by replaying their code against an event history. That only works if the orchestrator is deterministic — no direct clock reads, no random values, no I/O, no unordered async work outside the durable APIs. Break determinism and the replay diverges from history, and the orchestration wedges rather than fails cleanly.
Challenge
A batch of billing orchestrations stopped progressing after a deployment. The orchestrator called the system clock directly to compute a cutoff, generated a correlation GUID inline, and made an HTTP call from the orchestrator rather than from an activity. Existing in-flight instances had histories that the new code could not reproduce, so they stalled — and because they were not marked failed, no failure alert fired.
Environment
Durable Functions on Azure with fan-out/fan-in billing orchestrations spanning multiple activity functions and external calls. AceMQ works on the customer's orchestrator and activity code; Microsoft operates the platform.
Approach
AceMQ inspected the instance histories of stuck orchestrations to find where replay diverged, which pointed directly at the non-deterministic calls. We corrected the orchestrator, then handled the existing in-flight instances deliberately — some could be resumed, some had to be terminated and re-run from a known-safe point.
Solution
- Inspected instance event histories to identify exactly where replay diverged from recorded history
- Replaced direct clock reads with the durable context's deterministic current time
- Moved GUID generation and all I/O out of the orchestrator and into activity functions
- Triaged in-flight instances into resumable and terminate-and-replay groups with idempotent re-run
- Adopted versioned orchestrators so future changes do not break instances already in flight
- Added alerting on orchestration age and stuck-instance count, not just on failures
Outcome
The stalled billing orchestrations were cleared without duplicate charges, and orchestrator versioning means a deployment no longer risks wedging every instance already running.
Technologies
Azure FunctionsAzure Service Bus
PostgreSQL
Docker
RabbitMQ
Kafka
Kubernetes
Redis
Airbyte
