BlogEngineering

What is durable execution, and what does it demand of your code?

Durable execution lets long-running code survive crashes by replaying it. It only works if your code is deterministic. Here is what that demands.

Isha MagguIsha Maggu11 min read

Summarize with AI

What is durable execution, and what does it demand of your code?
On this page

Durable execution is a way to write a long-running process as ordinary sequential code (call a service, wait a week, call another) that survives process restarts, deployments and crashes. The program appears to run for a week, but the process hosting it does not. Its demand on your code is determinism: replaying the function must make the same decisions every time.

The mechanism is replay. When execution resumes, the engine runs your function again from the beginning and feeds back the recorded results of everything that already happened instead of repeating it. Replay is elegant, and it limits how you can write the code in ways that are easy to violate by accident.

What we'll cover

Key takeaways

  • Write workflow code as deterministic orchestration, and put every network call, database write and random or time-dependent value in an activity.
  • Make every activity idempotent, because a timed-out call is retried and the engine cannot tell whether the first attempt succeeded.
  • Plan for versioning from the start, because running instances are replayed against whatever code is deployed.
  • Keep a production history as a test fixture, because replaying it against a change catches determinism breaks before deployment.
  • Use durable execution for processes that wait days or must not be lost, not for queues, data pipelines or low-latency requests.

The illusion and how it works

Your code reads sequentially. You can call a service, wait for an approval, wait thirty days and call another service, all written as one function instead of a state machine spread across handlers.

Every external interaction is recorded in an event history: what was called, what came back.

On resume, the function runs again from the top. Calls that already happened return their recorded results immediately, without contacting anything, until execution reaches the point it had reached before.

Then it continues normally. From the code's perspective nothing happened; from the infrastructure's perspective the process died and a new one picked up.

Waiting costs almost nothing. A workflow paused for thirty days holds no thread, no connection and no memory. It is a row in storage.

Determinism, the constraint

The whole model depends on replay producing the same decisions, which means your workflow code must be deterministic.

No direct clock reads. Calling the system clock gives a different answer on replay, which can send execution down a different branch than the original run took. Use the engine's time functions, which are recorded.

No random values. Same reason.

No direct network or database calls from workflow code. Every external interaction goes through the recorded mechanism.

No unordered iteration. Iterating a structure with non-deterministic ordering can produce different sequences across replays.

No reliance on process state, such as global variables, caches or environment differences between the original run and the replay.

Violating this rule produces a nondeterminism error, or worse, silently different behaviour. Determinism is the largest source of confusion for teams adopting durable execution, and the rule makes sense only once the replay model is clear.

What goes in an activity

Activities are the escape hatch from the determinism rules, and understanding them resolves most of the confusion above.

Activities are where the real work happens: network calls, database writes and anything with a side effect or a nondeterministic result.

Workflow code calls activities and the engine records their results, which is what makes replay safe.

The engine retries activities independently, with their own policy, without rerunning the workflow's earlier steps.

Activities must be idempotent, because a retry after a timeout may repeat work that already succeeded. The engine cannot know whether a call that timed out was received.

Keep workflow code thin. Put orchestration, sequencing and decisions in the workflow, and everything else in activities. Teams that follow this split rarely meet a determinism problem.

Versioning a running workflow

Versioning is a problem specific to long-running code, and it usually arrives a few months after launch.

Instances outlive deployments. A workflow running for six weeks will be replayed against whatever code is deployed at the time.

Changing the code changes the replay. If the new version makes different calls in a different order, replaying an old instance against it produces a history mismatch.

Engines provide versioning mechanisms, such as a marker that lets code branch on the version an instance started under.

Keep old branches until every instance that started on them has finished, which takes a long time for a long workflow.

Plan for the accumulation. A workflow changed six times has six branches, and pruning them requires knowing no instances remain on the old paths.

Additive changes at the end are safe. Reordering or removing steps in the middle is where instances break.

What durable execution does well

Processes that wait. Approvals, delivery windows, trial periods, anything measured in days.

Multi-step sequences with compensation. When step four fails and steps one to three must be undone in order, durable code expresses the undo logic more directly than a hand-built state machine.

Work that must not be lost. The history is persisted, so a crash mid-process resumes rather than restarts.

Complex orchestration in readable form. The alternative, a state machine spread across queue handlers and database columns, is harder to read and harder to change.

Visibility. You can ask where a specific instance is, which a hand-built equivalent rarely provides.

What it is not

Durable execution is sometimes proposed for jobs it does not suit, so the limits are worth stating.

Not a queue. Durable execution orchestrates processes and does not distribute high volumes of independent work efficiently.

Not a data pipeline. Moving large volumes through workflow code is the wrong shape; call something built for it from an activity.

Not low latency. Persistence and recording add overhead that matters for a request in a user's browser.

Not free operationally. A durable execution system is either a service you run or a vendor you depend on.

Not a substitute for design. A process you do not understand is still not understood when written as durable code.

Testing something that replays

Test workflows and activities separately. Activities are ordinary code and test ordinarily; workflows test against mocked activity results.

Test the replay explicitly. Engines typically provide a way to replay a recorded history against new code, which is how you catch a determinism break before deploying.

Keep histories from production as test fixtures, especially for long-running workflows, because those are the cases replayed against your change.

Test the failure paths. A step failing after three succeeded is the scenario that motivates the architecture, and it rarely appears in a test suite.

Test a version transition, with an instance that started on the old code, because that is the change most likely to break something already running.

The history, which grows

The history grows over time, and knowing this in advance avoids a production surprise.

Every activity call and every signal is appended to the instance's history, which is what makes replay possible.

Long workflows accumulate large histories, and replay cost grows with them. A workflow with a very large history is slow to resume.

Engines can cap history size (check your engine's documentation for the limit), and a workflow that exceeds the cap fails instead of slowing down.

Loops are the usual cause. A workflow that polls once a minute for a 30-day month records about 43,000 events (30 x 24 x 60) for what is conceptually one wait.

Continue-as-new is the standard remedy: the workflow completes and immediately starts a fresh instance carrying its state forward, truncating the history.

Design for it where a workflow is long-lived or loops, instead of discovering the limit in production. The failure arrives at an unpredictable point and looks like no other failure.

Durable automation with CodeWords

CodeWords handles durability for you instead of exposing it: you describe a process that spans days and waits for people, and the automation handles state, retries and resumption. CodeWords connects to 3,000+ integrations.

  1. Open CodeWords and start a new automation.
  2. Describe the process in plain language to Cody, the automation builder: the steps in order, what waits for what, and where a person decides.
  3. Describe the exceptions: a step that fails, an approval that never arrives, an input that arrives twice.
  4. Connect the systems each step touches, once.

You describe the outcome; Cody builds it, connects it, and deploys it. The Free plan covers light use. Pro costs $39 per month ($33 per month billed yearly) and Business costs $100 per month (CodeWords pricing page, 2026); details are on the pricing page.

What to get right first

Get the split between workflow code and activities right first: orchestration and decisions in the workflow, every side effect and every nondeterministic value in an activity. Teams that learn this early rarely meet a determinism error, and teams that do not can spend their first month debugging replays.

Two habits make the difference. Make every activity idempotent from the start, because the engine retries a timed-out call and cannot know whether the original succeeded. And keep a production history as a test fixture per workflow, because replaying it against a change is the only reliable way to catch a determinism break before it reaches running instances.

Frequently asked questions

How does a workflow survive a crash?

A workflow survives a crash by replaying. The engine runs your function again from the beginning, returns the recorded result for everything that already happened, and continues normally once execution reaches the point where the crash occurred. The code never sees the interruption.

Why must workflow code be deterministic?

Replay must make the same decisions as the original run. A clock read, a random value or an unordered iteration can send a replay down a different branch than the original took, which breaks the reconstruction and raises a nondeterminism error.

Where do network calls go?

Network calls go in activities. The engine records their results, so replay returns the recorded value instead of calling again. Keeping workflow code thin and pushing everything else into activities avoids nearly every determinism problem teams meet, because activities may do anything while the workflow only decides.

Do activities need to be idempotent?

Yes. The engine retries a call that times out and cannot know whether the original was received, so the same work may run twice. This is the same at-least-once reasoning that applies to any distributed call, and idempotency keys make retries safe.

What happens when I change a workflow's code?

Running instances are replayed against the new code, which can produce a history mismatch if steps change order. Use the engine's versioning mechanism and keep old branches until every instance that started on them has finished. Test the transition with an instance that began on the old code.

Is durable execution the same as a workflow engine?

Durable execution is one kind of workflow engine. It is distinguished by writing the process as ordinary sequential code instead of a graph or a configuration, and by achieving durability through replay instead of explicit state transitions that you define and maintain yourself.

When is durable execution the wrong tool?

It is the wrong tool for high-volume independent work, data movement and anything latency-sensitive. Durable execution orchestrates processes. It is not a queue or a pipeline, and its persistence overhead matters for a request that someone is waiting on in a browser.

Why did a long-running workflow suddenly fail?

The history probably reached the engine's size limit. Every activity call is appended, so a workflow that polls in a loop accumulates a large history. Continue-as-new truncates it by starting a fresh instance that carries the state forward, which resets the size.

Does replay get slower over time?

Yes, in proportion to the recorded history, because resumption replays all of it. A workflow with a very large history is noticeably slow to resume, which is another reason to avoid tight loops in workflow code and to use continue-as-new.

Can a running workflow receive new information?

Yes. Engines provide signals for this, and a signal is how an approval or a cancellation reaches an instance already in flight. The engine records signals like any other event, so they replay correctly after a crash or a deployment.

Should every long process use durable execution?

No. If restarting from the beginning after a failure is acceptable, and nothing waits on a person or an external event for days, the operational cost of a durable execution engine buys very little. Use a simple job or a queue instead.

Get started today

Your first workflow is free to build.

Describe what you need. Cody handles the build, the connections, and the deployment.