Implementing Durable Workflows on Postgres Without an External Orchestrator

When we started building Kestrel Workflows (deterministic workflows for automating incident response, cloud provisioning, CI/CD, and developer requests), we hit the problem every automation system eventually hits. Our workflows had to be durable and survive all types of failures.

A workflow can be defined to trigger on failing Kubernetes workloads, run an AI root cause analysis, generate a fix, wait for a human to approve the fix via Slack, and then open a GitOps pull request. If any of our systems are rescheduled in the middle of such a workflow (in Kubernetes, it most likely will) we couldn't just lose the workflow execution. We needed durable execution.

The idea behind durable execution is quite simple and powerful. As a program runs, it checkpoints its progress to an external system like a database. If the process dies, another one reloads the last checkpoint from the external system and continues from the last completed step. It is like autosave from a video game, but for backend code.

The Default: Using an External Orchestrator

The usual advice for implementing durable execution is to use an external orchestrator like Temporal or AWS Step Functions, which act as central orchestrators that coordinate steps in a workflow. In this model, a client submits a workflow, the orchestrator persists it in its own data store, and dispatches it to a worker.

For every completed step, the workflow sends its results back to the orchestrator, which checkpoints the progress and then dispatches the next step. If a worker dies, the orchestrator reassigns its steps to a healthy worker.

Figure 1. External orchestration, where a dedicated orchestrator coordinates dispatch, queues, checkpoints, and workers. (Source: created by author.)

We almost went this way, but reconsidered when we looked at what it would cost us.

Using an external orchestrator means having another stateful system to deploy, secure, monitor, and upgrade. It sits in the critical path of every workflow, so it becomes a single point of failure. It needs its own access controls and audit handling because workflow step payloads, which for us include things like infrastructure topology, source code, and logs, now transmit to an external system. It brings its own data model, which is usually a key-value store optimized for workflow state rather than arbitrary relational queries, making ad-hoc analysis less straightforward than querying Postgres directly.

But here's the thing: Durable execution is fundamentally about a database since it's all about checkpointing state to a durable external system. We already run Postgres as our system of record and have invested heavily in making sure it performs at scale, so we asked ourselves: Why use an external orchestrator at all? Why not make Postgres the orchestrator?

Figure 2. Postgres-backed durable execution, where stateless application servers coordinate workflow state directly through Postgres. (Source: created by author.)

Postgres as the Orchestrator

In Kestrel, there is no orchestrator process. Each application server runs an embedded durable workflow library and talks straight to Postgres. Every trigger, from PagerDuty alerts to GitHub webhooks, inserts a row into a workflow_executions table:

CREATE TABLE workflow_executions (
	id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
 	workflow_name TEXT NOT NULL,
 	status TEXT NOT NULL DEFAULT 'enqueued',
 	input JSONB NOT NULL,
	owner_id TEXT, -- which worker holds the lease
	lease_expires TIMESTAMPTZ, -- when the lease lapses
	created_at TIMESTAMPTZ NOT NULL DEFAULT now(), 
	updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Servers poll this table to claim work. For each completed step, the server checkpoints the output to Postgres itself.

One Clause That Makes Everything Safe

All coordination happens through the Postgres database. The trick that makes it safe is one clause:

SELECT id, input
FROM workflow_executions
WHERE status = 'enqueued'
ORDER BY created_at
LIMIT 1
FOR UPDATE SKIP LOCKED;

FOR UPDATE SKIP LOCKED is the quiet hero of most Postgres-backed queues. It locks the rows a worker grabs and tells every other worker to skip past them rather than block, giving the system exactly-once processing guarantees. Two servers can poll the same table at the same time and never hand the same workflow to two executors.

Although this article focuses on Postgres, this queue pattern is not specific to Postgres: MySQL 8.0+, MariaDB 10.6+, and Oracle also support SKIP LOCKED, while Db2 and SQL Server provide equivalent mechanisms for skipping locked rows.

This claim happens inside a short transaction that also flips the status and writes the lease, so the worker owns the row before it does any work:

BEGIN;
 
WITH claimed AS (
	SELECT id
	FROM workflow_executions
	WHERE status = 'enqueued'
	ORDER BY created_at
	LIMIT 1
	FOR UPDATE SKIP LOCKED
)
UPDATE workflow_executions e
SET status = 'running',
     	owner_id = $1, -- the worker's id
     	lease_expires = now() + INTERVAL '30 seconds',
     	updated_at = now()
FROM claimed
WHERE e.id = claimed.id
RETURNING e.id, e.input;
COMMIT;

This transaction is the entire dispatcher. You don't need a broker, leader election, or Redis locks with TTLs. A few things that should be called out for production implementations:

• The claim transaction should be tiny. It should just lock the row, flip the status, and commit. If you hold the FOR UPDATE transaction open while the step runs, you will pin a row lock for the duration of potentially long running, external work.
• LIMIT can be more than 1 to claim a batch of executions and amortize round trips, but the larger the batch is, the larger the blast radius if the worker dies before it finishes them.
• SKIP LOCKED does not preserve FIFO under contention; for our workflows that is fine and actually desirable, but if you need strict ordering, this pattern won't work.

Let the Database Enforce Idempotency

Step checkpoints make use of a second Postgres primitive: integrity constraints. Each step writes its output to an operation_outputs table keyed by (execution_id, step_id) as the primary key:

CREATE TABLE operation_outputs (
	execution_id BIGINT NOT NULL REFERENCES workflow_executions(id),
	step_id TEXT NOT NULL,
	output JSONB NOT NULL,
	created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
	PRIMARY KEY (execution_id, step_id)
);

When a step finishes, the worker records the result with an upsert that never overwrites:

INSERT INTO operation_outputs (execution_id, step_id, output)
VALUES ($1, $2, $3)
ON CONFLICT (execution_id, step_id) DO NOTHING
RETURNING output;

If a recovering worker reruns a step that has already committed, the unique constraint prevents a duplicate checkpoint from being inserted. The worker reads the existing checkpoint and returns the prior result instead. The database, rather than the application code, enforces idempotency.

Before executing a step, the Orchestrator SDK running on the worker just checks operation_outputs for the (execution_id, step_id). If a row exists, it skips execution and returns the stored output downstream. Here again, we express an invariant of durable execution as a constraint and let Postgres primitives uphold it, instead of writing distributed systems bookkeeping from scratch.

Crash Recovery: Leases and a Sweeper

The interesting failures are the ones where a worker claims an execution, sets status to running, and then dies, because OOM kills, node drains, pod evictions, or a dozen other Kubernetes failure reasons. These rows get stuck in running state without a live owner; this is where the lease columns come into play.

A worker that owns a running execution heartbeats it on an interval, pushing the lease forward:

UPDATE workflow_executions
SET lease_expires = now() + INTERVAL '30 seconds',
	updated_at = now()
WHERE id = $1 and owner_id = $2;

Implementing recovery with Postgres is just as simple as enforcing concurrency and idempotency. A sweeper resets executions whose worker has not renewed its lease and a healthy worker reloads the checkpoints and resumes from the most recently completed steps:

UPDATE workflow_executions
SET status = 'enqueued', owner_id = NULL
WHERE status = 'running'
AND lease_expires < now();

There are two important details that matter when implementing this for production use. The lease duration is a tradeoff. If it's too short, a slow-but-alive worker can have its work stolen, resulting in two workers running the same execution (this is safe thanks to the idempotent checkpoints, but still wasteful). If it's too long, then recovery after a crash is delayed.

Without idempotent checkpoints, which are free with Postgres, we would not be able to use aggressive leases because a false-positive sweep would cause double-execution with side effects, not just duplicated work.

Durable Sleeps and Human Approvals

Even waiting for a human to approve workflows from Slack, which can take an arbitrary amount of time, is durable. It is simply a row inserted into a workflow_waits table with a wake_at timestamp, instead of goroutines praying the pod it's running on lives long enough to not miss the approval decision:

CREATE TABLE workflow_waits (
	execution_id BIGINT NOT NULL REFERENCES workflow_executions(id),
	step_id TEXT NOT NULL,
	wake_at TIMESTAMPTZ NOT NULL,
	PRIMARY KEY (execution_id, step_id)
);

When a workflow execution hits a sleep or approval gate, it sets its own status to waiting and inserts a workflow_waits row with a wake_at timestamp, while a periodic sweeper re-enqueues anything due:

UPDATE workflow_executions e
SET status = 'enqueued'
FROM workflow_waits w
WHERE w.execution_id = e.id
	AND e.status = 'waiting'
	AND w.wake_at <= now();

This way, wait and approval workflow steps survive any number of restarts or reschedules, because their state is stored durably in Postgres, not a thread in memory. No matter how long a wait or approval window is, whether it is two hours or two weeks, the cost is the same: just one row.

Why This Scales

Scalability and availability are the hard problems when you push orchestration into Postgres, which is a good thing, because they are well-understood with proven solutions. In Kestrel's case, these were problems we had previously tackled. We scale throughput by adding stateless workers; the bottleneck becomes how fast Postgres can clear the queue. We have seen that a single instance can handle tens of thousands of workflows per second without adding read replicas or using tools like Citus.

Availability becomes whatever your Postgres High Availability setup already gives, such as streaming replication and managed multi-AZ failover, because workers are fungible and can recover each other's state.

Operational Considerations

Connection pressure is something to look out for, because hundreds of polling workers, each with a handful of connections, can exhaust Postgres' connection limit quickly. A solution to this is to use PgBouncer transaction pooling in front of Postgres and have idle workers back off with a poll interval and jitter. But if you need fan-out across thousands of workers and sub-millisecond dispatch latency, an external durable execution engine like Temporal may be the better architectural choice. Because our workloads are I/O-bound automation pipelines, which can run from seconds to hours, need only modest concurrency, and are heavy on external calls, using Postgres is not a compromise, it is actually a better fit.

Another thing to watch for is dead tuples becoming a bottleneck at high churn. A busy queue table can become a hotspot of UPDATEs and DELETEs, which generates bloat. Thankfully, tuning autovacuum on these tables, keeping the hot row set small with partial indexes, and partitioning completed executions away from the live queue will keep the working set small and avoid this problem entirely.

Observability for Free

Observability is now free, which is what I like most about using Postgres here. Every workflow and every step is a row in a table, so monitoring is just SQL. For example, if you want every run that errored this month, you just execute:

SELECT * FROM workflow_executions
WHERE created_at > NOW() - INTERVAL '1 month'
AND status = 'error';

This looks extremely trivial, but it's only possible because Postgres is relational. You can do a lot more powerful things, like joining executions against steps and approvals to answer questions like "Which fixes got rejected and at which step?" 

We also maintain a few targeted secondary indexes on columns like status and creation time to keep these queries fast without adding much write overhead. These indexes let us build a real-time workflow observability dashboard with no extra infrastructure. With external orchestrators, these kinds of relational queries require exporting workflow state from key-value stores to separate analytics systems.

Reliability and Security Collapse to One Dependency

Another benefit of using Postgres for durable workflows is that reliability and security collapse into a single dependency. The only thing that can take down Kestrel workflows is Postgres failing, but as our primary system of record, if Postgres fails, so does everything else. At higher workflow volumes, it can make sense to isolate workflow state from application state in a separate Postgres deployment for resource isolation, while keeping the same operational model. Because no workflow data ever leaves the database, we implemented durable execution without adding new attack surface and systems to audit.

Databases like Postgres are already part of so many companies' infrastructure and teams already know how to operate them at scale. So, we decided it made more sense to reuse ours instead of bolting on an external orchestrator on top of it. If you already run Postgres, you have everything you need for durable workflows too.