Durable Work in PostgresPart 7
Cross-service handoffs
Transactional outbox, transport choices, webhooks, contracts, and when brokers fit. Work that leaves your database commit.
Payment for order:9182 commits while the search service is unavailable. The order service can
truthfully tell the customer that payment succeeded, but search still shows the order as pending.
Once those records live in databases owned by different services, no local transaction can make
both changes happen at the same instant or roll one back when the other is temporarily unreachable.
This is where a local queue becomes a cross-service handoff. Inside one database, Postgres can commit the business change and follow-up row atomically. Across databases and deployable services, that guarantee ends. What you can preserve is responsibility: the producer records durable intent, the transport keeps attempting delivery, and the receiver applies the event idempotently.
I draw that boundary explicitly because phrases like “eventually consistent” are too vague for the people affected. Product needs a lag promise, support needs a status to inspect, and engineers need to know where retries and deduplication live. This post builds that contract with a transactional outbox, a transport, and an inbox on the receiving side.
When the commit boundary ends
Inside one database, the story is clean: one commit can update an order and enqueue the follow-up work. The moment work leaves that database or deployable unit, you no longer have one transaction. You have a handoff.
From that point on, you need transport: a way for intent to cross from your Postgres into someone else’s process. The important boundary is not necessarily a network hop. It is the point where one transaction and one team can no longer control both sides, such as a different database, deployable service, broker, or external API. On each side of that boundary, a durable row records what the local system still owes.
The transport delivers bytes. Each service still owns its durable state and recovery.
Use an outbox relay on the producer and an inbox on the receiver. Transport does not replace leases or claim loops on either side.
What each part can guarantee
Your database can make the order update and outbox insert succeed or fail together. It cannot make a broker and search index join that same local transaction, so “paid” and “visible in search” become different guarantees. The first is true when the producer commits. The second becomes true later, after transport and the consumer succeed, and should have its own measurable lag objective.
Transport and downstream consumers are always eventual. Only the producer database gets ACID on write.
| Layer | What holds | Consistency | Postgres tools |
|---|---|---|---|
| Inside your DB | Business row + outbox/inbox row commit together. Claims are exclusive | Strong (ACID, row locks) | Transactional outbox, SKIP LOCKED, leases |
| Your worker layer | Each row processed at least once. Optional per-key FIFO | At-least-once + idempotent effects | Hash ring, heartbeats, ordering guard, fencing |
| Transport | Message or HTTP delivery may retry, reorder (across keys), or lag | Eventual, at-least-once typical | Broker, webhook, outbox relay |
| Downstream consumer | Their read model, inbox, or side effect catches up later | Eventual: bounded lag is the service-level objective (SLO) | Their idempotency + your contract |
Cross-boundary architecture
Guarantees stop at each service boundary. Make the handoff explicit: durable intent on your side, idempotent consumption on theirs, and transport in between that may retry.
Three zones: producer DB (ACID), transport (at-least-once), consumer (catches up later).
Producer outbox and consumer inbox
Open in StudioTransactional outbox for order:9182
Make the intent durable. When payment succeeds, you need search to know eventually. You cannot call search inside the transaction. You can insert an outbox row that says “publish OrderPaid” in the same commit as UPDATE orders.
The Transactional Outbox pattern is the bridge: write intent in the same transaction as the domain mutation, then let a relay worker publish to your transport. In this implementation, the relay is the same competing-consumers loop from earlier chapters, applied to outbound events. You get:
- Atomic intent: the paid order and
OrderPaidoutbox row share one commit - Relay: a worker claims the outbox row and publishes it to the transport
- At-least-once publish: a crash after publish but before completion causes a retry
- Consumer deduplication: the subscriber uses
event_idas its idempotency key
BEGIN;
UPDATE orders SET status = 'paid' WHERE id = 9182; -- business fact the API must not lie about
INSERT INTO outbox (partition_key, event_type, payload, idempotency_key)
VALUES (
'order:9182',
'OrderPaid',
'{"order_id":9182,"paid_at":"2026-07-01T12:00:00Z"}'::jsonb,
'evt-order-9182-paid-v1'
); -- durable publish intent, not the wire publish
COMMIT; -- both rows exist or neither. No ghost OrderPaid events
Follow the same event across the boundary:
| Time | State | What support can say |
|---|---|---|
| 12:00:00.000 | Order and outbox row commit | Payment is recorded and search update is owed |
| 12:00:00.120 | Relay claims evt-order-9182-paid-v1 | Delivery is in progress |
| 12:00:03.120 | Publish times out | Outcome is ambiguous, so the same event id will retry |
| 12:00:05.400 | Relay publishes the retry | Transport accepted the event |
| 12:00:05.470 | Search inbox deduplicates and updates | Search has caught up in 5.47 seconds |
That timeline gives product a lag measure and gives support one id to follow. It also shows why a timeout cannot create a new event id.
Choosing a transport
After the outbox row is durable, choose transport based on the consumer, the SLO, and who operates the middleware.
| Kind | Good when | Optimizes for | Breaks when |
|---|---|---|---|
| Message broker | Service-to-service delivery, routing keys, ops-owned middleware | Fast handoff and per-queue routing | You need long retention replay for many independent consumers |
| Event log / stream | Many subscribers, replay, high-volume shared log | Fan-out and retention | Point-to-point only with one consumer team |
| Managed queue | Cloud-native, minimal ops, push to one consumer group | Managed durability and scaling | Strict per-key ordering without paying for FIFO SKUs |
| HTTP / webhooks | One partner or SaaS exposes a URL | Simple integration surface | Partner outage blocks your relay unless you queue and retry |
| Poll / RPC pull | No push. Batch export or client-pull APIs | Receiver controls cadence | Low latency SLOs without aggressive polling |
| CDC | Downstream needs row changes, not domain events | Decoupling from app publish code | Schema refactors and domain event contracts |
Inbound: webhooks and broker consumers become inbox rows
The receiver applies the mirror image. It verifies the transport, inserts an inbox row using
event_id as a unique key, and returns 2xx only after the row commits. The claim loop handles the
business effect later. For the full HTTP version, including signatures, timeouts, and retry policy,
see Build your own webhook transport.
Contracts across boundaries
Shared transactions are gone. Shared vocabulary is not.
When two teams own two databases, coordination becomes documentation plus stable ids. The table
below is the minimum contract for OrderPaid on order:9182 to land safely in search.
| Contract piece | Producer owes | Consumer owes |
|---|---|---|
event_id / idempotency key | Stable, unique per logical event | Dedupe store or unique constraint |
partition_key / message key | Same key for all ordered events in a stream | Single consumer per key (or ordering guard) |
| Schema version | Backward-compatible changes. Bump version on breaks | Reject or dead-letter unknown versions |
| Delivery semantics | Document at-least-once. No silent drops | Idempotent handlers. Expose lag metrics |
| Replay | Retention policy or archive for re-publish | Safe to re-process historical events |
Sagas and long flows across services
Some workflows outlive one request. Charging a card, reserving inventory, shipping, and updating search are several local commits linked by durable facts, not one distributed lock. Keep each step idempotent, give it a timeout, and model compensation when a committed effect must be countered. Use a saga or workflow record to track business progress instead of stretching the transport into a business state machine.
Explaining lag to product
Payment in your DB can be immediate while search and analytics catch up on a separate timeline. Use this distinction when someone asks why search lags payment by thirty seconds.
- User-facing write path. Strong in your DB: “Payment recorded” is true when API returns success.
- Derived views. Eventual. Search, analytics, recommendations catch up in seconds (or minutes: define SLO).
- External systems. Eventual with retries. A webhook may arrive twice. Partner ledger may lag. Contract defines max delay. Same mechanics whether the other party is another team or Stripe.
- Failure mode: At-least-once everywhere outside ACID. Duplicates are normal. Idempotency is not optional.
Where this implementation fits
Match the tool to the constraint. Postgres coordinates work inside a service. Brokers and queues help at the boundary when fan-out, retention, or cross-team delivery require them.
Strong fit
- Durable work with claims, leases, and retries
- Postgres already in the stack (or inbox/outbox tables are acceptable)
- Per-entity ordering or partition affinity matters
- Producers and workers share a database. Or outbox/inbox at each service boundary
- Backlog depth and stuck rows should be visible in SQL
Weaker fit: consider other tools
- Shared event log with long retention and many independent replay consumers → Kafka / Pulsar as platform infra
- Fan-out streaming is the product, not per-service claim coordination
- Jobs-only background work, no custom ordering → pg-boss, Graphile Worker, or River
- No database on either end can participate in inbox/outbox
- Postgres is saturated after tuning and coordination itself must move off-DB
Source
These companion files collect the SQL and TypeScript used in this article. The link labels distinguish full implementations from focused examples and worker skeletons.