Durable Work in PostgresPart 3

Many workers

Multi-worker contention, delivery guarantees, and per-key ordering before consistent hash.

Traffic doubled. You added a second worker to the pool. Within an hour, two different processes tried to send the receipt for order:9182. Support got a duplicate email.

The bug is claiming without rules. You need clear delivery promises, such as at-least-once delivery and per-key ordering, plus guards that enforce them.

Competing consumers, same inbox

The producer inserts into inbox. Every worker runs the same loop: clean up expired leases, claim with SKIP LOCKED, handle, complete. No central dispatcher. Whoever wins the row lock processes the row.

That pattern is Competing Consumers: many workers, one queue table, and row locks instead of a broker assigning jobs. It scales throughput, but it does not automatically pin order:9182 to one process.

Workers registry and shared inbox
ConcernMechanism here
Throughput across machinesMany workers + SKIP LOCKED
Crash recoveryLeases + lease cleanup
Duplicate inbox rowsidempotency_key UNIQUE
Duplicate side effectsIdempotent handlers + fencing
Related work stays togetherNot automatic. Needs consistent hash + heartbeats

How to scale past one worker

There are three practical ways to coordinate multiple workers on the same inbox, from simplest to most controlled:

  1. Any worker claims any row: simplest multi-process setup. Throughput increases, but there is no per-key affinity or ordering. Use when jobs are independent.
  2. Per-key ordering guard: sibling check on claim gives FIFO within one partition_key without a coordinator. Hot keys bottleneck one worker.
  3. Consistent hash + heartbeats: stable affinity so a new worker moves ~1/N keys instead of the whole backlog. Rebalance windows and briefly stale ring caches are the tradeoff. Use when autoscale or deploy churn makes hash % N too disruptive.

Delivery guarantees to document

You have two workers. Write down what you promise before you ship.

PromiseStatusNotes
Row deliveryAt-least-onceLeases + lease cleanup. May retry unless dead_letter
Per-partition_key orderConditionalOnly with ordering guard + idempotent handler
Duplicate inbox rowsPreventedidempotency_key UNIQUE on enqueue
Duplicate side-effectsYour responsibilityIdempotent handlers. Fencing at downstream stores
Global FIFONot providedBy design. Single partition_key or ordered log

Idempotency at enqueue and in the handler

At enqueue: idempotency_key UNIQUE on INSERT. Protects producer retries.

In the handler: Worker may process the same row twice after reclaim. Check “already done” before side effects.

Per-key ordering

Same partition_key does not imply FIFO unless you opt in. Add a sibling guard on the claim query: refuse to claim if another row for that key is processing. Order by created_at among pending rows.

GuaranteeHowBreaks when
FIFO per partition_keyNOT EXISTS sibling processing; ORDER BY created_atTwo workers claim different keys concurrently (fine). Same key without guard (bad)
No duplicate inbox rowsON CONFLICT against the idempotency indexProducer omits key
No duplicate side-effectsHandler checks processed-keys / fence tokenHandler is not idempotent
Stable affinity per keyNot in this chapter. Needs consistent hash.

Ordering is opt-in. The base competing-consumer pattern gives parallelism, not automatic serialization. Add the sibling guard when per-key FIFO matters.

Simulation: two workers, no ring yet

Where this breaks down

  • No affinity. Two workers can still race on related keys unless you add ring ownership.
  • Hot partition_key. One tenant stream bottlenecks whichever worker wins the lock. Shard hot keys explicitly.
  • Rebalance window. When you add consistent hash, keys move while old leases may still be in flight. Plan for fencing during rebalance.
  • Split-brain after long pauses. A stale worker must fail complete and lose fence tokens downstream.