Resilience with events, sagas, and outbox

Enterprise onboarding is long-running work.

When a global customer signs, the system may need to:

• create the tenant configuration;
• reserve quotas;
• seed tenant rows, role templates, and feature defaults in the correct regional data-plane cluster;
• validate SAML or OIDC federation metadata;
• register billing entitlements;
• open or update an InfoSec review ticket;
• wait for Northstar’s Entra ID administrator to release production SAML metadata;
• compensate if a later step fails.

Putting all of that work inside one HTTP request creates slow responses, fragile retries, and unclear recovery after partial failure.

For Northstar Health Group, a single onboarding run may need to create Northstar US in the US data-plane cluster, reserve 100 seats, validate SAML metadata from Entra ID, create a security-review ticket, and keep Northstar EU out of the US runtime entirely. Some of that work is internal. Some depends on Northstar’s IT team. Some can fail hours after the initial request.

Events record facts; sagas coordinate long-running work

The dual-write problem

This is the trap:

await tenantRepository.save(configuration);
await messageBus.publish(new TenantRegistrationInitiated(configuration.tenantId));

If the database write succeeds and the broker publish fails, the tenant exists but provisioning never starts. If the broker succeeds and the database rolls back, downstream systems react to a tenant that does not exist.

That is a dual write. It will fail eventually.

Transactional outbox

Save the aggregate and event in the same database transaction:

await db.transaction(async (tx) => {
  await tenantConfigurationRepository.save(tx, configuration);

  await outbox.save(tx, {
    id: eventId(),
    type: "tenant.registration_initiated.v1",
    aggregateId: configuration.tenantId,
    occurredAt: clock().toISOString(),
    payload: toIntegrationEvent(new TenantRegistrationInitiated(configuration.tenantId)),
  });
});

Then a separate dispatcher publishes unsent outbox rows:

const messages = await outbox.claimBatch({ limit: 100 });

for (const message of messages) {
  await messageBus.publish(message.type, message.payload);
  await outbox.markPublished(message.id);
}

The domain transaction is fast. Event delivery is retryable.

Example onboarding flow

The happy path is a sequence of small, observable steps:

1. TenantRegistrationInitiated
2. SeedTenantDataPlane(Northstar US, us-east)
3. ValidateIdentityMetadata(Entra SAML)
4. RequestSeatAllocation(100 seats)
5. OpenSecurityReviewTicket
6. TenantOnboardingCompleted

The onboarding context does not do all of that work itself. It records the fact that registration started, then the process manager coordinates commands across Platform, IAM, Billing, and Security.

Domain events versus integration events

Keep the distinction clear:

class TenantRegistrationInitiated {
  constructor(
    readonly tenantId: TenantId,
    readonly organizationId: GlobalOrganizationId,
    readonly occurredAt: Date,
  ) {}
}

The integration event is a versioned contract:

type TenantRegistrationInitiatedV1 = {
  type: "tenant.registration_initiated.v1";
  tenantId: string;
  organizationId: string;
  occurredAt: string;
};

Map between them at the boundary. Do not leak TypeScript domain objects onto the message bus.

Saga / process manager

The saga coordinates work across bounded contexts:

type OnboardingSagaState = {
  tenantId: TenantId;
  dataPlaneSeeded: boolean;
  identityMetadataValidated: boolean;
  billingEntitlementsGranted: boolean;
  securityReviewTicketId: string | null;
  failedStep: string | null;
};

It listens to events and issues commands:

async function onTenantRegistrationInitiated(event: TenantRegistrationInitiatedV1) {
  const saga = await sagaRepository.loadOrCreate(event.tenantId);

  await commandBus.send({
    type: "SeedTenantDataPlane",
    tenantId: event.tenantId,
    dataRegion: event.dataRegion,
    correlationId: saga.correlationId,
  });
}

If a step fails, the saga decides the compensating action:

async function onSeatAllocationRejected(event: SeatAllocationRejectedV1) {
  const saga = await sagaRepository.load(event.correlationId);

  await commandBus.send({
    type: "RevokeDataPlaneAccess",
    tenantId: saga.tenantId,
    reason: "seat allocation rejected",
  });

  await sagaRepository.save(saga.blocked("billing", event.reason));
}

This is not an aggregate. It is a process manager.

Error modes and compensating actions

Workflows fail in different ways. Treating every failure as “retry later” is how systems end up with half-provisioned tenants.

Compensation should undo external side effects, not rewrite history. If tenant seed rows were created and no user can access them, the system may keep them for retry. If data-plane access was granted before Billing rejected the seat allocation, the saga must revoke access and leave an audit trail.

Test the saga decision

The important test is not “does the message handler run?” It is “does the process make the right business decision after partial failure?”

it("blocks onboarding and revokes partition access when Billing rejects seats", async () => {
  const parsedTenantId = tenantId("ten_claims_us");
  const saga = onboardingSaga({
    tenantId: parsedTenantId,
    dataPlaneSeeded: true,
  });

  sagaRepository.load.mockResolvedValue(saga);

  await onSeatAllocationRejected({
    type: "billing.seat_allocation_rejected.v1",
    tenantId: parsedTenantId,
    correlationId: saga.correlationId,
    reason: "requested 500 seats but contract allows 100",
  });

  expect(commandBus.send).toHaveBeenCalledWith({
    type: "RevokeDataPlaneAccess",
    tenantId: parsedTenantId,
    reason: "seat allocation rejected",
  });
  expect(sagaRepository.save).toHaveBeenCalledWith(expect.objectContaining({ status: "blocked" }));
});

That test gives the team confidence that a partially seeded tenant does not quietly become active after a commercial rejection.

AsyncLocalStorage for traceability

Background work still needs tenant and correlation context.

type ExecutionContext = {
  tenantId?: TenantId;
  correlationId: CorrelationId;
};

const executionContext = new AsyncLocalStorage<ExecutionContext>();

export function withExecutionContext<T>(context: ExecutionContext, fn: () => Promise<T>) {
  return executionContext.run(context, fn);
}

Handlers can log and tag work without passing tenantId through every method:

logger.info("seeding tenant data plane", executionContext.getStore());

This is especially useful when a saga fans out into background promises, retries, and event handlers.

Event loop hygiene

Do not block Node.js with heavy provisioning work. Break the flow into commands and handlers:

1. TenantRegistrationInitiated
2. SeedTenantDataPlane
3. ValidateIdentityMetadata
4. GrantBillingEntitlements
5. OpenSecurityReviewTicket
6. TenantOnboardingCompleted

Each step is retryable, observable, and small enough not to monopolize the event loop.

The takeaway

Decoupling long-running operations with transactional events keeps the core onboarding flow fast and resilient.

The outbox makes state change and event publication atomic. The saga coordinates time. Compensating actions make partial failure explicit.

Principles to apply in your own work

1. Identify every place you save domain state and publish a message in separate operations.
2. Move those messages into an outbox written in the same database transaction.
3. Give each long-running workflow persistent saga state and a correlation id.
4. Define one compensating command for every external side effect.
5. Make every event handler idempotent before adding retries.

Next: Bulletproof shared-schema isolation.