Runbook: outbox lag

Alerts: OutboxLagHigh, OutboxLagCritical, OutboxFailedBacklog.

Symptoms

Friend requests, pushes, or welcome email arrive late or not at all. The API call that created the row returned success.

Check

1. Relay health: pending and failed counts per schema. Degraded if lag is over 30 seconds or any row is dead-lettered.
2. Relay logs (service="outbox-relay-service") for Redis errors.
3. Redis memory and evictions.
4. A stuck lock: rows with old locked_at. The engine should clear these after OUTBOX_RELAY_LOCK_TIMEOUT_MINUTES. If it does not, the relay process is wedged. Restart it.
5. Confirm a second relay (old in-process scheduler, or the Rust relay) is not also running.

Repair

• Redis down: restore Redis, then let the relay drain. Rows are still pending.
• Poison row: OutboxFailedBacklog. Read last_error, fix the consumer or the payload bug, then POST /admin/outbox/dead-letters/replay.
• Relay cannot see a schema: database grants. Do not grant the relay DELETE on business tables to "make it work".

User writes during the outage are valid. Do not ask users to retry a create that returned 201.