Skip to main content

Runbook

What healthy looks like​

  • GET /actuator/health is UP.
  • Kong route for this service returns a non-5xx for a known probe path.
  • If the service produces events, public JDBC plus polled schemas auth, social, feed, messaging, expense.event_outbox pending count is near zero and lag is under 30 seconds.

Failure modes​

  • All integration events stop: this process is down or cannot reach Redis. Producers keep committing outbox rows. Restart the relay; do not delete rows.
  • Duplicate side effects after a crash: consumers must dedupe on event_id. The relay may republish a row it published but failed to mark.

Diagnose​

  1. Take the traceId from the client error body or X-Request-Id.
  2. In Loki, filter service and traceId. Confirm outcome and httpStatus.
  3. If the symptom is delayed side effects, check outbox pending for schema public JDBC plus polled schemas auth, social, feed, messaging, expense and the consumer group named in Events.
  4. Check Hikari pool exhaustion and Redis latency on the service dashboard before restarting.

Repair​

Restart the single service container or Gradle process. Do not truncate outbox, inbox, or stream data to "clear" an error. Replay dead letters from the relay admin API after the consumer bug is fixed.