Skip to main content

Runbook: service down

Alert: ServiceDown, or a Kong 502 on one route.

Check​

  1. Container or process state and the last exit.
  2. Flyway errors in the log. A failed migration refuses to serve.
  3. Dependency: Postgres, Redis, or a missing env var. Missing env usually fails at startup, not after an hour.
  4. Port clash. Messaging and the relay both want 8088 inside a naive config. Compose maps the relay to host 8089. A manual start of both on 8088 fails one of them.
  5. Disk full (HostDiskSpaceLow) prevents logs and sometimes the JVM.

Repair​

Start the process with the same image tag if the failure is environmental. Roll back the image if the failure started at deploy and startup loops on a migration you cannot fix forward in minutes.

Kong returns 502 when the upstream is down. Fixing Kong does not help if the upstream is the one that exited.

If auth is the service that is down, every client fails login and refresh. Other services may still serve already-issued access tokens until they expire. Say that in the status note.