Alerting
Page when users cannot complete a core action or a backlog will not drain by itself.
| Alert | Likely cause | First move |
|---|---|---|
ServiceDown | Process crashed or scrape failed | Container logs, then restart that service |
KongHigh5xxRate / HighErrorRate5xx | Bad deploy or dependency | Trace Explorer for a sample id, then roll back the image |
OutboxLagHigh | Relay or Redis | Outbox runbook |
OutboxFailedBacklog | Poison payload | Fix handler, then replay dead letters |
InboxBacklogHigh | Consumer stuck | Pending entries for that group |
PushDeadLetterRateHigh | APNs or FCM rejecting | Push runbook |
HikariPoolExhausted | Slow queries or a leak | Database runbook |
ScheduledJobFailing | Cron exception | Service logs for the job name |
RedisMemoryHigh / RedisEvictions | Cache or stream maxlen | Check SHARED_EVENTS_STREAM_MAXLEN and presence keys |
JvmHeapPressure | Leak or undersized heap | Heap dump on a non-prod clone first if you can; restart if the error budget is burning |
Warning-level host alerts (disk warning, CPU high, I/O wait) go to a ticket, not a night page, unless they accompany ServiceDown.
Silence an alert only with an expiry and an incident note.