Skip to main content

Alerting

Page when users cannot complete a core action or a backlog will not drain by itself.

AlertLikely causeFirst move
ServiceDownProcess crashed or scrape failedContainer logs, then restart that service
KongHigh5xxRate / HighErrorRate5xxBad deploy or dependencyTrace Explorer for a sample id, then roll back the image
OutboxLagHighRelay or RedisOutbox runbook
OutboxFailedBacklogPoison payloadFix handler, then replay dead letters
InboxBacklogHighConsumer stuckPending entries for that group
PushDeadLetterRateHighAPNs or FCM rejectingPush runbook
HikariPoolExhaustedSlow queries or a leakDatabase runbook
ScheduledJobFailingCron exceptionService logs for the job name
RedisMemoryHigh / RedisEvictionsCache or stream maxlenCheck SHARED_EVENTS_STREAM_MAXLEN and presence keys
JvmHeapPressureLeak or undersized heapHeap dump on a non-prod clone first if you can; restart if the error budget is burning

Warning-level host alerts (disk warning, CPU high, I/O wait) go to a ticket, not a night page, unless they accompany ServiceDown.

Silence an alert only with an expiry and an incident note.