Skip to main content

Runbook: push dead letter

Alert: PushDeadLetterRateHigh.

Symptoms​

In-app inbox has the notification. The device does not. Or neither arrives, which means the stream consumer did not write the inbox row. Those are different faults. Check notification.notifications first.

Check​

  1. Dead-letter stream lag and DeadLetterLagMonitor logs.
  2. A sample token error from APNs or FCM. Invalid token is hygiene (TokenHygieneJob), not an outage. A burst of authentication errors means APNS_* or FCM_* is wrong or revoked.
  3. Preference flags. A muted category stores the inbox row and skips push. That is success.
  4. Clock and bundle id. A key for the wrong bundle fails every send.

Repair​

  • Bad credential: rotate in the secret manager, redeploy notification-service, do not commit the key.
  • Bad payload: fix the mapper, then replay from the retry path. Replaying without a fix grows the dead letter.
  • FCM disabled on purpose: Android push will not send. Say so in the incident note instead of treating it as a regression.

Inbox rows are the product record. Do not delete them to clear the alert.