On-call checklist
First five minutes
- Read the alert name and the service label. Open Alerting.
- Open Service Health and Async Pipeline.
- If there is a user report, take
traceIdfrom the error body. - Decide: user-facing (5xx, login, send message, create expense) or delayed side effect (push, email, feed projection). Side effects usually mean the relay or a consumer, not a rollback of the write.
While it is happening
- One person changes production. Say what you changed.
- Restart a single service before you restart the host.
- Do not truncate
event_outbox, streams, or inbox tables. - Do not flip Pub/Sub toggles unless you are executing the ADR 0003 rollback on purpose.
- Capture the image tag that is running before you roll back.
After
- Note the start, the mitigation, and the follow-up.
- If the fix is a code change, add or adjust the runbook section that would have saved the next person ten minutes.
- If an alert did not fire, add one. If an alert fired and nothing was wrong, narrow it.
Escalation is a human list kept by the team, not a name baked into git.