A response runbook for production incidents in Next.js and Kafka environments, with triage flow, ownership model, and postmortem template.
Incident classification
In production Next.js and Apache Kafka stacks, outages can occur at different layers. A network timeout can cause database pool saturation, while a consumer error can stall message queues, delaying inventory updates or order routing. To handle these events without panic, engineering teams must operate under a structured, pre-defined incident response runbook.
This post details the operational runbook we use to triage and resolve incidents, categorizing failures by severity to allocate engineering resources efficiently:
- Class A (Critical): Outages directly impacting checkout, payment gateways, or primary databases. Requires immediate escalation and paging of the on-call SRE lead.
- Class B (Major): Severe latency degradation or failure of secondary features (e.g., search suggestions) with no immediate workaround. Requires team-wide mobilization during business hours.
- Class C (Minor): Partial failure of a service with an available workaround (e.g., catalog images failing to render but checkout remains functional). Handled via standard Jira ticketing.
Triage and escalation sequence
The diagram below outlines the communication flow from the initial alert trigger down to the deployment of a fix or rollback:
Required incident artifacts
Once an incident is resolved, the responding team must document the event. We maintain four key records for every Class A or B issue:
- Event Timeline: A chronological list of events, decisions, and system alerts with UTC timestamps.
- Decision Log: A list of critical actions taken (e.g., triggering a database rollback) and the name of the engineer who authorized it.
- Blast Radius Analysis: The list of affected customer segments, estimate of failed transactions, and database state audit.
- Remediation Actions: A set of ticket items with assigned owners and due dates, designed to prevent the issue from recurring.
Postmortem template
Our postmortems are blameless and focus entirely on structural improvements. The document is structured as follows:
### What happened
Provide a clear description of the incident from the user's perspective.
### Why it happened
Explain the technical root cause (e.g., database deadlock caused by overlapping cron updates).
### Detection and Recovery
Detail what metrics caught the issue and what steps were taken to restore service.
### Action Items
List the permanent engineering fixes, testing improvements, and monitoring updates needed.Our take
Incident response is an operational protocol, not a test of individual bravery. If your team resolves outages by calling senior engineers to debug in production without a plan, you will eventually cause data corruption or suffer prolonged downtime. Write your triage paths in advance, assign clear incident roles, and automate rollbacks based on telemetry limits.
