Kafka Consumer Lag Incident Runbook Builder
Create a production runbook for Kafka or stream-processing consumer lag incidents with triage, offset safety, scaling options, root-cause checks, alerts, and recovery verification.
Prompt Template
You are a senior data platform engineer writing an incident runbook for consumer lag in Kafka-compatible event pipelines. Build the runbook for: Event platform: [Kafka, Confluent Cloud, Redpanda, MSK, Pulsar, Pub/Sub with Kafka API, other] Consumer group and services: [group names, owning teams, languages, frameworks] Topics and partitions: [critical topics, partition counts, retention windows, message volume] Symptoms: [lag increasing, stale dashboards, delayed notifications, timeout errors, rebalance loop, dead-letter growth] Business impact: [customer-facing delay, analytics freshness, billing delay, downstream SLA, internal-only] Current observability: [consumer lag metrics, broker metrics, traces, logs, dashboards, alerts] Deployment model: [Kubernetes, VMs, serverless consumers, autoscaling, blue-green, canary] Recent changes: [deploy, schema change, traffic spike, broker issue, dependency outage, config change] Consumer behavior: [idempotent, exactly-once, at-least-once, external side effects, batch processing] Recovery constraints: [no data loss, no duplicate side effects, retention deadline, on-call staffing, change freeze] Create: 1. Severity classification based on lag size, lag age, retention risk, and business impact. 2. First 15-minute triage checklist with commands, dashboards, logs, and ownership checks. 3. Decision tree for producer spike, slow consumer, poison message, rebalance storm, broker issue, dependency outage, or bad deploy. 4. Safe mitigation options including pause, scale out, rollback, batch size tuning, backpressure, and dead-letter handling. 5. Offset reset guidance with explicit warnings, approval requirements, and validation steps. 6. Customer or stakeholder communication template if freshness or processing SLAs are affected. 7. Recovery verification checklist using lag drain rate, processed counts, idempotency checks, and downstream data checks. 8. Post-incident root-cause analysis questions and permanent fixes. 9. Alert thresholds for lag count, lag age, consumer errors, rebalance frequency, and retention risk. 10. Follow-up backlog items for capacity planning, autoscaling, load testing, and runbook gaps. Do not recommend skipping messages, resetting offsets, or increasing parallelism blindly. Call out data loss, duplicate processing, and ordering risks before each risky action.
Example Output
Triage Summary
If lag age is within two hours of retention expiry or the topic feeds billing, treat the incident as high severity even if the raw lag count looks modest. Lag age and business impact matter more than message count alone.
First Checks
| Check | Signal | Action |
|---|---|---|
| Producer rate spike | Incoming bytes doubled | Estimate drain rate before scaling |
| Consumer errors | Repeated deserialize failures | Inspect recent schema or poison message |
| Rebalances | Group rebalances every minute | Check pod restarts and session timeout |
| Downstream dependency | API timeouts or DB locks | Protect dependency before adding workers |
Safe Mitigation
Rollback the last consumer deploy if errors began immediately after release. If consumers are healthy but behind, add replicas only up to the partition count and confirm downstream services can absorb the catch-up load.
Offset Warning
Offset reset requires incident commander approval, a snapshot of current offsets, clear replay or skip rationale, and downstream duplicate-risk review.
Tips for Best Results
- 💡Track lag age alongside lag count; retention risk is often the real deadline.
- 💡Check downstream dependencies before scaling consumers or you may amplify the outage.
- 💡Write offset-reset steps as a controlled emergency path, not a routine fix.
- 💡Validate downstream records after lag drains; zero lag does not prove correct processing.
Related Prompts
Microservices Architecture Migration Planner
Plan a structured migration from a monolithic application to microservices, covering service decomposition, data ownership, API contracts, migration phases, and rollback strategies.
gRPC Error Handling and Retry Design Guide
Design reliable gRPC service error handling with status codes, deadlines, retries, idempotency, interceptors, and observability.
Event Sourcing Replay and Backfill Runbook Builder
Plan a safe event replay or projection backfill with idempotency rules, batching, validation, rollback triggers, and operator checklists.