The Art of Debugging in Production

Debugging production issues is fundamentally different from debugging in development. Your beautiful stack traces are gone, replaced by cryptic error codes and urgent Slack messages. The key difference is that you need structured logging, a systematic approach, and a clear incident playbook before problems occur.

Structured Logging Saves Lives

The single best investment I made was switching from `console.log` to structured JSON logging. Every log entry now has a correlation ID, timestamp, service name, and severity level. This makes logs searchable, filterable, and useful when tracing issues across distributed systems.

import pino from 'pino';

const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  formatters: {
    level: (label) => ({ level: label }),
  },
  base: {
    service: 'api-gateway',
    env: process.env.NODE_ENV,
  },
});

// Usage
logger.info({ userId, action: 'checkout', cartSize: items.length }, 'User initiated checkout');

The Incident Playbook

Every on-call rotation taught me something. Here is the systematic approach I follow for every production incident:

1. **Do not panic.** Breathe. Read the alert carefully. 2. **Check the dashboards** before touching any code. 3. **Communicate early.** A quick 'investigating' message buys goodwill. 4. **Roll back first, debug later.** If a deploy caused it, revert immediately. 5. **Write a post-mortem.** Not to blame, but to learn.

Systematic Debugging Approach

The best debuggers I know are not the smartest engineers. They are the most systematic. They form hypotheses, test them one at a time, and document what they find. It is science, not magic.

Before touching code, ask these questions: - What changed recently? (deploys, config changes, traffic spikes) - What is the blast radius? (one user, one region, everyone) - Is this a new issue or a recurring pattern? - What do the logs and metrics show?

Key Takeaways

- Structured JSON logging with correlation IDs is the single most important debugging investment - Always check dashboards and metrics before touching code during an incident - Roll back first and debug later when a deploy is the likely cause - Communicate early with a brief status update to buy time and build trust - Write post-mortems focused on learning, not blame - Systematic investigation (forming and testing hypotheses) beats intuition - Prepare your logging and monitoring before incidents happen, not during them