Shelf · Core Eng · Chapter 9

Chapter 9: Observability

Every "the Jenkins build is green but finance says the data's wrong" incident you've shipped had an observability gap at its root - no signal told anyone quickly, and nothing correlated what happened to why. This chapter gives you the four signal types, the metrics and SLO discipline behind sane alerting, and the actual method for responding to and diagnosing a production incident without a debugger attached.

Difficulty
Priority
ce-09-01Card 1 of 6

Observability model: logs, metrics, traces and events

MediumP2 · Very likelyTalk about it12 min
observabilitylogsmetricstraceseventsopentelemetrycorrelation-id
Needs: nothing. Start here.

Why you care

"You get paged. The dashboard says the checkout error rate is up 4x. Which tool do you open first, and why?" is the real version of "what's the difference between logs, metrics, and traces," and it gets asked constantly because a wrong instinct wastes the first ten minutes of every incident. Pure logs mean grepping blind across machines with no sense of "how often." Pure metrics mean you can see something is wrong with zero narrative for any one request. Someone who has only ever print()ed into a Jenkins console, with no trace, no metric, and no durable event store, will build a system nobody can debug once it leaves their laptop.

Explain like I'm 5

A hospital runs on all four signals at once. Logs are the nurse's handwritten chart on one patient: detailed, per-event, cheap to write and expensive to search across a thousand patients. Metrics are the ward's vital-sign board: a handful of numbers, refreshed every minute, that tell you something is wrong across every patient at once but nothing about any one of them. Traces are the patient's routing slip, stapled on at admission with a single ID, carried room to room and timestamped at every stop, so you can see the three hours they specifically spent waiting in radiology. Events are the entries that matter on their own, even without the slip: "medication administered at 14:02" has to survive and stay searchable long after the slip itself is thrown away.

Core concept

  • Logs: discrete, timestamped records, usually per request or per step. Cheap to write, expensive to search at volume unless they're structured (Add structured logs and Python OpenTelemetry).
  • Metrics: aggregated numeric time series. Cheap at any volume, but carry no per-request detail - you cannot ask a metric "which customer."
  • Traces: a tree of causally-linked spans showing one request's path across services. A span has a name, start/end time, attributes, a status, and can carry span events - timestamped notes attached to that span, same id space and same lifecycle as the trace.
  • Events: point-in-time occurrences. They either ride inside a trace as a span event, or stand alone in a durable store (an audit table, a business-event log) with their own retention.
  • Correlation ids: generate one id (the trace id) at the edge, log it on every line, and propagate it across process boundaries via headers (W3C Trace Context is the default). That one id is what lets you jump from metric to trace to log for a single request instead of guessing by time window across three tools.
  • Layers: instrumentation (what the app does - creates spans, counters, log lines), export (how signals leave the process - batched to a collector or backend), and storage (where they live and for how long) fail independently. "The logs exist" does not mean they're indexed or queryable yet.
  • Decision rule: pick the tool by the question. "Is something wrong right now, and how bad" → a metric. "What exactly happened to this one request" → a trace, then logs filtered by its id. "Did a specific business fact happen, and will someone ask about it in three months" → an independent, durably-stored event, never a span event, because a span event is not a system of record.

Under the hood

Span events live physically inside the span they belong to - no separate table, no separate id space. That is exactly why they vanish together with the trace. If your tracing backend samples 1% of traces to control cost, the 99% of span events attached to dropped traces are dropped too, silently, because the sampling decision runs independent of whether anyone later needed that event for an audit. A metric survives this because it's emitted independent of any one trace's sampling decision; a durable event survives it because you deliberately wrote it somewhere that is not the tracing backend at all.

Minimal example One trace_id ties three signals together

PythonA metric alone cannot answer "which request"runs
logs = [
    {"trace_id": "T1", "service": "checkout", "msg": "received order", "ts": 1},
    {"trace_id": "T2", "service": "checkout", "msg": "received order", "ts": 2},
    {"trace_id": "T1", "service": "payments", "msg": "card declined", "ts": 3},
]
spans = [
    {"trace_id": "T1", "span_id": "s1", "service": "checkout", "parent": None},
    {"trace_id": "T1", "span_id": "s2", "service": "payments", "parent": "s1"},
    {"trace_id": "T2", "span_id": "s3", "service": "checkout", "parent": None},
]
metric_checkout_errors_total = 1  # a counter: no trace_id, just a number


def narrative(trace_id):
    path = [s["service"] for s in sorted(spans, key=lambda s: s["span_id"]) if s["trace_id"] == trace_id]
    lines = [l["msg"] for l in logs if l["trace_id"] == trace_id]
    return path, lines


print(narrative("T1"))
print("metric alone tells us:", metric_checkout_errors_total, "error(s), for which request? unknown")
Output
(['checkout', 'payments'], ['received order', 'card declined'])
metric alone tells us: 1 error(s), for which request? unknown

In production A durable business event survives trace sampling

PythonA span event does not survive sampling; a written record doesruns
import json
import random
from pathlib import Path

random.seed(7)

AUDIT = Path("orders_audit.jsonl")
AUDIT.write_text("")  # fresh file, our durable event store


class FakeSpan:
    """Stand-in for an OpenTelemetry span: events live inside it."""

    def __init__(self, trace_id, sampled):
        self.trace_id = trace_id
        self.sampled = sampled
        self.events = []

    def add_event(self, name, **attrs):
        self.events.append({"name": name, **attrs})


def record_order_placed(trace_id, order_id, amount, sampled):
    span = FakeSpan(trace_id, sampled)
    span.add_event("order.placed", order_id=order_id, amount=amount)  # span event: rides with the trace

    with AUDIT.open("a") as f:  # durable event: independent of the trace's fate
        f.write(json.dumps({"trace_id": trace_id, "order_id": order_id, "amount": amount}) + "\n")

    if not span.sampled:
        span.events.clear()  # the backend never received this trace; its events are gone
    return span


kept = record_order_placed("T1", "O-100", 4999, sampled=True)
dropped = record_order_placed("T2", "O-101", 1299, sampled=False)

print("kept trace still has its span event:", len(kept.events) == 1)
print("dropped trace lost its span event:", len(dropped.events) == 0)
print("audit log has both orders regardless of sampling:",
      len(AUDIT.read_text().strip().splitlines()) == 2)
Output
kept trace still has its span event: True
dropped trace lost its span event: True
audit log has both orders regardless of sampling: True

See it happen What dies with the trace, and what doesn't

trace_id=T1propagate trace_idclient requestcheckout spantrace_id=T1payments span(child of svcA)log line(trace_id=T1)errors_total(no trace_id)order.placed(durable store)
  1. The request arrives and the edge generates trace_id=T1; nothing has been recorded yet.
  2. checkout starts a span and writes one log line stamped with trace_id=T1.
  3. checkout also increments errors_total, a plain number with no trace_id attached.
  4. trace_id propagates to payments, which starts a child span under svcA.
  5. payments writes order.placed to a durable event store, independent of the trace.
  6. If this trace is later sampled out, svcA and svcB disappear together with the log's trace_id context, but the durable event is untouched.
Notice: three of these four nodes die the moment the trace is sampled away; only the node written to its own durable store is immune to a sampling decision made somewhere else entirely.
Scenario A

In your world

The situation

Your ETL job writes log.info("loaded batch") lines to stdout, which Jenkins captures as console output. When finance asks "did settlement batch #4821 actually post," you have no way to answer except reading an entire day's console log top to bottom.

What actually breaks

Jenkins truncates console output past a size limit and recycles old build logs after 30 days. Three weeks later, the batch in question is gone, and the only trace of it is "job succeeded" in the build history, with no connection to which rows it touched.

What you'd do on Monday

Give every run a correlation id generated at the start, log it on every line, and additionally write one durable event per run to a store outside Jenkins - a file or table with batch id, correlation id, row count and status that outlives build-log retention entirely.

PythonCorrelation id in every log line, plus one durable event rowruns
import itertools
import json
import logging
import sys
from pathlib import Path

logging.basicConfig(level=logging.INFO, format="%(message)s", stream=sys.stdout)
log = logging.getLogger("settlements")
EVENTS = Path("batch_events.jsonl")
_next_cid = itertools.count(1)


def run_batch(rows):
    cid = f"batch-{next(_next_cid)}"
    log.info(json.dumps({"cid": cid, "msg": "batch started", "rows": len(rows)}))
    posted = sum(1 for r in rows if r["amount"] > 0)
    log.info(json.dumps({"cid": cid, "msg": "batch finished", "posted": posted}))
    with EVENTS.open("a") as f:
        f.write(json.dumps({"cid": cid, "posted": posted, "status": "done"}) + "\n")
    return cid


cid = run_batch([{"amount": 250}, {"amount": -10}, {"amount": 90}])
print("durable record for", cid, "exists:", cid in EVENTS.read_text())
Output
{"cid": "batch-1", "msg": "batch started", "rows": 3}
{"cid": "batch-1", "msg": "batch finished", "posted": 2}
durable record for batch-1 exists: True
Scenario B

Imagine you work at CRED

2M bill-payment requests/day across 40 microservices

The numbers

Illustrative: 2M requests/day ≈ 23/sec on average, each touching about 6 services end to end, so roughly 12M span-equivalents a day. At that count there is no "read all the logs"; the only way to find one request's path is correlation, not volume.

What the Scenario-A answer does here

The Scenario A fix (one correlation id per run) still works for a batch job, but across 40 services in a synchronous call chain, "each service mints its own id" quietly breaks it: a service that doesn't read the incoming id and generates a fresh one severs correlation for every hop downstream of it, and nobody notices until an incident needs the full path.

What flips at this scale

At this scale, trace-id propagation has to become a cross-service contract enforced at the platform layer, not a convention each team remembers - middleware that rejects or flags internal requests missing the propagated id - because one unpropagating service breaks correlation for every request that passes through it, not just its own.

The trade-off you'd defend

I would defend mandating header propagation as a shared middleware requirement over "ask every team to remember," trading some standardization work for the ability to answer "what happened to this one request" at all once you're past a handful of services.

illustrative figures to make the arithmetic concrete; not CRED-published.

Gotchas

Correlation id missing on some log lines

Symptom

half a request's log lines carry the id and half don't, usually from a background thread or an async task spawned mid-request.

Cause

the id lives in a context var or thread-local that a new thread or task doesn't automatically inherit.

Fix

propagate the id explicitly across any thread/task boundary instead of relying on ambient context alone.

A span event mistaken for an audit trail

Symptom

compliance asks for every refund issued last quarter, and the tracing backend's 7-day retention has nothing.

Cause

span events live and die with the trace that carries them.

Fix

write anything someone will ask about later to a durable, independently-retained event store from day one.

Two different trace ids for one request

Symptom

logs show id T1 upstream and T2 downstream for what was clearly a single user action.

Cause

a downstream service didn't read the incoming trace header and minted its own id instead.

Fix

enforce header propagation, and fail loudly in local testing if the incoming traceparent isn't honoured.

Tricky questions

Isn't a span event basically a log line?
Most people say

yes, same thing with extra metadata attached.

Actually

a log line exists independent of any trace and can have its own retention; a span event is physically part of the span and disappears if that trace isn't sampled or kept. Treat them as different storage lifetimes, not different formats of the same fact.

If I have detailed logs, do I even need traces?
Most people say

no, grep is enough once logs are detailed.

Actually

logs tell you what happened per line, not causality across services. Without a shared id you cannot order events from different services into one request's actual path, especially under concurrency where timestamps alone lie.

Scenario question

You're handed a service that logs plenty but every incident takes 40+ minutes because nobody can tell which of 15 log lines across 5 services belong to the slow request a customer reported. What's your first change, and what do you explicitly not build yet?

Model answer

I would add one thing first: a correlation id generated at the edge and propagated through every service boundary and into every log line, because that turns "read everything" into "filter by one id." I would not reach for a full tracing backend yet - the id-plus-structured-logs change is a day of middleware work and immediately cuts search time, while standing up a trace collector is a bigger investment I'd justify once log-filtering alone genuinely isn't enough, for example once I need per-service timing, not just an ordered list of what happened.

Interview corner

CommonWhat's the difference between logs, metrics, and traces?
“I'd say logs are discrete per-event records, metrics are aggregated numbers over time that are cheap at any volume, and traces are causally-linked spans showing one request's path across services. I pick the tool by the question: how bad right now is a metric, what happened to this one request is a trace plus logs filtered by its id.”
TrickIf my span has events on it, do I still need a separate audit table?
“I'd say yes, if anyone will ask about this specific fact later, because span events live and die with the trace's own sampling and retention, and they were never meant to be a system of record.”
Experience checkWhere have correlation ids actually fixed a real problem for you?
“On a batch job, I added a per-run id, logged it everywhere, and wrote a one-row durable event per run. Incident triage dropped from reading a whole day's console log to filtering on one id.”
DesignYou're adding observability to a new service. What goes in first?
“Structured logs carrying a correlation id, plus a couple of RED-style metrics at the service boundary. I'd add tracing once there's more than two or three services in the request path, because that's when 'what happened to this one request' stops being answerable from logs alone.”

Memory hook

A patient's routing slip ties every room visit to one id; throw away the slip and you can still find the chart, but not the story.

If you remember only one thing:

Pick the tool by the question you're asking, and never let a fact someone will ask about later live only inside a trace.

Self-check

  1. What's the one thing a trace gives you that logs alone don't?

    Causal ordering of one request's path across multiple services, not just a timestamp per line.

  2. Why can't you rely on a span event as an audit record?

    It's stored inside the span and disappears if that trace is sampled out or ages past trace retention.

  3. Why don't metrics usually carry a trace id?

    A unique value per request as a label would blow up cardinality (Choose Prometheus metric types and useful dashboards); metrics are meant to be aggregated across many requests, not keyed to one.

Prove it

Pick a script you run regularly. Add a correlation id generated once per run, log it on every line, and write one JSON line per run to a runs.jsonl file with the id, status and a count. Run it three times, then paste a filter that extracts the full log for exactly one run by its id, plus the matching line from runs.jsonl.

ce-09-02Card 2 of 6

Add structured logs and Python OpenTelemetry

MediumP3 · Role-dependentTalk about it11 min
loggingstructured-logsopentelemetrypythontracinginstrumentation

Why you care

Half the Python services at a TCS-style shop log with print() or bare logging.info("done"), no severity discipline, no correlation id, no machine-parseable shape. The moment you need "every ERROR for order 884213 in the last hour" across an aggregator, unstructured text means regex archaeology. Interviewers ask about structured logging and OTel setup because it's the exact gap between "the job prints a lot" and "the job is operable by someone who isn't you at 2am."

Explain like I'm 5

Imagine two incident reports. One is a paragraph of prose with no headings. The other is a form: Date, Severity, Service, Error Code, free-text note. The form is slower to glance at casually, but a computer can file a thousand of them by Error Code in a second; the paragraph needs a human to read every single one. Structured logging is choosing the form over the paragraph, every time, even though you're writing it for a machine first and a human second.

Core concept

  • Structured log = one record per line, usually JSON, with fixed fields: severity, service name, message, a correlation/trace id, and whatever business context matters (order_id, batch_id) - never free text where a field would do.
  • Python's logging module supports this today with no extra package: a custom Formatter turns every record into one JSON line; extra={...} attaches fields to a specific call without polluting the message string.
  • OpenTelemetry tracing, separately: a TracerProvider holds global tracing configuration; a BatchSpanProcessor wraps an exporter and ships finished spans in batches instead of one network call per span; tracer.start_as_current_span("name") opens a span as a context manager and makes it "current," so nested calls become child spans automatically; set_attribute, add_event, record_exception and set_status(Status(StatusCode.ERROR)) enrich it.
  • Context propagation (W3C Trace Context, the default) carries the trace id across process boundaries via HTTP headers when you use an instrumented client/server - that trace id is the correlation id from Observability model: logs, metrics, traces and events, so put it in your log formatter too.
  • Decision rule: structured logging is near-free, an afternoon, no new infra, and belongs in every service on day one. Full OTel tracing earns its setup cost once you have more than one or two hops in a request path, or an on-call rotation that needs "what did this request touch." Don't add tracing before structured logs - without correlation ids already in your logs, the trace backend becomes the only place anyone can search, which is a worse single point of failure than before.
  • Exclude secrets/PII from both: never put a password, token, card number or raw customer PII into a log field or a span attribute, since both tend to land in systems with weaker access control than your database. Redact at the point of logging, not after.
  • Avoid duplicate instrumentation: don't hand-wrap every outbound call in your own span and install the matching auto-instrumentation library; you'll get two spans for one call and a confusing trace. Let the auto-instrumentor own what it already covers; add manual spans only for your own business logic.

Under the hood

BatchSpanProcessor does exactly what its name says: finished spans sit in an in-memory queue and ship to the exporter on a timer, every few seconds by default, or once the queue hits a size limit - not the instant a span ends. That's good for throughput, since it batches network calls, but it means a process that exits abruptly - a crashed container, a kill -9, a serverless runtime frozen between invocations - can lose the last batch of spans that never flushed. The same shape of bug exists in Python's own buffered log handlers. The fix in both cases is identical: call shutdown() (on the TracerProvider, or logging.shutdown()) during graceful shutdown so the last batch flushes before the process exits, and don't trust observability data from the final seconds of an unclean exit.

Minimal example A JSON formatter turns records into filterable lines

PythonNo new package, just a Formatterruns
import json
import logging
import sys


class JsonFormatter(logging.Formatter):
    def format(self, record):
        payload = {
            "level": record.levelname,
            "service": "settlements-job",
            "msg": record.getMessage(),
        }
        if hasattr(record, "order_id"):
            payload["order_id"] = record.order_id
        return json.dumps(payload, sort_keys=True)


logger = logging.getLogger("demo")
logger.setLevel(logging.INFO)
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
logger.addHandler(handler)
logger.propagate = False

logger.info("order rejected", extra={"order_id": "O-9"})
logger.error("downstream timeout")
Output
{"level": "INFO", "msg": "order rejected", "order_id": "O-9", "service": "settlements-job"}
{"level": "ERROR", "msg": "downstream timeout", "service": "settlements-job"}

In production OpenTelemetry tracing setup (needs the opentelemetry-sdk package)

PythonNeeds opentelemetry-api and opentelemetry-sdk installed
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry.trace import Status, StatusCode

provider = TracerProvider(resource=Resource.create({"service.name": "settlements-job"}))
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("settlements")


def post_batch(batch_id, rows):
    with tracer.start_as_current_span("post_batch") as span:
        span.set_attribute("batch.id", batch_id)
        span.set_attribute("batch.rows", len(rows))
        try:
            posted = sum(1 for r in rows if r["amount"] > 0)
            span.add_event("batch.posted", {"posted": posted})
            return posted
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, str(exc)))
            raise


post_batch("B-4821", [{"amount": 250}, {"amount": -10}])
provider.shutdown()  # flush the BatchSpanProcessor before the process exits
Scenario A

In your world

The situation

Your nightly job logs with print(f"Loaded {n} rows") and errors with print(f"ERROR: {e}"), piped to a file Jenkins archives.

What actually breaks

A supplier complains three invoices were double-processed. You grep 40 build artifacts for "invoice" and find nothing useful, because the print statements never included the invoice id or batch id - just row counts.

What you'd do on Monday

Convert to structured logs with a JSON formatter, and always log the business key you'll be asked about later, not just a count.

PythonPrints become structured, searchable recordsruns
import json
import logging
import sys

logging.basicConfig(level=logging.INFO, format="%(message)s", stream=sys.stdout)
log = logging.getLogger("invoices")


def process_invoice(invoice_id, amount, seen):
    if invoice_id in seen:
        log.error(json.dumps({"event": "duplicate_invoice", "invoice_id": invoice_id}))
        return False
    seen.add(invoice_id)
    log.info(json.dumps({"event": "invoice_posted", "invoice_id": invoice_id, "amount": amount}))
    return True


seen = set()
for inv in [("INV-1", 500), ("INV-2", 300), ("INV-1", 500)]:
    process_invoice(*inv, seen)
Output
{"event": "invoice_posted", "invoice_id": "INV-1", "amount": 500}
{"event": "invoice_posted", "invoice_id": "INV-2", "amount": 300}
{"event": "duplicate_invoice", "invoice_id": "INV-1"}
Scenario B

Imagine you work at Urban Company

6,000 service bookings/hour across 18 microservices

The numbers

Illustrative: 6,000/hour ≈ 1.7/sec on average, spiking to roughly 10x at peak hours, so about 17 bookings/sec; each booking touches around 6 of the 18 services, so peak load is roughly 100 spans/sec just for one booking's path.

What the Scenario-A answer does here

The Scenario A fix (per-service JSON logs with a shared correlation id) still finds one booking's full trail, but "every team instruments manually" doesn't scale across 18 teams: inconsistent field names (order_id vs booking_id vs bookingId), duplicate spans where a team wraps a call the auto-instrumentor already covers, and no consistent service.name resource attribute, so one trace shows 18 different naming conventions instead of a coherent path.

What flips at this scale

At this scale, instrumentation becomes a platform concern, not a per-team choice: a shared logging/tracing wrapper that fixes field names, auto-attaches service.name, and ships via auto-instrumentation where available, with manual spans reserved for business logic the auto-instrumentor can't see.

The trade-off you'd defend

I would defend a mandated shared instrumentation wrapper over "every team configures OTel themselves," trading some team autonomy for traces and logs that are actually joinable across all 18 services, which is the entire point of doing this.

illustrative figures; not Urban Company-published.

Gotchas

Spans never show up in the backend

Symptom

no error, nothing appears in the trace UI after a deploy or on process exit.

Cause

BatchSpanProcessor buffers in memory and the process exited before the flush interval or shutdown() ran.

Fix

call provider.shutdown() during graceful shutdown, as covered in the hood.

Same call shows two spans

Symptom

a trace shows two nearly-identical child spans for one outbound HTTP call.

Cause

both a manual span wrapper and an auto-instrumentation library wrapped the same call.

Fix

pick one layer; keep manual spans only for logic the auto-instrumentor can't see.

A secret shows up in a trace attribute

Symptom

a security review finds a raw request body, including auth headers, attached as a span attribute.

Cause

attributes and log fields get copied wholesale from objects without filtering.

Fix

allowlist which fields get logged or attached; never attach a raw request/response object.

Tricky questions

Does logger.info(f"order {order_id} posted") count as structured logging?
Most people say

yes, the id is right there in the message.

Actually

no - the id is baked into a string a human reads, not a field a machine can filter on without a regex. Structured means order_id is its own key, independent of how the English wording changes later.

Is tracer.start_as_current_span the same as just creating a span?
Most people say

yes, same thing with a longer method name.

Actually

start_as_current_span also makes the span "current" in context, so any nested call that asks for the active span, your own code or an auto-instrumentor, finds it and attaches as a child automatically. Creating a span without entering it as current means nested spans won't nest.

Scenario question

You're handed a service with scattered plain-string logging.info calls and two hours before a demo to make tomorrow's incident response less painful. What do you actually do in two hours?

Model answer

I would not install OpenTelemetry tracing in two hours; that's a bigger, riskier change right before a demo. I'd add a JSON formatter to the existing logging config, which touches no call sites, and add one correlation id generated per request or job, passed via extra=, updating only the handful of log calls that matter most, like entry/exit and error paths. That gets filterable, correlated logs by end of day, and I'd schedule tracing as next week's work once I'm not touching anything right before a demo.

Interview corner

CommonHow do you add structured logging to an existing service without a big rewrite?
“I'd swap in a JSON Formatter on the existing logging config, which touches zero call sites, then add a correlation id via extra= at the handful of places that matter, like request entry and error handling, before trying to restructure every log call.”
TrickYou set up OpenTelemetry tracing but spans don't appear after the process exits. What's wrong?
“I'd check whether provider.shutdown() ran before exit, because BatchSpanProcessor buffers in memory and only flushes on a timer or on shutdown; an abrupt exit loses whatever hadn't flushed yet.”
Experience checkWhat do you deliberately keep out of logs and span attributes?
“Secrets, tokens, full card numbers and raw request/response bodies. I allowlist fields instead of logging objects wholesale, because both logs and traces tend to end up in systems with weaker access control than the database.”
DesignA team wants both their own spans and the auto-instrumentation library. How do you avoid duplicate spans?
“I'd let the auto-instrumentor own anything it already covers, like outbound HTTP or DB calls, and reserve manual spans for the team's own business logic, rather than wrapping the same call twice.”

Memory hook

A paragraph needs a reader; a form needs a filter - write for the filter.

If you remember only one thing:

If a value matters for a search later, it's a field, not a word inside the message string.

Self-check

  1. Why doesn't logger.info(f"order {oid} posted") count as structured logging?

    The id is embedded in free text, not an independent, machine-filterable field.

  2. Why can a process crash lose spans that already finished?

    BatchSpanProcessor queues them in memory and ships on a timer or on shutdown, not instantly.

  3. When should you skip manual spans for an HTTP call?

    When an auto-instrumentation library already wraps that call, to avoid two spans for one request.

Prove it

Take one real script of yours. Add a JSON log formatter and a correlation id passed through extra= on every log call. Run it twice with different inputs and paste the two sets of JSON lines, then paste a one-line filter that pulls out only one run's lines by its correlation id.

ce-09-03Card 3 of 6

Choose Prometheus metric types and useful dashboards

MediumP3 · Role-dependentTalk about it11 min
prometheusmetricsdashboardscardinalityhistograms

Why you care

Every candidate says "we used Prometheus and Grafana." Few can answer "why a histogram instead of a summary for that metric" or "why did adding a user_id label take down the metrics backend." Interviewers ask about metric types and cardinality because the wrong choice is the most common way a well-intentioned monitoring setup becomes an outage of its own: an unbounded number of stored series, or a dashboard with 40 panels nobody checks until it's too late to matter.

Explain like I'm 5

A counter is a car's odometer: it only goes up, or resets to zero with a new car, and you care about distance covered between two readings, not the raw number. A gauge is the fuel gauge: it goes up and down, and the current reading is the whole story. A histogram is a tollbooth sorting cars into lanes by speed - "0-60," "60-80," "80+" - and later you can ask "what fraction went over 80" from the lane counts, without ever recording one car's exact speed. A summary is one driver's own clipboard, where they personally wrote a running percentile of their own speed; you can't average two drivers' clipboards into a fleet-wide percentile.

Core concept

  • Counter: monotonically increasing, resets only on process restart. Never read the raw value; always rate() or increase() over it. Use for request counts, error counts, rows processed.
  • Gauge: current value, moves up and down. Use for queue depth, in-flight requests, memory used, open connections.
  • Histogram: observations sorted into configurable buckets (le="0.1", le="0.5", ...) plus _sum and _count. histogram_quantile() estimates a percentile from the bucket counts server-side, and because the raw buckets are plain counters, histograms aggregate correctly across many instances with sum(). Bucket boundaries must match your actual latency distribution and SLO - a histogram bucketed at 0.1s/0.5s/1s is useless for a service with a 2-second SLO.
  • Summary: computes quantiles client-side over a sliding window and exposes them directly. Cheaper per sample, but those quantiles cannot be meaningfully averaged or summed across instances; a p99 from server A and a p99 from server B do not combine into a fleet p99. Prefer a histogram for anything you'll aggregate across more than one instance.
  • Cardinality: a Prometheus series is uniquely identified by metric name plus the full set of label values. A label with high-cardinality values - a user id, an order id, a raw path with an embedded id - multiplies stored series by every distinct value ever seen, and that growth is one-directional until old series age out of retention. This is the single most common way to overload a metrics backend.
  • Dashboards: build around traffic, errors, latency and saturation (the "golden signals" from the Google SRE book), not every metric you happen to have. Mark deploy times on the axis so a latency bump is checked against "did we just ship something" first. A panel nobody uses to make a decision during an incident is a vanity panel - delete it or merge it.

Minimal example Counter rate vs raw value, and why a bad label explodes series count

PythonA counter's own value is almost never the number you wantruns
samples = [(0, 120), (15, 180), (30, 260)]  # (seconds, cumulative request_count)

def rate_per_sec(samples):
    (t0, v0), (t1, v1) = samples[0], samples[-1]
    return (v1 - v0) / (t1 - t0)

print("raw counter value:", samples[-1][1])
print("rate() over the window:", rate_per_sec(samples))

label_values_good = {"GET", "POST", "DELETE"}             # method: bounded
label_values_bad = {f"req-{i}" for i in range(1, 50_001)}  # request_id: unbounded
print("series with method label:", len(label_values_good))
print("series with request_id label:", len(label_values_bad))
Output
raw counter value: 260
rate() over the window: 4.666666666666667
series with method label: 3
series with request_id label: 50000

In production A tiny histogram - bucket observations, then estimate a percentile

This deliberately picks buckets topping out at 2 seconds; watch what the p95 estimate becomes once the tail breaches that top bucket.

PythonBuckets matched to a 2s SLO, and what happens when the tail breaches themruns
from bisect import bisect_right

BUCKETS = [0.1, 0.25, 0.5, 1.0, 2.0, float("inf")]  # seconds


class LatencyHistogram:
    """Stand-in for a Prometheus histogram: counts observations per bucket."""

    def __init__(self, buckets):
        self.buckets = buckets
        self.counts = [0] * len(buckets)
        self.total = 0

    def observe(self, value):
        idx = bisect_right(self.buckets, value)
        self.counts[min(idx, len(self.buckets) - 1)] += 1
        self.total += 1

    def quantile(self, q):
        target = q * self.total
        cumulative = 0
        for bound, count in zip(self.buckets, self.counts):
            cumulative += count
            if cumulative >= target:
                return bound
        return self.buckets[-1]


checkout_latency = LatencyHistogram(BUCKETS)
for value in [0.05, 0.08, 0.3, 0.4, 0.9, 1.8, 0.2, 0.15, 2.5, 0.6]:
    checkout_latency.observe(value)

print("requests observed:", checkout_latency.total)
print("bucket counts:", checkout_latency.counts)
print("approx p95 bucket boundary:", checkout_latency.quantile(0.95))
Output
requests observed: 10
bucket counts: [2, 2, 2, 2, 1, 1]
approx p95 bucket boundary: inf
Scenario A

In your world

The situation

Your team added request_latency_seconds as a Prometheus histogram and a dashboard panel, but someone also added .labels(user_id=user_id, order_id=order_id) on it "for debugging," after last month's incident needed to find one slow order.

What actually breaks

Within two weeks, the Prometheus server's memory climbs steadily and queries start timing out, because request_latency_seconds alone has created hundreds of thousands of distinct series - every new user_id/order_id pair is a brand-new series that doesn't disappear until it ages out of retention.

What you'd do on Monday

Remove the high-cardinality labels; keep the metric bounded to route, method and status code, and get "find the one slow order" from a trace or a correlation id in logs instead, which is exactly what those tools are for.

PythonCounting distinct label combinations before you ship themruns
def series_count(label_sets):
    from itertools import product
    return len(list(product(*label_sets)))

bounded = [["GET", "POST"], ["200", "500"], ["/orders", "/payments"]]
unbounded = [["GET", "POST"], ["200", "500"], [f"order-{i}" for i in range(1, 10_001)]]

print("series with route+method+status:", series_count(bounded))
print("series if order_id becomes a label:", series_count(unbounded))
Output
series with route+method+status: 8
series if order_id becomes a label: 40000
Scenario B

Imagine you work at Grafana Labs

300 internal services, 180,000 active time series

The numbers

Illustrative: 300 services × ~40 metric names × ~15 label combinations average ≈ 180,000 active series already. Adding one unbounded label to just 10 of those services, at roughly 5,000 unique values each, adds 10 × 5,000 = 50,000 more series overnight - a 28% jump from one bad label change in one afternoon.

What the Scenario-A answer does here

Scenario A's fix (strip the bad label, use traces for "find one order") still works per service, but "watch dashboards manually for cardinality creep" doesn't scale across 300 teams - someone adds a bad label every month, and nobody notices until the shared metrics backend itself is slow for everyone.

What flips at this scale

At this scale, cardinality needs an enforced budget, not a review: a per-service series-count limit checked in CI or at the metrics gateway, which rejects or drops any metric that would push a service over budget, so one team's mistake can't degrade the shared backend for everyone else.

The trade-off you'd defend

I would defend a hard, automatically-enforced per-service cardinality limit over "ask teams to be careful," trading the occasional friction of blocking a legitimate new label against the alternative of the whole metrics platform degrading because of one team's debug label.

illustrative figures; not Grafana Labs-published.

Counter vs gauge vs histogram vs summary

TypeDirectionTypical queryCross-instance aggregationCost per sample
Counterup only, resets on restartrate() / increase()sum() works correctlylowest
Gaugeup and downraw value, avg()/max()works, meaning depends on the metriclowest
Histogrambucketed countshistogram_quantile() over sum() of bucketsaggregates correctlylow-medium (N buckets/sample)
Summaryprecomputed quantilesread the quantile directlycannot be combined across instancesmedium (quantile math/sample)

Gotchas

Prometheus server memory climbs, queries time out

Symptom

steady memory growth and slow queries after a routine metric change.

Cause

a label with unbounded values, multiplying stored series.

Fix

audit labels for anything that scales with requests, users or orders; move identifiers to logs/traces instead.

A "fleet-wide p99" looks wrong after aggregating

Symptom

averaging each instance's summary-reported p99 doesn't match reality.

Cause

summary quantiles are computed per instance and can't be mathematically combined.

Fix

use a histogram and histogram_quantile(0.99, sum(rate(...)) by (le)) instead.

histogram_quantile returns +Inf

Symptom

a percentile query comes back as +Inf instead of a number.

Cause

too many observations landed above the highest finite bucket boundary, so the estimate can't be bounded.

Fix

add a higher bucket, or re-check whether the SLO-aligned top bucket is actually too low for current traffic.

Tricky questions

Should I just read the counter's current value to see "how many requests we got"?
Most people say

yes, that's the request count.

Actually

the raw value only reflects the total since the last process restart. "Requests in the last 5 minutes" needs rate() or increase() over a window, because a raw reading mixes different epochs across restarts.

A summary gives percentiles directly, so it's strictly better than a histogram?
Most people say

yes, it's more precise and does the math for you.

Actually

that precision is per-instance only. Summaries can't be aggregated across replicas, so the moment you have more than one instance, which is almost always, a histogram is the right default and a summary is the exception.

Scenario question

You're handed a dashboard with 40 panels for one service, and told incident response still takes too long. What do you actually do?

Model answer

I would cut it down to the four signals that answer "is this service healthy right now": request rate, error rate, latency as a histogram-backed percentile (not an average), and saturation, like queue depth or CPU/memory against its limit. I'd add a deploy-time annotation on the axis so a bump gets checked against "did we just ship" first. Everything else, the other 36 panels, I'd fold into a drill-down dashboard opened only after the first four flag something, or delete, because a panel nobody uses during an incident is actively slowing down the person trying to find the one that matters.

Interview corner

CommonWhen would you use a histogram instead of a summary?
“Almost always, because histograms aggregate correctly across instances with histogram_quantile() and sum(), while summary quantiles are per-instance and can't be combined. I'd reach for a summary only if I truly have one process and will never need a fleet-wide number.”
TrickWhat's the most common way a Prometheus setup takes itself down?
“A high-cardinality label, usually a user id, order id or raw path with an id in it, that multiplies stored series without bound. I check every label's possible values are drawn from a small, fixed set before shipping it.”
Experience checkHow do you decide what goes on an on-call dashboard?
“I start from rate, errors, latency and saturation, add a deploy marker on the axis, and resist adding more panels unless they changed an actual decision in a past incident. Vanity panels slow down triage.”
DesignYour histogram's p95 keeps returning +Inf. What do you check?
“Whether too many observations are landing above my highest finite bucket, usually because the bucket boundaries were copied from an example instead of matched to this service's actual latency and SLO. I'd add a higher bucket and re-check the distribution.”

Memory hook

A counter is an odometer, a gauge is the fuel needle, a histogram is tollbooth lanes by speed, a summary is one driver's own notebook - don't average notebooks.

If you remember only one thing:

If a label's possible values scale with requests, users or orders, it doesn't belong on a metric.

Self-check

  1. Why call rate() on a counter instead of reading its raw value?

    The raw value only reflects the total since the last process restart; rate()/increase() give the actual change over a time window.

  2. Why can't you average two servers' summary-reported p99s into a fleet p99?

    Summary quantiles are computed independently per instance over its own samples; there's no way to merge them after the fact.

  3. What does a +Inf result from histogram_quantile() usually mean?

    Enough observations landed above the highest finite bucket that the quantile can't be bounded - the buckets don't match the real distribution.

Prove it

Simulate 500 latency observations (mostly fast, a few slow outliers) into the LatencyHistogram class above, using two different bucket sets - one that tops out below your slowest outliers and one that doesn't. Print both p95 estimates and show the difference it makes.

ce-09-04Card 4 of 6

SLOs and incident signals: RED/USE and error budgets

HardP3 · Role-dependentTalk about it13 min
sloslierror-budgetred-methoduse-methodburn-ratealerting

Why you care

"What's the difference between an SLO and an alert threshold" and "why did your team get paged for something that didn't matter" are two sides of the same question, and they test whether you understand that alerting exists to protect a budget, not to catch every blip. An engineer who pages on every error-rate spike regardless of volume burns out their own on-call rotation and trains everyone to ignore pages, which is worse than no alerting at all - the one page that actually matters gets ignored too.

Explain like I'm 5

Think of an error budget as a monthly data allowance, not a strict daily limit. Burning 1% of a 30-day allowance in one hour, a huge, fast rate, means you'll be out of data by evening - that deserves an immediate, stop-what-you're-doing alert. Burning the same 1% spread evenly over three days is a completely different, far less urgent story, even though the total amount used is identical. RED and USE are two different ways of deciding what's worth watching in the first place: RED watches the requests flowing through your service from outside; USE watches the resources your service sits on from inside.

Core concept

  • RED (request-centric): Rate (requests/sec), Errors (rate of failing requests), Duration (latency distribution, as a histogram, never an average). Apply RED per service, and per endpoint once traffic is high enough to make that meaningful.
  • USE (resource-centric): for every resource - CPU, memory, disk, network, a connection pool, a queue - ask Utilization (how busy, % of time doing useful work), Saturation (how much queued work it can't service right now), Errors (error events on that resource). Walk every resource systematically rather than starting from whatever metric happens to already exist.
  • Decision rule: RED tells you if users are hurting right now; USE tells you why, by pointing at the specific saturated resource. A page should come from RED, the user-facing symptom; USE metrics are what you check next to diagnose, almost never what pages you directly - "CPU at 85%" is not automatically a user problem.
  • SLI is the measured ratio, usually good events over total events ("fraction of requests under 300ms," "fraction of batches that finished on time"). SLO is the target for that ratio over a window ("99.9% of requests under 300ms over 30 days"). Error budget is 1 - SLO worth of allowed bad events over that window; once reliability is framed as a budget, every reliability-vs-velocity argument becomes a number instead of a feeling.
  • Burn rate is how fast you're consuming that budget relative to a constant, even pace. A burn rate of 1 means using the budget evenly across the window - exactly on pace to land at the SLO. A burn rate of 14.4 sustained for an hour means you'd exhaust a 30-day budget in about two days - that's a page. A burn rate of 1 sustained for days is real, but not urgent enough to wake someone - a ticket, not a page.
  • Multi-window alerting (a short window and a long window both confirming) cuts false pages from brief blips while still catching genuine fast burns, which is why a single-window fixed threshold is usually either too noisy or too slow.
  • Freshness/completeness for ETL: the same SLI/SLO framing applies to batch jobs, not just request/response services - "fraction of nightly loads that completed before 6am" (freshness) and "fraction of expected source rows actually present" (completeness) are SLIs with their own budgets, measured in late or missing runs instead of failed requests.
  • Low-traffic caveat: with few requests, one failure is a large fraction of a small sample and burn-rate math gets noisy; mitigations include aggregating related services, adding synthetic traffic, or accepting that a strict SLO doesn't fit a low-volume path and a simpler threshold alert is more honest.

Under the hood

The reason multi-window burn-rate alerting beats a flat "alert if error rate > 5%" threshold is a sampling problem, not a tuning problem: at low-to-moderate traffic, a short error spike (one bad deploy, one database failover) looks statistically identical in a 5-minute window to steady background noise unless a longer window, say an hour, is also still elevated. A single short window alone fires on genuine but brief, self-healing blips; a single long window alone takes too long to catch a fast, severe burn before real budget is gone. Requiring both windows to agree is a cheap, two-line way to approximate "is this still happening and is it bad enough to matter," without building a proper statistical change-point detector - and it's why recommended thresholds pair a short confirmation window at roughly 1/12th the length of the long window: short enough to reset quickly once the problem clears, long enough that one noisy minute doesn't trigger it alone.

Minimal example Burn rate turns an error ratio into "how fast is the budget going"

PythonSame error ratio, very different urgency depending on the windowruns
def burn_rate(error_ratio, slo_target):
    allowed_bad_fraction = 1 - slo_target
    return error_ratio / allowed_bad_fraction


slo = 0.999  # 99.9% over 30 days
for window, error_ratio in [("5 min", 0.10), ("1 hour", 0.01), ("6 hours", 0.001)]:
    rate = burn_rate(error_ratio, slo)
    page = rate >= 14.4
    print(window, "burn rate:", round(rate, 1), "page?", page)
Output
5 min burn rate: 100.0 page? True
1 hour burn rate: 10.0 page? False
6 hours burn rate: 1.0 page? False

In production RED from a request log, USE from a worker queue

PythonRate/errors/duration, and utilization/saturation, from records you already haveruns
import math

requests = [
    {"status": 200, "duration_ms": 80}, {"status": 200, "duration_ms": 95},
    {"status": 500, "duration_ms": 40}, {"status": 200, "duration_ms": 110},
    {"status": 200, "duration_ms": 2200}, {"status": 200, "duration_ms": 90},
]
window_seconds = 60


def p95(values):
    s = sorted(values)
    idx = max(0, math.ceil(0.95 * len(s)) - 1)
    return s[idx]


def red_summary(reqs, window_s):
    total = len(reqs)
    errors = sum(1 for r in reqs if r["status"] >= 500)
    return {
        "rate_per_sec": round(total / window_s, 2),
        "error_rate": round(errors / total, 3),
        "p95_ms": p95(r["duration_ms"] for r in reqs),
    }


queue_samples = [2, 3, 2, 8, 9, 9, 9]   # in-flight jobs sampled over time
queue_capacity = 10


def use_summary(samples, capacity):
    busy_samples = sum(1 for s in samples if s > 0)
    saturated_samples = sum(1 for s in samples if s >= capacity * 0.8)
    return {
        "utilization": round(busy_samples / len(samples), 2),
        "saturation": round(saturated_samples / len(samples), 2),
    }


print("RED:", red_summary(requests, window_seconds))
print("USE:", use_summary(queue_samples, queue_capacity))
Output
RED: {'rate_per_sec': 0.1, 'error_rate': 0.167, 'p95_ms': 2200}
USE: {'utilization': 1.0, 'saturation': 0.57}

See it happen The same total budget burned at two different speeds

budget: fast burn (14.4x)##########budget: slow burn (1x)#########
  1. Day 0: both services start with a full 30-day error budget, shown as 10 tokens.
  2. Hour 1 of a fast burn already spends 2 tokens; at this rate the budget is gone in about 2 days.
  3. The same hour of a slow, steady burn barely moves: roughly 1 token spent over the full 30 days.
  4. By hour 6, the fast burn has used 5 tokens, and the page fires because both the 5-minute and 1-hour windows agree it's still happening.
  5. The slow burn, even a full day later, is still only down one token: real, worth a ticket, not worth paging anyone at 3am.
Notice: the fast row and the slow row can lose the exact same total amount of budget; what makes one a page and the other a ticket is purely how fast it's disappearing.
Scenario A

In your world

The situation

Your team sets one alert: "page if error rate > 1% for 5 minutes" on the API that posts settlement files.

What actually breaks

A single flaky retry during a routine deploy pushes the 5-minute error rate to 1.2% for one window, and you get paged at 2am for something that resolves itself before you open your laptop. Weeks later, a real multi-hour outage starts slow, at 0.3%, and never crosses the fixed 1% line in any 5-minute window, so nobody gets paged until a customer complains.

What you'd do on Monday

Define an SLO and alert on burn rate confirmed across two windows, not a single fixed threshold.

PythonReplace a single threshold with a two-window burn-rate checkruns
def should_page(short_window_rate, long_window_rate, threshold=14.4):
    return short_window_rate >= threshold and long_window_rate >= threshold


slo = 0.995
allowed_bad = 1 - slo

cases = [
    ("flaky blip", 0.08, 0.001),     # short window spikes, long window barely moves
    ("real outage", 0.08, 0.08),     # both windows agree
]
for name, short_err, long_err in cases:
    short_rate = short_err / allowed_bad
    long_rate = long_err / allowed_bad
    print(name, "pages:", should_page(short_rate, long_rate))
Output
flaky blip pages: False
real outage pages: True
Scenario B

Imagine you work at DoorDash

300M order-status API calls over 30 days

The numbers

Illustrative: 300M calls over 30 days ≈ 10M/day ≈ 115/sec on average. A 99.9% SLO allows about 0.1% × 300M = 300,000 failed calls across the whole month before the budget is spent.

What the Scenario-A answer does here

Scenario A's two-window burn-rate check still catches a sudden regional outage, but a single global SLO can mask a localized failure: if one region, say 5% of traffic (15M calls), is at 100% failure while the other 95% is healthy, the blended global error rate is only around 5% - which may not clear either burn-rate threshold depending on how the budget is set, even though every customer in that region is fully down.

What flips at this scale

At this scale, SLOs need to be sliced by the dimension that matters to users - region, client platform, a specific endpoint - not just one global number, because a healthy average can hide a 100%-down slice. The flip is maintaining several SLOs and aggregating alerts, not metrics, across them.

The trade-off you'd defend

I would defend per-region SLOs with their own burn-rate alerts over one global SLO, trading more alerting configuration to maintain for catching localized full-outages that a blended number would hide.

illustrative figures; not DoorDash-published.

Gotchas

Paged for a blip that resolved itself

Symptom

on-call paged at 2am for a 5-minute error spike nobody can reproduce by the time they look.

Cause

single-window, fixed-threshold alerting with no confirmation that a longer window is also elevated.

Fix

require both a short and a long window to cross the burn-rate threshold before paging.

Real outage never paged anyone

Symptom

a multi-hour, moderate error rate never crossed a high fixed threshold, so a customer complaint caught it first.

Cause

the alert was tuned for sudden spikes, not sustained-but-moderate degradation.

Fix

configure at least two severities - a fast page threshold and a slower ticket threshold.

A blended SLO hides a fully-down slice

Symptom

global error rate looks fine while a specific region or platform is at 100% failure.

Cause

averaging across a dimension that matters to users.

Fix

slice SLOs by region/platform/endpoint and alert per slice, as in Scenario B.

Low-traffic endpoint pages constantly

Symptom

an internal endpoint with 20 requests/hour pages every time one request fails.

Cause

one failure is already 5% of a tiny sample, which blows past almost any burn-rate threshold.

Fix

aggregate it into a broader service SLO, add synthetic traffic, or use a simple fixed-count alert instead of a burn-rate SLO for that endpoint.

Tricky questions

Does a burn rate of 1 mean something's broken?
Most people say

yes, any sustained burn is an incident.

Actually

a burn rate of 1 means using the budget evenly across the window - by definition, still meeting the SLO if it stays there. It's informative, not an incident, and typically maps to a ticket-level alert, not a page.

Is RED just "rate, errors, duration" and USE the same three things for a server?
Most people say

yes, they're the same idea applied in two places.

Actually

they point in different directions. RED is measured from outside the service, at the request boundary, and tells you if users are affected. USE is measured from inside, per resource, and tells you why. A CPU at 90% utilization is not automatically a RED problem if latency and error rate are still fine.

Scenario question

You're handed a service with no SLO at all, just a pile of dashboards, and asked to set up paging before next week's on-call rotation starts. What's the smallest useful thing you ship?

Model answer

I'd pick one SLI that reflects actual user pain, like "fraction of requests under 500ms with a 2xx/3xx status," set a defensible SLO such as 99.5% over 30 days based on current baseline performance, not an arbitrary round number, and wire a two-window burn-rate alert at a page threshold for fast, severe burns and a ticket threshold for slow, sustained ones. I would not try to define SLOs for every endpoint by next week; one well-chosen, user-facing SLI beats five rushed, badly-scoped ones, and I can slice by region or endpoint later once I see where real incidents cluster.

Interview corner

CommonWhat's the difference between an SLI, an SLO, and an error budget?
“I'd say an SLI is the measured ratio, like fraction of requests under 300ms. An SLO is the target for that ratio over a window, like 99.9% over 30 days. The error budget is just one minus the SLO - the amount of badness you're allowed before you've broken your own promise.”
TrickYour error rate crosses your alert threshold for one 5-minute window. Do you page?
“Not on that alone. I'd check whether a longer window is also elevated, because a single short window can't tell a real, ongoing problem from a brief, self-healing blip. Multi-window burn-rate alerting is exactly that two-window agreement check.”
Experience checkWhere have you seen RED and USE used differently in an actual incident?
“RED told us users were seeing errors; USE, specifically a saturated connection pool, told us why. I've learned to always page off the user-facing RED signal and use USE metrics as the next diagnostic step, not as the page trigger itself.”
DesignHow would you set an SLO for a nightly ETL job that doesn't serve live requests?
“I'd define the SLI as freshness, the fraction of nightly loads completing before a deadline, and completeness, the fraction of expected source rows actually present, each with its own budget measured in late or short runs over a rolling window, instead of forcing a request/response-shaped SLO onto a batch job.”

Memory hook

A fast burn pages you because the budget runs out before the month does; a slow burn gets a ticket because it won't.

If you remember only one thing:

Alert on burn rate confirmed by two windows, not a single fixed threshold.

Self-check

  1. What does a burn rate of 14.4 over a 1-hour window roughly mean?

    At that pace you'd exhaust a 30-day error budget in about two days - fast enough to justify an immediate page.

  2. Why does multi-window alerting need both a short and a long window to agree?

    A short window alone fires on brief, self-healing blips; a long window alone is too slow to catch a fast, severe burn before real budget is gone.

  3. Why page off a RED metric and not a USE metric?

    RED is measured at the user-facing request boundary, so it reflects actual user impact; a resource being saturated doesn't necessarily mean users are affected yet.

Prove it

Log success/failure and duration for 200 synthetic calls with a deliberate 10-minute bad patch in the middle, then compute and print the 5-minute and 1-hour burn rate at the worst point and show which multi-window page threshold it crosses, if any.

ce-09-05Card 5 of 6

Respond to an incident and write a useful postmortem

HardP2 · Very likelyTalk about it12 min
incident-responsepostmortemon-callseveritytimeline

Why you care

Anyone can "fix the bug." What separates a senior engineer during an incident is whether the right people know what's happening while it's happening, and whether the postmortem produces an owned follow-up instead of a document nobody reads. Interviewers ask about incident response and postmortems because your first real 3am incident at a product company is a bigger test of judgment than any coding round, and a bad postmortem culture, blame or vague "process will be improved," is one of the fastest ways a team repeats the same outage twice.

Explain like I'm 5

Think of a kitchen fire. The moment you see smoke, you don't start debugging why the fryer failed - you get people out, say "there's a fire in the kitchen" loudly and clearly, and decide who calls for help versus who grabs an extinguisher. That's incident response: stop the damage, communicate honestly, in that order. The postmortem is what you write the next day: not "whose fault was it," but exactly what let the smoke happen at all, what would have told you sooner, and what you're actually changing so it can't happen quite that way again.

Core concept

  • Severity is decided by user impact, not by how interesting the bug is: a typo in an internal admin tool is not the same severity as checkout being down, even if the checkout bug is "simpler." Define severity levels in advance (Sev1: broad user-facing outage; Sev2: degraded or partial; Sev3: internal/low-impact) so nobody is guessing under pressure.
  • Roles: an incident commander coordinates and decides, and isn't necessarily the one typing the fix; someone communicates status on a cadence; whoever is fixing it, fixes it. One person should not try to be all three above a Sev3 - that's how updates stop happening while someone goes heads-down.
  • Mitigation before root cause: the first move reduces user impact - rollback, feature flag off, failover, rate-limit - using the signals from SLOs and incident signals: RED/USE and error budgets to confirm it's working, not to find out exactly why it broke. Root-causing can happen in parallel or after; a mitigated incident with an unknown cause beats an ongoing incident with a known one.
  • Timeline and update cadence: write timestamps as you go (detected, mitigated, resolved), and commit to a visible cadence, even "no change, still investigating, next update in 30 min," so stakeholders stop pinging the person actually fixing it.
  • Postmortem is written after resolution and records: impact (duration, users/requests/rows affected), a timeline, contributing conditions (plural - almost nothing fails for exactly one reason), what detection and recovery gaps existed (why didn't an alert catch this sooner, why did mitigation take 40 minutes), and a short list of owned follow-ups with names and dates, not "we will review."
  • No blame, but no vagueness either: "human error" is never a root cause; it's a prompt to ask why the system allowed that error to reach production with that much impact. A good postmortem is specific enough that someone who wasn't in the incident can read it and understand exactly what happened and what's changing.
  • Apply SLOs and incident signals: RED/USE and error budgets's signals to decide severity and confirm mitigation worked; apply CI/CD and promotion: immutable artifacts to rollback's release actions (rollback, promote) as the actual mitigation tooling.

Under the hood

"Mitigate before root-causing" is the right default, not just a cultural preference, because most production incidents trace back to a recent change: a deploy, a config flip, a feature flag, a scaling event, a sudden traffic pattern. Rollback or flag-off is a mitigation that works whether or not you've identified exactly which recent change was the cause, because it reverts the whole category of "something that changed recently." Root-causing while the incident is still ongoing competes for the same attention, and often the same person's time, as actually stopping the bleeding, and a wrong hypothesis investigated under pressure costs more minutes than reverting the last deploy and checking if it helped. The familiar failure mode is someone spending 35 minutes reading code to find "the bug" while a one-command rollback was available the entire time.

Minimal example A timeline log, and mitigation found before root cause

PythonWhat got fixed, and what was merely explained, land at different timesruns
events = []


def log_event(ts, what):
    events.append((ts, what))
    print(f"[{ts:>3}m] {what}")


log_event(0, "detected: checkout error rate 40%, Sev1")
log_event(2, "incident commander assigned, status page updated")
log_event(5, "mitigation: rolled back last deploy (cause not yet known)")
log_event(9, "error rate back to baseline, incident mitigated")
log_event(40, "root cause found: new deploy missed a feature-flag default")

time_to_mitigate = next(ts for ts, what in events if "mitigated" in what)
time_to_root_cause = next(ts for ts, what in events if "root cause" in what)
print("time to mitigate:", time_to_mitigate, "minutes")
print("time to root cause:", time_to_root_cause, "minutes, found after the incident was already resolved")
Output
[  0m] detected: checkout error rate 40%, Sev1
[  2m] incident commander assigned, status page updated
[  5m] mitigation: rolled back last deploy (cause not yet known)
[  9m] error rate back to baseline, incident mitigated
[ 40m] root cause found: new deploy missed a feature-flag default
time to mitigate: 9 minutes
time to root cause: 40 minutes, found after the incident was already resolved

In production A postmortem that refuses to go out without owned follow-ups

PythonValidating the two things that make a postmortem actually usefulruns
from dataclasses import dataclass, field


@dataclass
class FollowUp:
    description: str
    owner: str
    due: str


@dataclass
class Postmortem:
    title: str
    impact: str
    contributing_conditions: list
    detection_gap: str
    follow_ups: list = field(default_factory=list)

    def validate(self):
        problems = []
        if not self.follow_ups:
            problems.append("no owned follow-ups: 'we will review' is not an action")
        for f in self.follow_ups:
            if not f.owner or not f.due:
                problems.append(f"follow-up missing owner/date: {f.description!r}")
        if "human error" in self.contributing_conditions:
            problems.append("'human error' is not a contributing condition on its own")
        return problems


good = Postmortem(
    title="Checkout 500s during 14:05-14:14 UTC",
    impact="9 minutes, ~4,800 failed checkouts",
    contributing_conditions=["new deploy changed a default", "no alert on that specific default"],
    detection_gap="caught by customer report, not by monitoring",
    follow_ups=[FollowUp("Add alert on this config default drifting", "priya", "2026-10-10")],
)

bad = Postmortem(
    title="Checkout 500s, take two",
    impact="9 minutes",
    contributing_conditions=["human error"],
    detection_gap="caught by customer report",
    follow_ups=[],
)

print("good postmortem problems:", good.validate())
print("bad postmortem problems:", bad.validate())
Output
good postmortem problems: []
bad postmortem problems: ["no owned follow-ups: 'we will review' is not an action", "'human error' is not a contributing condition on its own"]

See it happen What a Sev1 actually looks like, start to finish

monitoringon-callincident commanderstatus pagepage: checkout error rate 40%declaring Sev1public: investigating checkout issuerollback last deployerror rate back to baselinepublic: resolved, monitoringschedule postmortem, assign owner
  1. A burn-rate alert fires and pages the on-call engineer, not a human watching a dashboard.
  2. Severity is declared immediately from user impact, before the cause is known.
  3. The first external update goes out within minutes, with no claim about the cause yet.
  4. Mitigation targets the most likely recent change, not a diagnosis.
  5. The same signal that triggered the page confirms the mitigation actually worked.
  6. Resolution is announced only once the signal, not a guess, says so.
  7. The postmortem is scheduled as its own owned task, not left to happen informally.
Notice: every public update and the final "resolved" statement is driven by the same signal that triggered the page, never by how confident anyone feels.
Scenario A

In your world

The situation

Your settlement job fails at 1am. The Jenkins build goes red, you SSH in, read logs for 25 minutes, find the bug, fix and redeploy by 2am, go back to sleep. No one else was told anything.

What actually breaks

Finance opens at 9am to a half-posted settlement file from the failed run, and nobody knows the job failed and was manually patched overnight, because there was no mitigation step, just a direct fix, no communication, and no record beyond your memory of what you changed at 1:30am.

What you'd do on Monday

Treat even a solo-handled job failure as a mini-incident: post one line somewhere visible the moment you see it, mitigate first if possible (rerun from the last good checkpoint rather than hand-editing data), and write three sentences afterward: what broke, what you did, what you're changing so it doesn't repeat.

PythonA one-line incident note, written at detection time, not afterruns
import json
from pathlib import Path

INCIDENTS = Path("incidents.jsonl")
INCIDENTS.write_text("")


def note(ts, status, detail):
    with INCIDENTS.open("a") as f:
        f.write(json.dumps({"ts": ts, "status": status, "detail": detail}) + "\n")


note("01:00", "detected", "settlement job failed, 0 of 3 files posted")
note("01:05", "mitigating", "rerunning from last checkpoint instead of hand-editing output")
note("01:40", "resolved", "all 3 files posted; follow-up: alert on partial completion")

print(INCIDENTS.read_text().strip().count("\n") + 1, "entries logged before anyone had to ask")
Output
3 entries logged before anyone had to ask
Scenario B

Imagine you work at Lyft

~1,200 microservices, a Sev1 roughly every 2 weeks

The numbers

Illustrative: about 26 Sev1s/year at that cadence. If each takes an average of 45 minutes to mitigate and 2 engineer-hours to write up properly, that's roughly 19.5 hours of mitigation time and 52 engineer-hours of postmortem writing a year, assuming every one actually gets documented.

What the Scenario-A answer does here

Scenario A's "write three sentences afterward" doesn't scale once 10 teams are involved in one incident and each has its own idea of what happened; an owner-less postmortem thread turns into a 200-comment argument about whose service was "really" at fault.

What flips at this scale

At this scale, incident response needs a named, rotating incident-commander role that is explicitly not the person fixing the bug, a single source-of-truth timeline instead of six Slack threads, and a review step before publishing a postmortem, specifically to turn "contributing conditions" into a real plural list instead of a blame target.

The trade-off you'd defend

I would defend a dedicated incident-commander rotation plus a shared, append-only timeline tool over "whoever noticed it first coordinates," trading process overhead for incidents that resolve faster and postmortems people trust enough to actually learn from.

illustrative figures; not Lyft-published.

Gotchas

No one knew it was happening

Symptom

a multi-hour outage with internal confusion about whether it's even still ongoing.

Cause

no committed update cadence, so people either over-ping the fixer or assume silence means it's fine.

Fix

commit to a visible cadence, even "no change, next update in 30 min," regardless of actual progress.

The fix took 3 hours because of debugging, when a rollback existed the whole time

Symptom

described in the hood - a slow, code-level investigation while impact continues.

Cause

root-causing was prioritized over mitigating.

Fix

always ask "what's the fastest way to reduce user impact right now" before "what caused this."

The postmortem says "human error, process will be improved"

Symptom

the document produces no actual change, and the same class of incident recurs in three months.

Cause

no owned, dated follow-up, and a non-cause stood in for the real contributing conditions.

Fix

require every postmortem to carry at least one follow-up with a named owner and a date before it's considered done.

Severity was argued about mid-incident

Symptom

10 minutes spent debating "is this a Sev1 or Sev2" while impact grows.

Cause

no pre-agreed severity definitions tied to measurable user impact.

Fix

define severities in advance from the RED signals in SLOs and incident signals: RED/USE and error budgets, so declaring severity is a lookup, not a debate.

Tricky questions

Isn't fixing the bug the most important thing during an incident?
Most people say

yes, get the real fix in as fast as possible.

Actually

reducing user impact is the most important thing, and a rollback or flag-off often does that faster than finding and fixing the actual bug. Chasing the "real fix" first, while impact continues, is a common and costly mistake.

Does a postmortem need to assign blame to be honest?
Most people say

yes, otherwise nobody takes it seriously.

Actually

a good postmortem is specific about contributing conditions, plural and systemic, without naming an individual as "the cause." Blame makes people defensive and less likely to disclose the exact details that would actually prevent a repeat.

Scenario question

You're handed an ongoing incident: checkout error rate is at 15% and climbing, and someone on the team is confident they know the exact line of code causing it, but the fix needs review and a full deploy, usually 20 minutes. What do you do?

Model answer

I would ask first whether a faster mitigation exists, like rolling back the last deploy or flipping a feature flag, and check whether the timing lines up with a recent change, before waiting on the reviewed fix. If a rollback is viable and matches the timeline, I'd do that immediately to stop the bleeding, using the error-rate signal itself to confirm it worked, and let the "exact line of code" fix go through normal review afterward, since the incident needs the impact to stop, not the permanent fix in the next five minutes. The trade-off is that a rollback might also remove something good that shipped with the bad change, so I'd note that explicitly in the timeline rather than pretend it was free.

Interview corner

CommonWalk me through how you'd handle a Sev1 incident.
“I'd confirm severity from user-facing impact, assign or take the incident-commander role, post an initial status update immediately, then prioritize mitigation, rollback or flag-off, over root-causing. I'd confirm mitigation worked using the same metric that triggered the alert, keep a visible update cadence, and only then move to finding the actual cause.”
TrickYour postmortem says the cause was "developer error, more code review needed." What's wrong with that?
“'Developer error' isn't a contributing condition, it's a restatement that a human did something, which is true of almost every incident and explains nothing. I'd dig into why the system let that error reach production with that much impact, like missing tests or no alert on that failure mode, and make the follow-up specific and owned.”
Experience checkTell me about an incident where you prioritized mitigation over root cause.
“A settlement job failed overnight; instead of hand-patching the output at 1am, I reran it from the last checkpoint, which fixed the user-facing problem in minutes, and left the actual root cause, a bad default in a new deploy, for the next morning with a clear head.”
DesignHow would you design severity levels for a new service before it ever has an incident?
“I'd tie each severity directly to the RED metrics from the service's SLO, so Sev1 is 'error rate crosses the page-level burn-rate threshold' and Sev2 is the ticket-level threshold, so declaring severity during a real incident is a lookup against a number, not a debate.”

Memory hook

Smoke first, root cause after: get people out before you ask why the fryer failed.

If you remember only one thing:

Mitigate using the fastest lever that reduces user impact; find the cause once the bleeding has stopped.

Self-check

  1. Why mitigate before finding the root cause?

    Most incidents trace back to a recent change, so a rollback or flag-off reverts the likely cause category immediately, without needing to first identify exactly which change was responsible.

  2. Why is "human error" not an acceptable postmortem cause?

    It restates that a person made a mistake without explaining why the system allowed that mistake to reach production with that much impact, and it doesn't lead to an owned fix.

  3. What two things does every postmortem follow-up need to actually produce change?

    A named owner and a date.

Prove it

Write a short incident-response runbook for one real job or service you own: severity definitions tied to a concrete metric, who you'd notify and how fast, your first three mitigation options in priority order, and the one metric you'd check to confirm mitigation worked.

ce-09-06Card 6 of 6

Debug production without an attached debugger

HardP2 · Very likelyTalk about it12 min
debuggingproductionhypothesis-testingcorrelationincident-diagnosis

Why you care

You cannot pdb.set_trace() a live payments service with real traffic, and most product-company incidents happen where attaching a debugger is either impossible, a managed container, a serverless function, a customer's production tenant, or actively dangerous, since pausing a process mid-request pauses it for every concurrent request too. Interviewers ask "how do you debug production with no debugger" because the honest answer, "I correlate evidence and test hypotheses against it," is the real day-to-day skill that separates someone who's only run code locally from someone who's operated it.

Explain like I'm 5

You're a doctor who can't open the patient up to look. You have symptoms (user reports), vitals (metrics), a chart of where they've been (traces and logs), and recent events (what changed, what medication started). You form a theory from that evidence, run one safe test that would prove or disprove it, a blood test, not exploratory surgery, and only act once the evidence actually points one way. Debugging production is the same discipline: you reason from what the patient already shows you; you don't get to cut them open to see "live."

Core concept

  • Start from what changed. Line up the symptom's start time against deploys, config/flag changes, dependency updates and traffic or scaling events (CI/CD and promotion: immutable artifacts to rollback). Most "mystery" production bugs are a known change, badly correlated.
  • Correlate across signals, not within one. Use the correlation id from Observability model: logs, metrics, traces and events to pull one failing request's full path, then its logs at every hop, then the metric slice, not the aggregate, for that specific dimension - one endpoint, one region, one customer tier, during exactly that window.
  • Form a hypothesis, then run the smallest check that could disprove it, not the broadest one. "I think the connection pool is exhausted" is tested by looking at that pool's saturation metric for the window, not by redeploying with more logging and waiting for it to happen again.
  • Compare healthy vs unhealthy cohorts. If 2% of requests fail, pull a sample of the failing 2% and a sample of the healthy 98%, same endpoint, same window, and diff what's different: a header, a payload shape, a specific customer id, a region. The difference is usually the whole answer.
  • Use the OS/process tools you already have. ps/top and df -h/df -i to rule out resource exhaustion, lsof for leaked file descriptors or connections, strace on a bounded, approved process to see what syscall it's actually stuck on (Diagnose CPU, disks, descriptors and syscalls); in Kubernetes, kubectl logs --previous and describe pod for events, and knowing when CrashLoopBackOff means "the app is panicking on startup" rather than "scale it more" (Read CrashLoopBackOff and know when k8s is overkill).
  • Preserve evidence before it rotates away. Pull the exact failing request's trace, logs and metric slice into a ticket or incident doc immediately - log retention, trace sampling and container restarts will erase your evidence, often within hours.
  • Sensitive-data boundary. When comparing samples of failing requests, redact or avoid payload fields that are PII or secrets even while debugging. "I need the raw request to debug this" is not a blanket exception to data-handling policy, and most of what you need, shape, size, headers, status, doesn't require the sensitive fields anyway.

Under the hood

"Compare healthy vs unhealthy cohorts" works as a diagnostic shortcut, not just a nice idea, because it turns an open-ended search, "why did this fail," into a bounded diff, "what's different about these two groups," which is a much smaller search space once you've already narrowed by time window and endpoint using RED/USE signals. It's the same reason git bisect beats reading the whole diff since the last known-good commit: you're not trying to understand the system, you're trying to find the one dimension that correlates with failure, and a diff between two already-narrowed samples usually surfaces that dimension, a specific customer, a payload size, a header, faster than reasoning about the code path in the abstract. It fails when the two cohorts aren't actually comparable, like different endpoints or wildly different times, which is why narrowing first with metrics matters more than the diff step itself.

Minimal example Diffing a failing cohort against a healthy one finds the one differing field

PythonA bounded diff instead of reading every requestruns
requests = [
    {"id": 1, "status": 200, "region": "us", "payload_kb": 2},
    {"id": 2, "status": 200, "region": "us", "payload_kb": 3},
    {"id": 3, "status": 500, "region": "eu", "payload_kb": 2},
    {"id": 4, "status": 200, "region": "us", "payload_kb": 2},
    {"id": 5, "status": 500, "region": "eu", "payload_kb": 3},
]

failing = [r for r in requests if r["status"] >= 500]
healthy = [r for r in requests if r["status"] < 500]

for field in ("region", "payload_kb"):
    failing_values = {r[field] for r in failing}
    healthy_values = {r[field] for r in healthy}
    only_in_failing = failing_values - healthy_values
    print(field, "- values only in failing cohort:", only_in_failing)
Output
region - values only in failing cohort: {'eu'}
payload_kb - values only in failing cohort: set()

In production Correlate a deploy time with an error spike, then narrow by cohort

PythonWhat changed, lined up against who actually brokeruns
from collections import Counter

deploys = [
    {"service": "checkout", "version": "v128", "deployed_at": 100},
    {"service": "payments", "version": "v45", "deployed_at": 140},
]

requests = [
    {"ts": 90, "service": "checkout", "status": 200, "region": "us"},
    {"ts": 105, "service": "checkout", "status": 500, "region": "eu"},
    {"ts": 108, "service": "checkout", "status": 500, "region": "eu"},
    {"ts": 110, "service": "checkout", "status": 200, "region": "us"},
    {"ts": 150, "service": "payments", "status": 200, "region": "us"},
]


def errors_after(service, since_ts):
    return [r for r in requests if r["service"] == service and r["ts"] >= since_ts and r["status"] >= 500]


for d in deploys:
    bad = errors_after(d["service"], d["deployed_at"])
    print(d["service"], d["version"], "errors since deploy:", len(bad))

checkout_deploy_ts = next(d["deployed_at"] for d in deploys if d["service"] == "checkout")
before = [r for r in requests if r["service"] == "checkout" and r["ts"] < checkout_deploy_ts]
after = [r for r in requests if r["service"] == "checkout" and r["ts"] >= checkout_deploy_ts]

print("checkout error rate before deploy:", sum(1 for r in before if r["status"] >= 500), "/", len(before))
print("checkout error rate after deploy:", sum(1 for r in after if r["status"] >= 500), "/", len(after))

failing_regions = Counter(r["region"] for r in after if r["status"] >= 500)
print("regions in the post-deploy failures:", dict(failing_regions))
Output
checkout v128 errors since deploy: 2
payments v45 errors since deploy: 0
checkout error rate before deploy: 0 / 1
checkout error rate after deploy: 2 / 3
regions in the post-deploy failures: {'eu': 2}
Scenario A

In your world

The situation

A customer reports "my export is empty sometimes." Your ETL job has no debugger attached, runs in a Jenkins agent container that's gone by the time you look, and you can't reproduce it locally.

What actually breaks

You spend an hour adding print statements and redeploying, waiting for it to happen again, which it doesn't, because the condition only shows up for specific customers at a specific time of day.

What you'd do on Monday

Pull the evidence you already have instead of trying to reproduce blind: compare the empty-export runs against the successful ones on every field you logged, and test the hypothesis the diff actually points to.

PythonFinding what's different about the empty-export runs, from logs you already haveruns
runs = [
    {"run_id": "r1", "customer": "A", "rows_out": 500, "source_bytes": 12000},
    {"run_id": "r2", "customer": "B", "rows_out": 0, "source_bytes": 0},
    {"run_id": "r3", "customer": "A", "rows_out": 480, "source_bytes": 11000},
    {"run_id": "r4", "customer": "C", "rows_out": 0, "source_bytes": 0},
]

empty = [r for r in runs if r["rows_out"] == 0]
non_empty = [r for r in runs if r["rows_out"] > 0]

print("empty runs all have source_bytes == 0:", all(r["source_bytes"] == 0 for r in empty))
print("non-empty runs all have source_bytes > 0:", all(r["source_bytes"] > 0 for r in non_empty))
print("hypothesis: export is empty exactly when the source file itself was empty, not a bug in the export step")
Output
empty runs all have source_bytes == 0: True
non-empty runs all have source_bytes > 0: True
hypothesis: export is empty exactly when the source file itself was empty, not a bug in the export step
Scenario B

Imagine you work at Instacart

40,000 order-batch jobs/day across 3,000 store integrations

The numbers

Illustrative: 40,000 jobs/day ≈ 28/minute. If 0.5% produce an empty export, that's roughly 200 empty-export incidents a day, each individually indistinguishable from the others without correlation.

What the Scenario-A answer does here

Scenario A's "diff empty vs non-empty runs by hand" doesn't scale to 200/day; nobody can eyeball 200 run records a day across 3,000 store integrations and spot the pattern manually before the next batch of complaints arrives.

What flips at this scale

At this scale, the diff itself needs to be automated and run continuously, not performed ad hoc during an incident: a scheduled job that buckets failing/empty runs by every available dimension, store, time of day, source file size, upstream vendor, and flags any dimension where the failure rate is disproportionate, surfacing a ranked hypothesis list before a human even looks.

The trade-off you'd defend

I would defend investing in an automated cohort-diff job over relying on manual diffing during each incident, trading the upfront build and maintenance cost for incident response that starts with "here's the likely cause" instead of "let me look."

illustrative figures; not Instacart-published.

Gotchas

Spent an hour reproducing locally and never saw it

Symptom

hours spent trying to recreate the bug on a laptop with no result.

Cause

the triggering condition is a production-only configuration or data condition, like one customer's empty source file, that local environments can't produce.

Fix

diagnose from production evidence first; only reproduce locally once you have a specific, narrow hypothesis to test.

Added debug prints and redeployed, twice

Symptom

two deploy cycles burned chasing a problem that might not recur before you're paged for something else.

Cause

guessing which line needs more logging instead of pulling the one failing request's existing logs and trace by correlation id.

Fix

pull existing evidence by correlation id before adding new instrumentation.

The fix "worked" but you're not sure why

Symptom

a config change made the symptom go away, with no clear mechanism.

Cause

testing the broadest possible change instead of the smallest check that would prove or disprove a specific hypothesis.

Fix

state the hypothesis in one sentence and the one piece of evidence that would prove it before applying a fix; if you can't state either, you're guessing.

Debugging stalled waiting on access

Symptom

diagnostic tooling needs elevated access that wasn't requested until the incident was already underway.

Cause

bounded, read-only diagnostics weren't pre-approved before they were needed.

Fix

know and have pre-approved the diagnostics you're allowed to run (ps/top/df/lsof, kubectl logs/describe) before an incident, not during one.

Tricky questions

If you can't attach a debugger, is logging more the only option?
Most people say

yes, add more logs and wait for it to happen again.

Actually

logging more and redeploying is often the slowest option, since it requires the bug to recur after your change ships. Correlating existing signals, trace, logs by id, metric slice, recent changes, against what you already have usually answers it without waiting for a repeat.

Is comparing a few failing requests to a few healthy ones basically guessing?
Most people say

it's just eyeballing examples, not real engineering.

Actually

it's a bounded, repeatable comparison that narrows "why did this fail" into "what's structurally different about these two groups" - exactly the hypothesis you then confirm or reject with a specific check.

Scenario question

You're handed a report: "the API is slow for some users, not all, starting about an hour ago," with no debugger access and no obvious error in the logs. Walk me through your first 15 minutes.

Model answer

First, I'd check what changed in roughly that window - a deploy, a config/flag flip, a scaling event, an upstream dependency's status - using the promotion history from CI/CD and promotion: immutable artifacts to rollback. In parallel, I'd pull the latency metric sliced by whatever dimensions I have, like region, endpoint, customer tier, to see if "some users" maps to one slice or is spread evenly, since that's the difference between a localized cause and a systemic one. If I can get a sample of slow requests via their trace ids, I'd diff them against a sample of fast ones from the same window for anything structurally different, a payload size, a specific endpoint, a cache miss. I would not start reading application code line by line yet; that's a last resort once the evidence points at a specific component, not a starting point.

Interview corner

CommonHow do you debug a production issue with no debugger attached?
“I correlate what I already have: a trace or correlation id to pull one request's full path, logs at each hop, and a metric slice narrowed to the right dimension and time window. I form a specific hypothesis and run the smallest check that could disprove it, rather than adding logging and waiting for a recurrence.”
TrickYou can't reproduce the bug locally. Now what?
“I wouldn't keep trying blind. I'd treat production itself as the evidence source: pull the failing requests, compare them against healthy ones from the same window, and look for what's structurally different, since the triggering condition is often something local can't recreate anyway, like one customer's data or a specific scale.”
Experience checkTell me about a production bug you diagnosed without a debugger.
“An export came back empty for some runs. Instead of adding prints and redeploying, I compared the empty runs against the successful ones using logs we already had, and found every empty run traced back to an empty source file, not a bug in our export code at all.”
DesignHow would you reduce mean time to diagnosis across your team generally?
“I'd make sure the basics are already in place before an incident: correlation ids on every log line, RED metrics sliceable by the dimensions that matter, and a visible deploy/change timeline, because most diagnosis time is spent gathering evidence that should already be queryable, not reasoning about the code.”

Memory hook

You're a doctor who can't open the patient up; diagnose from the chart, the vitals, and what changed, not from a guess.

If you remember only one thing:

Correlate what you already have and test the smallest check that could disprove your hypothesis; don't add logging and wait for a repeat.

Self-check

  1. Why is "add more logging and redeploy" usually the slowest debugging move?

    It requires the bug to recur after your change ships, instead of using evidence, trace, logs, metrics, recent changes, you can already pull right now.

  2. What makes a healthy-vs-unhealthy cohort comparison useful?

    It turns an open-ended "why did this fail" into a bounded diff over a narrowed, comparable sample, which usually surfaces the one differing dimension.

  3. What should you check first when a symptom starts at a specific time?

    What changed around that time - a deploy, a config/flag flip, a scaling event or an upstream dependency.

Prove it

Take a dataset with at least one failing and several healthy records (synthetic is fine). Write a function that reports, for each field, which values appear only in the failing group, the way the minimal example does. Run it against two synthetic datasets, one where the cause is obvious in one field and one where it's a red herring across two correlated fields, and paste both outputs.