Observability model: logs, metrics, traces and events
Why you care
"You get paged. The dashboard says the checkout error rate is up 4x. Which tool do you open first, and why?" is the real version of "what's the difference between logs, metrics, and traces," and it gets asked constantly because a wrong instinct wastes the first ten minutes of every incident. Pure logs mean grepping blind across machines with no sense of "how often." Pure metrics mean you can see something is wrong with zero narrative for any one request. Someone who has only ever print()ed into a Jenkins console, with no trace, no metric, and no durable event store, will build a system nobody can debug once it leaves their laptop.
Explain like I'm 5
A hospital runs on all four signals at once. Logs are the nurse's handwritten chart on one patient: detailed, per-event, cheap to write and expensive to search across a thousand patients. Metrics are the ward's vital-sign board: a handful of numbers, refreshed every minute, that tell you something is wrong across every patient at once but nothing about any one of them. Traces are the patient's routing slip, stapled on at admission with a single ID, carried room to room and timestamped at every stop, so you can see the three hours they specifically spent waiting in radiology. Events are the entries that matter on their own, even without the slip: "medication administered at 14:02" has to survive and stay searchable long after the slip itself is thrown away.
Core concept
- Logs: discrete, timestamped records, usually per request or per step. Cheap to write, expensive to search at volume unless they're structured (Add structured logs and Python OpenTelemetry).
- Metrics: aggregated numeric time series. Cheap at any volume, but carry no per-request detail - you cannot ask a metric "which customer."
- Traces: a tree of causally-linked spans showing one request's path across services. A span has a name, start/end time, attributes, a status, and can carry span events - timestamped notes attached to that span, same id space and same lifecycle as the trace.
- Events: point-in-time occurrences. They either ride inside a trace as a span event, or stand alone in a durable store (an audit table, a business-event log) with their own retention.
- Correlation ids: generate one id (the trace id) at the edge, log it on every line, and propagate it across process boundaries via headers (W3C Trace Context is the default). That one id is what lets you jump from metric to trace to log for a single request instead of guessing by time window across three tools.
- Layers: instrumentation (what the app does - creates spans, counters, log lines), export (how signals leave the process - batched to a collector or backend), and storage (where they live and for how long) fail independently. "The logs exist" does not mean they're indexed or queryable yet.
- Decision rule: pick the tool by the question. "Is something wrong right now, and how bad" → a metric. "What exactly happened to this one request" → a trace, then logs filtered by its id. "Did a specific business fact happen, and will someone ask about it in three months" → an independent, durably-stored event, never a span event, because a span event is not a system of record.
Under the hood
Span events live physically inside the span they belong to - no separate table, no separate id space. That is exactly why they vanish together with the trace. If your tracing backend samples 1% of traces to control cost, the 99% of span events attached to dropped traces are dropped too, silently, because the sampling decision runs independent of whether anyone later needed that event for an audit. A metric survives this because it's emitted independent of any one trace's sampling decision; a durable event survives it because you deliberately wrote it somewhere that is not the tracing backend at all.
Minimal example One trace_id ties three signals together
logs = [
{"trace_id": "T1", "service": "checkout", "msg": "received order", "ts": 1},
{"trace_id": "T2", "service": "checkout", "msg": "received order", "ts": 2},
{"trace_id": "T1", "service": "payments", "msg": "card declined", "ts": 3},
]
spans = [
{"trace_id": "T1", "span_id": "s1", "service": "checkout", "parent": None},
{"trace_id": "T1", "span_id": "s2", "service": "payments", "parent": "s1"},
{"trace_id": "T2", "span_id": "s3", "service": "checkout", "parent": None},
]
metric_checkout_errors_total = 1 # a counter: no trace_id, just a number
def narrative(trace_id):
path = [s["service"] for s in sorted(spans, key=lambda s: s["span_id"]) if s["trace_id"] == trace_id]
lines = [l["msg"] for l in logs if l["trace_id"] == trace_id]
return path, lines
print(narrative("T1"))
print("metric alone tells us:", metric_checkout_errors_total, "error(s), for which request? unknown")(['checkout', 'payments'], ['received order', 'card declined'])
metric alone tells us: 1 error(s), for which request? unknownIn production A durable business event survives trace sampling
import json
import random
from pathlib import Path
random.seed(7)
AUDIT = Path("orders_audit.jsonl")
AUDIT.write_text("") # fresh file, our durable event store
class FakeSpan:
"""Stand-in for an OpenTelemetry span: events live inside it."""
def __init__(self, trace_id, sampled):
self.trace_id = trace_id
self.sampled = sampled
self.events = []
def add_event(self, name, **attrs):
self.events.append({"name": name, **attrs})
def record_order_placed(trace_id, order_id, amount, sampled):
span = FakeSpan(trace_id, sampled)
span.add_event("order.placed", order_id=order_id, amount=amount) # span event: rides with the trace
with AUDIT.open("a") as f: # durable event: independent of the trace's fate
f.write(json.dumps({"trace_id": trace_id, "order_id": order_id, "amount": amount}) + "\n")
if not span.sampled:
span.events.clear() # the backend never received this trace; its events are gone
return span
kept = record_order_placed("T1", "O-100", 4999, sampled=True)
dropped = record_order_placed("T2", "O-101", 1299, sampled=False)
print("kept trace still has its span event:", len(kept.events) == 1)
print("dropped trace lost its span event:", len(dropped.events) == 0)
print("audit log has both orders regardless of sampling:",
len(AUDIT.read_text().strip().splitlines()) == 2)kept trace still has its span event: True
dropped trace lost its span event: True
audit log has both orders regardless of sampling: TrueSee it happen What dies with the trace, and what doesn't
- The request arrives and the edge generates trace_id=T1; nothing has been recorded yet.
- checkout starts a span and writes one log line stamped with trace_id=T1.
- checkout also increments errors_total, a plain number with no trace_id attached.
- trace_id propagates to payments, which starts a child span under svcA.
- payments writes order.placed to a durable event store, independent of the trace.
- If this trace is later sampled out, svcA and svcB disappear together with the log's trace_id context, but the durable event is untouched.
In your world
The situation
Your ETL job writes log.info("loaded batch") lines to stdout, which Jenkins captures as console output. When finance asks "did settlement batch #4821 actually post," you have no way to answer except reading an entire day's console log top to bottom.
What actually breaks
Jenkins truncates console output past a size limit and recycles old build logs after 30 days. Three weeks later, the batch in question is gone, and the only trace of it is "job succeeded" in the build history, with no connection to which rows it touched.
What you'd do on Monday
Give every run a correlation id generated at the start, log it on every line, and additionally write one durable event per run to a store outside Jenkins - a file or table with batch id, correlation id, row count and status that outlives build-log retention entirely.
import itertools
import json
import logging
import sys
from pathlib import Path
logging.basicConfig(level=logging.INFO, format="%(message)s", stream=sys.stdout)
log = logging.getLogger("settlements")
EVENTS = Path("batch_events.jsonl")
_next_cid = itertools.count(1)
def run_batch(rows):
cid = f"batch-{next(_next_cid)}"
log.info(json.dumps({"cid": cid, "msg": "batch started", "rows": len(rows)}))
posted = sum(1 for r in rows if r["amount"] > 0)
log.info(json.dumps({"cid": cid, "msg": "batch finished", "posted": posted}))
with EVENTS.open("a") as f:
f.write(json.dumps({"cid": cid, "posted": posted, "status": "done"}) + "\n")
return cid
cid = run_batch([{"amount": 250}, {"amount": -10}, {"amount": 90}])
print("durable record for", cid, "exists:", cid in EVENTS.read_text()){"cid": "batch-1", "msg": "batch started", "rows": 3}
{"cid": "batch-1", "msg": "batch finished", "posted": 2}
durable record for batch-1 exists: TrueImagine you work at CRED
The numbers
Illustrative: 2M requests/day ≈ 23/sec on average, each touching about 6 services end to end, so roughly 12M span-equivalents a day. At that count there is no "read all the logs"; the only way to find one request's path is correlation, not volume.
What the Scenario-A answer does here
The Scenario A fix (one correlation id per run) still works for a batch job, but across 40 services in a synchronous call chain, "each service mints its own id" quietly breaks it: a service that doesn't read the incoming id and generates a fresh one severs correlation for every hop downstream of it, and nobody notices until an incident needs the full path.
What flips at this scale
At this scale, trace-id propagation has to become a cross-service contract enforced at the platform layer, not a convention each team remembers - middleware that rejects or flags internal requests missing the propagated id - because one unpropagating service breaks correlation for every request that passes through it, not just its own.
The trade-off you'd defend
I would defend mandating header propagation as a shared middleware requirement over "ask every team to remember," trading some standardization work for the ability to answer "what happened to this one request" at all once you're past a handful of services.
illustrative figures to make the arithmetic concrete; not CRED-published.
Gotchas
Correlation id missing on some log lines
half a request's log lines carry the id and half don't, usually from a background thread or an async task spawned mid-request.
the id lives in a context var or thread-local that a new thread or task doesn't automatically inherit.
propagate the id explicitly across any thread/task boundary instead of relying on ambient context alone.
A span event mistaken for an audit trail
compliance asks for every refund issued last quarter, and the tracing backend's 7-day retention has nothing.
span events live and die with the trace that carries them.
write anything someone will ask about later to a durable, independently-retained event store from day one.
Two different trace ids for one request
logs show id T1 upstream and T2 downstream for what was clearly a single user action.
a downstream service didn't read the incoming trace header and minted its own id instead.
enforce header propagation, and fail loudly in local testing if the incoming traceparent isn't honoured.
Tricky questions
yes, same thing with extra metadata attached.
a log line exists independent of any trace and can have its own retention; a span event is physically part of the span and disappears if that trace isn't sampled or kept. Treat them as different storage lifetimes, not different formats of the same fact.
no, grep is enough once logs are detailed.
logs tell you what happened per line, not causality across services. Without a shared id you cannot order events from different services into one request's actual path, especially under concurrency where timestamps alone lie.
Scenario question
You're handed a service that logs plenty but every incident takes 40+ minutes because nobody can tell which of 15 log lines across 5 services belong to the slow request a customer reported. What's your first change, and what do you explicitly not build yet?
Model answer
I would add one thing first: a correlation id generated at the edge and propagated through every service boundary and into every log line, because that turns "read everything" into "filter by one id." I would not reach for a full tracing backend yet - the id-plus-structured-logs change is a day of middleware work and immediately cuts search time, while standing up a trace collector is a bigger investment I'd justify once log-filtering alone genuinely isn't enough, for example once I need per-service timing, not just an ordered list of what happened.
Interview corner
CommonWhat's the difference between logs, metrics, and traces?
TrickIf my span has events on it, do I still need a separate audit table?
Experience checkWhere have correlation ids actually fixed a real problem for you?
DesignYou're adding observability to a new service. What goes in first?
Memory hook
A patient's routing slip ties every room visit to one id; throw away the slip and you can still find the chart, but not the story.
Pick the tool by the question you're asking, and never let a fact someone will ask about later live only inside a trace.
Self-check
What's the one thing a trace gives you that logs alone don't?
Causal ordering of one request's path across multiple services, not just a timestamp per line.
Why can't you rely on a span event as an audit record?
It's stored inside the span and disappears if that trace is sampled out or ages past trace retention.
Why don't metrics usually carry a trace id?
A unique value per request as a label would blow up cardinality (Choose Prometheus metric types and useful dashboards); metrics are meant to be aggregated across many requests, not keyed to one.
Pick a script you run regularly. Add a correlation id generated once per run, log it on every line, and write one JSON line per run to a runs.jsonl file with the id, status and a count. Run it three times, then paste a filter that extracts the full log for exactly one run by its id, plus the matching line from runs.jsonl.