CloudWatch and X-Ray observability
Why you care
Your Jenkins job today has one failure mode: it prints to a console log and you scroll it by eye. The moment that job becomes a Lambda chain feeding SQS feeding another Lambda, a failure spans processes you cannot tail -f. CloudWatch Logs and metrics tell you that something happened in one service; X-Ray (or its OpenTelemetry successor) tells you where in the chain the time or error went. Interviewers ask about observability to see whether you've ever debugged a problem that crossed a network boundary—a different skill from reading one log file.
Explain like I'm 5
A building has CCTV cameras in every room, each writing to its own tape (a log stream). If a visitor causes trouble, the guard watches every tape and guesses which clips belong to the same person. A trace is a wristband: the visitor is tagged at the door, and every camera that sees it writes that down. The guard searches one wristband ID across all tapes and gets the exact path, in order.
Core concept
Three building blocks, each answering a different question.
- Logs answer "what happened in this process". A log group holds log streams sharing retention/access settings; a stream is one source. Logs Insights queries across streams. By default a log group keeps data indefinitely until you set a retention policy—a silent storage bill on a busy pipeline.
- Metrics answer "how is this measured over time". A metric lives in a namespace with dimensions (
AWS/Lambda,FunctionName); CloudWatch stores it as time-series data you can graph, alarm on, or combine with metric math. - Traces answer "where did the request's time go, across every service it touched". X-Ray groups the segments each instrumented service sends into one trace, keyed by a trace ID propagated in the
X-Amzn-Trace-Idheader. A segment can break into subsegments (a DB call, an S3 call); X-Ray assembles all of it into a service graph.
None of this works without correlation: every log line, metric and segment for one request needs a shared identifier. Structured JSON logs with a trace_id field are the cheapest version—but that's not a distributed trace unless the ID is actually passed to the next service, not just logged locally.
Sampling matters. X-Ray's default rule records the first request each second plus five percent of the rest. At low volume this looks like "everything is traced"; at real traffic most requests—including most failures—are never captured unless you write a custom rule.
OpenTelemetry migration. AWS is steering instrumentation toward OTel/ADOT; the X-Ray SDK/daemon entered a maintenance milestone on 2026-02-25. This is an instrumentation-layer change, not a claim the X-Ray backend is retired—new code should default to OTel/ADOT.
Decision rule: use Logs/metrics for "what happened and how much" inside one service; add tracing once a request crosses more than one service and you need to know which hop owns the latency, and prefer OTel/ADOT for new instrumentation.
Minimal example Correlated vs orphaned log lines
import json
def log(service, trace_id, msg):
line = {"service": service, "trace_id": trace_id, "msg": msg}
print(json.dumps(line))
return line
a = log("api", "trc-1", "received request")
b = log("worker", "trc-1", "loaded from queue")
c = log("worker", None, "loaded from queue")
print("correlated:", a["trace_id"] == b["trace_id"])
print("orphaned:", c["trace_id"] is None){"service": "api", "trace_id": "trc-1", "msg": "received request"}
{"service": "worker", "trace_id": "trc-1", "msg": "loaded from queue"}
{"service": "worker", "trace_id": null, "msg": "loaded from queue"}
correlated: True
orphaned: TrueIn production Reconstruct a service graph from segments
SEGMENTS = [
{"id": "s1", "parent": None, "name": "api", "start": 0.000, "end": 0.420},
{"id": "s2", "parent": "s1", "name": "orders-worker", "start": 0.015, "end": 0.390},
{"id": "s3", "parent": "s2", "name": "postgres", "start": 0.040, "end": 0.360},
{"id": "s4", "parent": "s2", "name": "s3-get", "start": 0.360, "end": 0.385},
]
BY_ID = {s["id"]: s for s in SEGMENTS}
children_of = {}
for seg in SEGMENTS:
children_of.setdefault(seg["parent"], []).append(seg)
root = next(s for s in SEGMENTS if s["parent"] is None)
print(f"root latency: {root['end'] - root['start']:.3f}s")
for seg in SEGMENTS:
kids = children_of.get(seg["id"], [])
child_time = sum(k["end"] - k["start"] for k in kids)
total = seg["end"] - seg["start"]
print(f"{seg['name']:<16} total={total:.3f}s own={total - child_time:.3f}s children={len(kids)}")
edges = sorted((BY_ID[seg["parent"]]["name"], seg["name"]) for seg in SEGMENTS if seg["parent"])
print("service graph edges:", edges)root latency: 0.420s
api total=0.420s own=0.045s children=1
orders-worker total=0.375s own=0.030s children=2
postgres total=0.320s own=0.320s children=0
s3-get total=0.025s own=0.025s children=0
service graph edges: [('api', 'orders-worker'), ('orders-worker', 'postgres'), ('orders-worker', 's3-get')]See it happen Trace ID propagation across the chain
- No trace header yet. The first instrumented service mints the trace ID.
- The api service sends its segment, tagged with a freshly generated trace ID.
- The trace ID rides as a message attribute, not just inside the log line.
- Worker reads the message, including the trace_id attribute.
- Worker's segment carries the same trace ID, so it joins the same trace, not a new one.
- The backend assembles every segment sharing trc-1 into one trace and one graph.
In your world
The situation
Your nightly pipeline is an S3-triggered extractor Lambda that drops a message on SQS, and a loader Lambda that reads it and writes to Postgres. Both use print() for logging, captured into two separate log groups with no shared field.
What actually breaks
23 of 1,400 nightly invocations (1.6%) raise psycopg2.OperationalError: could not connect. You can see the 23 error lines in the loader's log group, but can't tell which upstream S3 files they correspond to—neither log group has a field tying one Lambda's invocation to the other's. Matching timestamps by eye only works when invocations don't overlap, and on a bad night they do.
What you'd do on Monday
Generate a request ID in the extractor, pass it as an SQS message attribute, and log it as a structured field in both Lambdas. One Logs Insights query across both log groups filtered on that ID now returns the full story in one place.
import json
import itertools
_counter = itertools.count(1)
def extract(s3_key):
request_id = f"req-{next(_counter):04d}"
print(json.dumps({"service": "extractor", "request_id": request_id, "msg": "sent to queue", "s3_key": s3_key}))
return {"MessageAttributes": {"request_id": {"StringValue": request_id}}, "Body": s3_key}
def load(message):
request_id = message["MessageAttributes"]["request_id"]["StringValue"]
print(json.dumps({"service": "loader", "request_id": request_id, "msg": "load failed", "s3_key": message["Body"]}))
msg = extract("raw/orders/2026-01-15/part-004.csv")
load(msg){"service": "extractor", "request_id": "req-0001", "msg": "sent to queue", "s3_key": "raw/orders/2026-01-15/part-004.csv"}
{"service": "loader", "request_id": "req-0001", "msg": "load failed", "s3_key": "raw/orders/2026-01-15/part-004.csv"}import boto3
logs = boto3.client("logs", region_name="ap-south-1")
logs.start_query(
logGroupNames=["/aws/lambda/orders-extractor", "/aws/lambda/orders-loader"],
startTime=1736899200, endTime=1736985600,
queryString="fields @timestamp, service, msg | filter request_id = 'req-8041' | sort @timestamp asc",
)Imagine you work at Groww
The numbers
Illustrative: during the busiest 10-minute batch window, invocations peak at roughly 50/s, so about 30,000 occur in that window. X-Ray's default rule records 1/s × 600s = 600 traces, plus 5% of the remaining 29,400 ≈ 1,470, for about 2,070 of 30,000—roughly 7% sampled, 93% logged-but-not-traced.
What the Scenario-A answer does here
The Scenario A fix (propagate a request ID, correlate logs) still finds the failing invocation's logs. But if you were relying on default sampling to also hand you a trace for that failure, you're usually wrong: at 93% unsampled, that request almost certainly has no trace.
What flips at this scale
At this volume, sampling has to be a decision, not a default. Add a custom rule that traces 100% of requests on the error-prone path while keeping the default rate on the high-volume success path, and move new instrumentation to OTel/ADOT.
The trade-off you'd defend
I would defend a custom, error-biased sampling rule over simply raising the global sampling percentage, trading blanket completeness for lower trace cost while keeping full visibility where incidents happen.
Illustrative sampling arithmetic built from X-Ray's documented default rule; invocation volumes are not Groww-published figures.
Gotchas
Default X-Ray sampling drops most traces silently
The error-rate graph shows hundreds of failures today, but the X-Ray console has traces for only a handful.
The default rule records 1 request/second plus 5% of the rest; at real traffic that's a small minority of requests, including failing ones.
Add a custom sampling rule that traces errors and the critical path at or near 100%, default elsewhere.
Log groups never expire by default
The CloudWatch Logs bill climbs steadily month over month even though invocation volume is flat.
A log group's retention is indefinite unless you explicitly set an expiration.
Set an explicit retention policy (30 or 90 days) on every log group as part of the IaC that creates it.
Unstructured prints can't be queried
A Logs Insights query for a specific order ID returns nothing, even though you can see the line by scrolling.
print() with no consistent field names has nothing for Logs Insights to parse.
Emit JSON lines with the same field names across every service.
Tricky questions
Yes, once instrumentation is in place every request produces a trace.
No. The default rule records roughly the first request per second plus 5% of the rest. Most requests—often the one you care about—aren't traced unless you write a sampling rule saying otherwise.
Yes, if every service logs structured JSON you have tracing.
A structured log line only describes that one service. It joins a trace only if the identifier is forwarded to the next hop—logging an ID locally without passing it on breaks the chain right there.
Scenario question
You're handed a checkout flow built from five Lambda functions chained by SQS and EventBridge. Checkout is intermittently slow, and nobody can say which function is responsible. Each logs to its own log group with no shared field. How do you find the slow hop without rewriting the whole chain first?
Model answer
I would add one structured field—a request ID generated at the entry point, passed through every SQS attribute and EventBridge payload—to every function's existing logs. One Logs Insights query across all five log groups then shows timestamps per hop, usually finding the slow function within a day. Once I know which hop matters, I'd add X-Ray/OTel there first, with sampling biased toward that path, rather than instrumenting all five uniformly. The trade-off: manual correlation is slower to query than a real trace, but it ships same-day.
Interview corner
CommonWhat is the difference between CloudWatch Logs, CloudWatch metrics and X-Ray traces?
TrickYou've enabled X-Ray on every Lambda in the chain. Why are half your incidents missing a trace?
Experience checkHow have you debugged a cross-service failure without full tracing in place?
DesignWould you adopt OTel/ADOT or the classic X-Ray SDK for a new service today?
Memory hook
A trace is a wristband, not a notebook entry: it only works if every camera it passes writes it down.
An identifier that isn't forwarded to the next hop isn't tracing—it's just a local log field.
Self-check
What does X-Ray's default sampling rule record?
The first request each second, plus five percent of any additional requests in that second.
What happens to CloudWatch log data if you never set a retention policy?
It is stored indefinitely, and the storage cost grows with it.
Why is a structured JSON log line, by itself, not a distributed trace?
Because it only describes one service's view. It becomes part of a trace only when the identifier is propagated to the next service, not just logged locally.
Extend the production example with a fifth segment that is a sibling of postgres and s3-get (another subsegment of orders-worker), choose start/end times so the three children's combined duration exceeds the parent's total duration, and show that own time goes negative—then explain in one sentence why that signals overlapping (concurrent) subsegments rather than a bug in the arithmetic.