Shelf · AWS · Chapter 9

Chapter 9: Operations & resilience

Deploying a job is the easy half; knowing it is healthy, and being able to prove you can bring it back from the dead, is the half nobody rehearses until an incident forces it. This chapter builds the operating loop—logs, metrics, traces, alarms, backups, DR strategy, controlled failure drills and the difference between a release-ticket clone and an actual recovery—so a 2 a.m. page has a runbook instead of a guess.

Difficulty
Priority
aws-09-01Card 1 of 6

CloudWatch and X-Ray observability

MediumP3 · Role-dependentTalk about it11 min
cloudwatchx-rayotellogsmetricstracingcorrelation
Needs: nothing. Start here.

Why you care

Your Jenkins job today has one failure mode: it prints to a console log and you scroll it by eye. The moment that job becomes a Lambda chain feeding SQS feeding another Lambda, a failure spans processes you cannot tail -f. CloudWatch Logs and metrics tell you that something happened in one service; X-Ray (or its OpenTelemetry successor) tells you where in the chain the time or error went. Interviewers ask about observability to see whether you've ever debugged a problem that crossed a network boundary—a different skill from reading one log file.

Explain like I'm 5

A building has CCTV cameras in every room, each writing to its own tape (a log stream). If a visitor causes trouble, the guard watches every tape and guesses which clips belong to the same person. A trace is a wristband: the visitor is tagged at the door, and every camera that sees it writes that down. The guard searches one wristband ID across all tapes and gets the exact path, in order.

Core concept

Three building blocks, each answering a different question.

  • Logs answer "what happened in this process". A log group holds log streams sharing retention/access settings; a stream is one source. Logs Insights queries across streams. By default a log group keeps data indefinitely until you set a retention policy—a silent storage bill on a busy pipeline.
  • Metrics answer "how is this measured over time". A metric lives in a namespace with dimensions (AWS/Lambda, FunctionName); CloudWatch stores it as time-series data you can graph, alarm on, or combine with metric math.
  • Traces answer "where did the request's time go, across every service it touched". X-Ray groups the segments each instrumented service sends into one trace, keyed by a trace ID propagated in the X-Amzn-Trace-Id header. A segment can break into subsegments (a DB call, an S3 call); X-Ray assembles all of it into a service graph.

None of this works without correlation: every log line, metric and segment for one request needs a shared identifier. Structured JSON logs with a trace_id field are the cheapest version—but that's not a distributed trace unless the ID is actually passed to the next service, not just logged locally.

Sampling matters. X-Ray's default rule records the first request each second plus five percent of the rest. At low volume this looks like "everything is traced"; at real traffic most requests—including most failures—are never captured unless you write a custom rule.

OpenTelemetry migration. AWS is steering instrumentation toward OTel/ADOT; the X-Ray SDK/daemon entered a maintenance milestone on 2026-02-25. This is an instrumentation-layer change, not a claim the X-Ray backend is retired—new code should default to OTel/ADOT.

Decision rule: use Logs/metrics for "what happened and how much" inside one service; add tracing once a request crosses more than one service and you need to know which hop owns the latency, and prefer OTel/ADOT for new instrumentation.

Minimal example Correlated vs orphaned log lines

PythonStructured logs correlated by a shared trace idruns
import json

def log(service, trace_id, msg):
    line = {"service": service, "trace_id": trace_id, "msg": msg}
    print(json.dumps(line))
    return line

a = log("api", "trc-1", "received request")
b = log("worker", "trc-1", "loaded from queue")
c = log("worker", None, "loaded from queue")

print("correlated:", a["trace_id"] == b["trace_id"])
print("orphaned:", c["trace_id"] is None)
Output
{"service": "api", "trace_id": "trc-1", "msg": "received request"}
{"service": "worker", "trace_id": "trc-1", "msg": "loaded from queue"}
{"service": "worker", "trace_id": null, "msg": "loaded from queue"}
correlated: True
orphaned: True

In production Reconstruct a service graph from segments

PythonReconstruct a trace from segments, like an X-Ray service graphruns
SEGMENTS = [
    {"id": "s1", "parent": None, "name": "api",           "start": 0.000, "end": 0.420},
    {"id": "s2", "parent": "s1", "name": "orders-worker",  "start": 0.015, "end": 0.390},
    {"id": "s3", "parent": "s2", "name": "postgres",       "start": 0.040, "end": 0.360},
    {"id": "s4", "parent": "s2", "name": "s3-get",         "start": 0.360, "end": 0.385},
]
BY_ID = {s["id"]: s for s in SEGMENTS}

children_of = {}
for seg in SEGMENTS:
    children_of.setdefault(seg["parent"], []).append(seg)

root = next(s for s in SEGMENTS if s["parent"] is None)
print(f"root latency: {root['end'] - root['start']:.3f}s")

for seg in SEGMENTS:
    kids = children_of.get(seg["id"], [])
    child_time = sum(k["end"] - k["start"] for k in kids)
    total = seg["end"] - seg["start"]
    print(f"{seg['name']:<16} total={total:.3f}s  own={total - child_time:.3f}s  children={len(kids)}")

edges = sorted((BY_ID[seg["parent"]]["name"], seg["name"]) for seg in SEGMENTS if seg["parent"])
print("service graph edges:", edges)
Output
root latency: 0.420s
api              total=0.420s  own=0.045s  children=1
orders-worker    total=0.375s  own=0.030s  children=2
postgres         total=0.320s  own=0.320s  children=0
s3-get           total=0.025s  own=0.025s  children=0
service graph edges: [('api', 'orders-worker'), ('orders-worker', 'postgres'), ('orders-worker', 's3-get')]

See it happen Trace ID propagation across the chain

clientapiSQSworkerX-Ray/OTelPOST /orderssegment: api (trace=trc-1)SendMessage + trace_id attributeReceiveMessagesegment: worker (trace=trc-1)service graph: api -> SQS -> worker
  1. No trace header yet. The first instrumented service mints the trace ID.
  2. The api service sends its segment, tagged with a freshly generated trace ID.
  3. The trace ID rides as a message attribute, not just inside the log line.
  4. Worker reads the message, including the trace_id attribute.
  5. Worker's segment carries the same trace ID, so it joins the same trace, not a new one.
  6. The backend assembles every segment sharing trc-1 into one trace and one graph.
Notice: the trace ID is useless the moment it stops being forwarded. A log line that mentions a trace ID without passing it to the next hop breaks the chain exactly where you need it most.
Scenario A

In your world

The situation

Your nightly pipeline is an S3-triggered extractor Lambda that drops a message on SQS, and a loader Lambda that reads it and writes to Postgres. Both use print() for logging, captured into two separate log groups with no shared field.

What actually breaks

23 of 1,400 nightly invocations (1.6%) raise psycopg2.OperationalError: could not connect. You can see the 23 error lines in the loader's log group, but can't tell which upstream S3 files they correspond to—neither log group has a field tying one Lambda's invocation to the other's. Matching timestamps by eye only works when invocations don't overlap, and on a bad night they do.

What you'd do on Monday

Generate a request ID in the extractor, pass it as an SQS message attribute, and log it as a structured field in both Lambdas. One Logs Insights query across both log groups filtered on that ID now returns the full story in one place.

PythonA request id travels with the message, not just the log lineruns
import json
import itertools

_counter = itertools.count(1)

def extract(s3_key):
    request_id = f"req-{next(_counter):04d}"
    print(json.dumps({"service": "extractor", "request_id": request_id, "msg": "sent to queue", "s3_key": s3_key}))
    return {"MessageAttributes": {"request_id": {"StringValue": request_id}}, "Body": s3_key}

def load(message):
    request_id = message["MessageAttributes"]["request_id"]["StringValue"]
    print(json.dumps({"service": "loader", "request_id": request_id, "msg": "load failed", "s3_key": message["Body"]}))

msg = extract("raw/orders/2026-01-15/part-004.csv")
load(msg)
Output
{"service": "extractor", "request_id": "req-0001", "msg": "sent to queue", "s3_key": "raw/orders/2026-01-15/part-004.csv"}
{"service": "loader", "request_id": "req-0001", "msg": "load failed", "s3_key": "raw/orders/2026-01-15/part-004.csv"}
PythonQuery both log groups by request_id (needs boto3 + CloudWatch Logs Insights)
import boto3

logs = boto3.client("logs", region_name="ap-south-1")
logs.start_query(
    logGroupNames=["/aws/lambda/orders-extractor", "/aws/lambda/orders-loader"],
    startTime=1736899200, endTime=1736985600,
    queryString="fields @timestamp, service, msg | filter request_id = 'req-8041' | sort @timestamp asc",
)
Scenario B

Imagine you work at Groww

40,000 Lambda invocations/night across 12 functions

The numbers

Illustrative: during the busiest 10-minute batch window, invocations peak at roughly 50/s, so about 30,000 occur in that window. X-Ray's default rule records 1/s × 600s = 600 traces, plus 5% of the remaining 29,400 ≈ 1,470, for about 2,070 of 30,000—roughly 7% sampled, 93% logged-but-not-traced.

What the Scenario-A answer does here

The Scenario A fix (propagate a request ID, correlate logs) still finds the failing invocation's logs. But if you were relying on default sampling to also hand you a trace for that failure, you're usually wrong: at 93% unsampled, that request almost certainly has no trace.

What flips at this scale

At this volume, sampling has to be a decision, not a default. Add a custom rule that traces 100% of requests on the error-prone path while keeping the default rate on the high-volume success path, and move new instrumentation to OTel/ADOT.

The trade-off you'd defend

I would defend a custom, error-biased sampling rule over simply raising the global sampling percentage, trading blanket completeness for lower trace cost while keeping full visibility where incidents happen.

Illustrative sampling arithmetic built from X-Ray's documented default rule; invocation volumes are not Groww-published figures.

Gotchas

Default X-Ray sampling drops most traces silently

Symptom

The error-rate graph shows hundreds of failures today, but the X-Ray console has traces for only a handful.

Cause

The default rule records 1 request/second plus 5% of the rest; at real traffic that's a small minority of requests, including failing ones.

Fix

Add a custom sampling rule that traces errors and the critical path at or near 100%, default elsewhere.

Log groups never expire by default

Symptom

The CloudWatch Logs bill climbs steadily month over month even though invocation volume is flat.

Cause

A log group's retention is indefinite unless you explicitly set an expiration.

Fix

Set an explicit retention policy (30 or 90 days) on every log group as part of the IaC that creates it.

Unstructured prints can't be queried

Symptom

A Logs Insights query for a specific order ID returns nothing, even though you can see the line by scrolling.

Cause

print() with no consistent field names has nothing for Logs Insights to parse.

Fix

Emit JSON lines with the same field names across every service.

Tricky questions

Does turning on X-Ray mean every request gets traced?
Most people say

Yes, once instrumentation is in place every request produces a trace.

Actually

No. The default rule records roughly the first request per second plus 5% of the rest. Most requests—often the one you care about—aren't traced unless you write a sampling rule saying otherwise.

Is a structured JSON log line the same thing as a distributed trace?
Most people say

Yes, if every service logs structured JSON you have tracing.

Actually

A structured log line only describes that one service. It joins a trace only if the identifier is forwarded to the next hop—logging an ID locally without passing it on breaks the chain right there.

Scenario question

You're handed a checkout flow built from five Lambda functions chained by SQS and EventBridge. Checkout is intermittently slow, and nobody can say which function is responsible. Each logs to its own log group with no shared field. How do you find the slow hop without rewriting the whole chain first?

Model answer

I would add one structured field—a request ID generated at the entry point, passed through every SQS attribute and EventBridge payload—to every function's existing logs. One Logs Insights query across all five log groups then shows timestamps per hop, usually finding the slow function within a day. Once I know which hop matters, I'd add X-Ray/OTel there first, with sampling biased toward that path, rather than instrumenting all five uniformly. The trade-off: manual correlation is slower to query than a real trace, but it ships same-day.

Interview corner

CommonWhat is the difference between CloudWatch Logs, CloudWatch metrics and X-Ray traces?
“I'd say logs tell you what happened inside one process, metrics tell you how a number changed over time, and traces tell you where a request's time went across every service it touched. A mature setup uses all three with a shared identifier connecting them.”
TrickYou've enabled X-Ray on every Lambda in the chain. Why are half your incidents missing a trace?
“I'd point at the default sampling rule—one request a second plus five percent of the rest. At real traffic most requests aren't sampled, including failures. I'd fix that with a custom rule prioritizing error responses and the critical path.”
Experience checkHow have you debugged a cross-service failure without full tracing in place?
“I retrofitted a single request ID into existing log lines, passed through SQS message attributes, then queried across log groups with Logs Insights. Not as clean as a trace, but it ships in hours and tells me exactly what to instrument next.”
DesignWould you adopt OTel/ADOT or the classic X-Ray SDK for a new service today?
“I'd pick OTel/ADOT. AWS has moved the classic SDK and daemon into a maintenance posture and is steering new instrumentation toward OpenTelemetry, so starting on the legacy SDK just means a migration later.”

Memory hook

A trace is a wristband, not a notebook entry: it only works if every camera it passes writes it down.

If you remember only one thing:

An identifier that isn't forwarded to the next hop isn't tracing—it's just a local log field.

Self-check

  1. What does X-Ray's default sampling rule record?

    The first request each second, plus five percent of any additional requests in that second.

  2. What happens to CloudWatch log data if you never set a retention policy?

    It is stored indefinitely, and the storage cost grows with it.

  3. Why is a structured JSON log line, by itself, not a distributed trace?

    Because it only describes one service's view. It becomes part of a trace only when the identifier is propagated to the next service, not just logged locally.

Prove it

Extend the production example with a fifth segment that is a sibling of postgres and s3-get (another subsegment of orders-worker), choose start/end times so the three children's combined duration exceeds the parent's total duration, and show that own time goes negative—then explain in one sentence why that signals overlapping (concurrent) subsegments rather than a bug in the arithmetic.

aws-09-02Card 2 of 6

Alarms, SLOs and Well-Architected reviews

HardP3 · Role-dependentTalk about it11 min
cloudwatch-alarmsmissing-datacomposite-alarmsslowell-architected

Why you care

A dashboard nobody is paged from is decoration. The gap between "we have monitoring" and "we get woken up for the right thing" is almost entirely two unglamorous details: how an alarm treats missing data, and whether you've reduced twelve noisy alarms into one actionable signal. Well-Architected reviews exist because "it works in the demo" and "it survives a bad week in production" are different claims. Interviewers who ask about alarms are checking whether you've been paged for something that wasn't actually a problem—everyone who has operated a service has been.

Explain like I'm 5

A smoke alarm that goes silent the moment its battery dies is worse than no alarm, because you stop checking it. CloudWatch alarms have the same question built in: if the metric stops reporting, should the alarm assume fine, assume the worst, freeze where it was, or admit it doesn't know? You pick the answer per alarm, because "the sensor went quiet" means something different for a disk-full alarm than a once-a-day batch metric.

Core concept

A metric alarm watches one metric (or a math expression) against a threshold over evaluation periods; a composite alarm combines other alarms' states with AND/OR logic and only reaches ALARM when its rule is satisfied—useful for turning twelve flapping alarms into one page that fires only when they agree.

Every alarm is in one of three states: OK, ALARM, or INSUFFICIENT_DATA. Each data point is breaching, not breaching, or missing. You configure treat missing data per alarm as one of:

  • missing (default) — if every point in the evaluation range is missing, the alarm goes to INSUFFICIENT_DATA.
  • notBreaching — missing points are treated as good; a quiet metric doesn't page you.
  • breaching — missing points are treated as bad; use when silence itself is the symptom.
  • ignore — the alarm holds whatever state it was already in.

CloudWatch backfills gaps with real older data points before ever consulting your missing-data setting (mechanics in the hood section below)—so a 3-period alarm can survive one or two gaps using older real points.

Decision rule: pick breaching for anything that should always be emitting and whose silence is the incident (a heartbeat, a per-minute batch counter); pick notBreaching for metrics only emitted on failure (no baseline "0"); leave missing for anything you'd rather investigate manually.

Well-Architected reviews walk six pillars—operational excellence, security, reliability, performance efficiency, cost optimization, sustainability—answered honestly against a real workload. The review's only output is a "high risk" list and a plan; it has no enforcement, so its value is entirely whether the team acts on it. An SLO is the number you commit to, built from actionable alarms: every page should have a known next step, or it's noise.

Under the hood

CloudWatch requests a wider evaluation range than the configured evaluation periods, specifically to tolerate gaps. If the most recent N points are all real, it uses only those. If some are missing but enough real older points exist in the range, it uses those real points and your missing-data setting is never consulted—it only fills gaps when there genuinely aren't enough real points. A second rule avoids "premature OK": if the oldest breaching point within range is already as old as Datapoints to Alarm requires, the alarm fires even with more recent points missing rather than breaching—CloudWatch would rather alarm a little early than let a real breach hide behind trailing missing data.

Minimal example Alarm state from a hand-rolled missing-data policy

PythonSame gaps, four different alarm outcomes depending on missing-data treatmentruns
def evaluate(points, treat, n=3):
    recent = points[-n:]
    if not any(p is not None for p in recent):
        return {"missing": "INSUFFICIENT_DATA", "ignore": "RETAIN_CURRENT",
                "breaching": "ALARM", "notBreaching": "OK"}[treat]
    return "ALARM" if any(recent) else "OK"

ALL_MISSING = [False, None, None, None]  # last 3 points (the evaluation window) are all missing
for treat in ("missing", "notBreaching", "breaching", "ignore"):
    print(f"{treat:<14} -> {evaluate(ALL_MISSING, treat)}")
Output
missing        -> INSUFFICIENT_DATA
notBreaching   -> OK
breaching      -> ALARM
ignore         -> RETAIN_CURRENT

In production Composite alarm reduces noise across a pipeline

PythonComposite alarm fires only when child alarms agreeruns
class MetricAlarm:
    def __init__(self, name, state="OK"):
        self.name, self.state = name, state

class CompositeAlarm:
    def __init__(self, name, rule):
        self.name, self.rule = name, rule  # rule(states: dict) -> bool

    def evaluate(self, children):
        states = {c.name: c.state for c in children}
        return "ALARM" if self.rule(states) else "OK"

children = [
    MetricAlarm("lambda-errors", "ALARM"),
    MetricAlarm("sqs-dlq-depth", "OK"),
    MetricAlarm("rds-cpu", "ALARM"),
]

# Page only when BOTH an application error signal AND a resource-pressure signal agree
rule = lambda s: s["lambda-errors"] == "ALARM" and s["rds-cpu"] == "ALARM"
composite = CompositeAlarm("checkout-degraded", rule)
print("checkout-degraded:", composite.evaluate(children))

children[2].state = "OK"
print("checkout-degraded:", composite.evaluate(children))

for c in children:
    print(f"  {c.name:<16} {c.state}")
Output
checkout-degraded: ALARM
checkout-degraded: OK
  lambda-errors    ALARM
  sqs-dlq-depth    OK
  rds-cpu          OK
Scenario A

In your world

The situation

Your ETL pipeline has one CloudWatch alarm per Lambda function—12 alarms on error count, each with the default missing-data treatment. On-call gets paged by whichever alarm's metric goes quiet, including functions that only run once a night.

What actually breaks

Every night, three of the twelve alarms flip to INSUFFICIENT_DATA between runs and page on-call, because those functions emit no metric data when idle and the default missing treatment surfaces that as alarm-worthy. On-call has stopped looking at pages from this pipeline—including the one night a function paged for a genuine ALARM, buried in the same noise.

What you'd do on Monday

Set notBreaching on the three idle-between-runs functions, keep breaching on anything that should emit continuously, and wrap the pipeline's error alarms in one composite alarm that only pages when more than one agrees something is wrong.

PythonSet missing-data treatment and build a composite alarm (needs boto3)
import boto3

cw = boto3.client("cloudwatch", region_name="ap-south-1")

cw.put_metric_alarm(
    AlarmName="orders-loader-errors",
    Namespace="AWS/Lambda", MetricName="Errors",
    Dimensions=[{"Name": "FunctionName", "Value": "orders-loader"}],
    Statistic="Sum", Period=300, EvaluationPeriods=3, Threshold=0,
    ComparisonOperator="GreaterThanThreshold",
    TreatMissingData="notBreaching",   # idle between nightly runs is expected
    AlarmActions=["arn:aws:sns:ap-south-1:123456789012:etl-alerts"],
)

cw.put_composite_alarm(
    AlarmName="orders-pipeline-degraded",
    AlarmRule="ALARM(orders-loader-errors) AND ALARM(orders-rds-cpu-high)",
    AlarmActions=["arn:aws:sns:ap-south-1:123456789012:oncall-page"],
)
Scenario B

Imagine you work at Postman

60 microservices, 300+ metric alarms

The numbers

Illustrative: 60 services × ~5 alarms each ≈ 300 metric alarms. If even 2% flap between OK and INSUFFICIENT_DATA on a quiet night, that's 6 alarms paging for nothing—enough that on-call starts triaging "which alarm is this" before "is this real."

What the Scenario-A answer does here

The Scenario A fix (correct missing-data treatment, composite alarms) still works per-service, but at 300 alarms hand-reviewing each one doesn't scale, and one composite per pipeline isn't enough structure for 60 independently owned services.

What flips at this scale

Missing-data treatment and composite-alarm structure become part of the service template, not a per-incident fix: every new service's IaC module ships a correct default and a required composite "service health" alarm before it can page a shared rotation. Alarm hygiene becomes a reviewed interface contract, not something each team reinvents.

The trade-off you'd defend

I would defend a mandatory, templated alarm structure over trusting each team to configure missing-data treatment correctly, trading some autonomy for a shared on-call rotation that doesn't drown in avoidable noise.

Illustrative counts; not Postman-published alarm inventories.

Alarm structures

ApproachNoise at scaleSetup costDebuggabilityBest for
One alarm per metricHighLowHigh (clear cause)Small single-service pipelines
Composite alarm (AND rule)LowMediumMedium (must check children)Correlated failure signals (errors + resource pressure)
Anomaly-detection alarmMediumMediumLow (threshold is a learned band)Metrics with no fixed healthy value (seasonal traffic)
Log alarm on Logs Insights queryLow–MediumMediumHigh (query shows exact matches)Pattern/error detection not exposed as a metric

Gotchas

Idle function pages every quiet period

Symptom

An alarm on a nightly-only Lambda flips to INSUFFICIENT_DATA and pages on-call every day between runs.

Cause

The default missing treatment surfaces an expected quiet period as insufficient data.

Fix

Set TreatMissingData to notBreaching for metrics legitimately silent outside a known window.

Twelve alarms, one real incident, twelve pages

Symptom

On-call gets paged a dozen times for one root cause (a hot database causing errors everywhere it's touched).

Cause

Every metric alarm pages independently with no aggregation.

Fix

Wrap correlated alarms in a composite alarm with an AND rule, and page on the composite, not every child.

"It works" demo, "it survives" never checked

Symptom

A Well-Architected review surfaces the same five reliability gaps every quarter, and none get fixed.

Cause

The review has no enforcement mechanism; it only produces a list unless leadership commits time against it.

Fix

Treat high-risk items as backlog work with an owner and a date, not a report to file away.

Tricky questions

If a metric alarm's missing-data setting is ignore, what happens when all 5 of the last 5 data points are missing?
Most people say

It goes to OK, since there's no bad data.

Actually

It retains whatever state it was already in—ignore means "don't change your mind". If it was already in ALARM, it stays there silently.

Does a composite alarm replace the need for its underlying metric alarms?
Most people say

Yes, once the composite pages correctly you can mute the children.

Actually

No. The composite only reaches ALARM on its rule, but the children still carry the diagnostic detail. Muting them removes the audit trail you need to actually fix the incident.

Scenario question

You're handed a pipeline where every one of eight CloudWatch alarms independently pages on-call, and the team has started a "mute everything, check the dashboard manually" workaround. How do you redesign without losing root-cause diagnosis?

Model answer

I'd keep all eight metric alarms for diagnosis, but stop paging on each individually. I'd group them by failure mode—four indicating database pressure, four indicating application errors—and build one composite alarm per group with an OR rule, paging on the composite. I'd also audit each alarm's missing-data treatment, since idle-period false pages are a common cause of this fatigue. The trade-off is slightly slower triage, since the page says "database pressure" not the exact metric, but the on-call engineer opens the composite's child states in one click, and nobody is muting the whole system anymore.

Interview corner

CommonWhat are the three states a CloudWatch alarm can be in?
“I'd say OK, ALARM, and INSUFFICIENT_DATA. The third one trips people up—it just means CloudWatch didn't have enough real data to decide, and what happens next depends on the missing-data setting.”
TrickAn alarm with TreatMissingData=ignore has been silent for two days. What state is it in right now?
“I'd say whatever it was in before data stopped. ignore means hold the current state, not reset to OK—if it was already in ALARM, it's sitting there silently, which is dangerous if nobody's watching.”
Experience checkHow did you reduce alert fatigue on a pipeline with too many independent alarms?
“I grouped alarms by what they actually indicate, built composite alarms per group, and paged on the composite instead of every child. I also fixed missing-data treatment, since idle-period false pages were half the noise.”
DesignHow would you turn a Well-Architected review into something that actually changes the system?
“I'd treat every high-risk finding as a ticket with an owner and a target sprint, not a filed report. A review with no enforcement is just a to-do list until someone's accountable for it.”

Memory hook

A smoke alarm with the battery pulled is worse than no alarm, because you stop checking it—CloudWatch's missing-data setting is you deciding, in advance, what silence means.

If you remember only one thing:

Page on composite alarms built from correlated signals; keep individual metric alarms for diagnosis, not for waking someone up.

Self-check

  1. What are the four values you can set for how a CloudWatch alarm treats missing data?

    missing (default), notBreaching, breaching, and ignore.

  2. What is the difference between a metric alarm and a composite alarm?

    A metric alarm watches one metric or math expression against a threshold. A composite alarm combines the states of other alarms with a rule expression and only alarms when that rule is satisfied.

  3. Name the six pillars of the AWS Well-Architected Framework.

    Operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability.

Prove it

Extend the composite-alarm example to a three-child AND rule (errors, DLQ depth, and CPU), and show the eight possible combinations of child states with the resulting composite state printed for each. Confirm the composite is ALARM for exactly one of those eight combinations.

aws-09-03Card 3 of 6

Backup strategy and restore evidence

HardP3 · Role-dependentTalk about it11 min
aws-backupretentionvault-lockpoint-in-time-recoveryrestore-testing

Why you care

Every TCS-style audit asks "do you have backups," and the honest answer for most teams is "yes, and we have never restored one." A green nightly backup job tells you the backup ran; it tells you nothing about whether the data is usable, whether you have the right key to decrypt it, or whether your application can come back up pointed at it. Interviewers ask because "we take backups" is table stakes and "we have tested a restore and timed it" separates someone who has operated a production database from someone who has only configured one.

Explain like I'm 5

A backup is a photocopy locked in a different building. Taking it every night and never reading it back isn't the same as having a usable copy—maybe the copier jammed, maybe the lock needs a key nobody can find at 3 a.m. The only way to know the copy works is to walk over, unlock it, and read the page.

Core concept

AWS Backup is the managed, cross-service way to do this instead of per-service snapshot scripts. The pieces:

  • A backup vault stores backups separate from the source, with its own key and access policy—so a compromised credential that can delete your database usually can't delete the vault too.
  • A backup plan defines schedule, lifecycle and which resources it applies to, usually via tags.
  • Vault Lock turns a vault write-once-read-many: once locked, nobody—including account root—can shorten retention or delete backups early. This survives an insider threat or compromised credential, which a deletable backup doesn't.
  • Cross-Region/account copy puts a copy somewhere a single-Region disaster can't reach.
  • Coverage varies: AWS Backup directly manages RDS/Aurora, DynamoDB, EBS, EFS, Redshift and S3. S3 needs Versioning enabled, and comes as continuous backups (35-day point-in-time restore, depends on an EventBridge rule) or periodic snapshots (up to 99-year retention). CRR with versioning is a separate, complementary region-level pattern, not the S3 backup mechanism itself.

Backups are not replication—conflating the two is the single most common DR mistake. Replication gives a near-real-time copy with low RPO, but faithfully replicates mistakes too: delete a row and replication carries that damage within seconds. A backup is deliberately behind; that lag is the feature. A real strategy needs both: replication for availability, backups for recovering from corruption or deletion.

Point-in-time recovery (PITR), where supported (RDS, Aurora, DynamoDB), restores to any second within the retention window, not just the last snapshot—useful when you don't know exactly when the bad write happened.

Decision rule: back up anything you can't rebuild from an upstream source; for anything you can rebuild (a derived S3 layer regenerated from raw data), invest in the raw layer's durability instead. Either way, restore testing isn't optional—an untested backup is a hypothesis, not a plan.

Under the hood

The first backup of a resource is a full copy; every one after stores only the delta, but still represents a complete, independently restorable point in time—AWS Backup reassembles the chain for you, so deleting an earlier increment is safe (later increments keep what they need), unlike a hand-rolled incremental scheme. Restore time isn't proportional to recency either, since reassembly happens on AWS's side regardless of which point you pick—so RTO testing has to measure actual restore time for your resource size, not assume "newest restores fastest."

Minimal example Replication carries damage; a backup predates it

PythonA deleted row survives in yesterday's backup, not in the live replicaruns
import copy

primary = {"orders": {"ORD-1": 500, "ORD-2": 300, "ORD-3": 150}}
replica = copy.deepcopy(primary)          # continuous replication: near-instant mirror
backup_from_yesterday = copy.deepcopy(primary)  # point-in-time snapshot, taken before today's mistake

# Today: an operator fat-fingers a DELETE with no WHERE clause narrow enough
del primary["orders"]["ORD-2"]
replica = copy.deepcopy(primary)          # replication faithfully propagates the deletion

print("primary:", primary["orders"])
print("replica (replication):", replica["orders"])
print("backup (point-in-time):", backup_from_yesterday["orders"])
print("ORD-2 recoverable from replica:", "ORD-2" in replica["orders"])
print("ORD-2 recoverable from backup:", "ORD-2" in backup_from_yesterday["orders"])
Output
primary: {'ORD-1': 500, 'ORD-3': 150}
replica (replication): {'ORD-1': 500, 'ORD-3': 150}
backup (point-in-time): {'ORD-1': 500, 'ORD-2': 300, 'ORD-3': 150}
ORD-2 recoverable from replica: False
ORD-2 recoverable from backup: True

In production Restore-test harness that verifies, not just restores

PythonA backup job that only "succeeds" if the restored data checks outruns
import sqlite3
import hashlib

def take_backup(db):
    rows = db.execute("SELECT order_id, amount FROM orders ORDER BY order_id").fetchall()
    checksum = hashlib.sha256(repr(rows).encode()).hexdigest()[:12]
    return {"rows": rows, "checksum": checksum, "row_count": len(rows)}

def restore_backup(backup):
    restored = sqlite3.connect(":memory:")
    restored.execute("CREATE TABLE orders (order_id TEXT PRIMARY KEY, amount REAL)")
    restored.executemany("INSERT INTO orders VALUES (?, ?)", backup["rows"])
    restored.commit()
    return restored

def verify_restore(restored, backup):
    rows = restored.execute("SELECT order_id, amount FROM orders ORDER BY order_id").fetchall()
    checksum = hashlib.sha256(repr(rows).encode()).hexdigest()[:12]
    count_ok = len(rows) == backup["row_count"]
    checksum_ok = checksum == backup["checksum"]
    return count_ok and checksum_ok, len(rows), checksum_ok

source = sqlite3.connect(":memory:")
source.execute("CREATE TABLE orders (order_id TEXT PRIMARY KEY, amount REAL)")
source.executemany("INSERT INTO orders VALUES (?, ?)",
                    [("ORD-1", 500.0), ("ORD-2", 300.0), ("ORD-3", 150.0)])
source.commit()

backup = take_backup(source)
print(f"backup taken: {backup['row_count']} rows, checksum_len={len(backup['checksum'])}")

restored_db = restore_backup(backup)
ok, restored_count, checksum_ok = verify_restore(restored_db, backup)
print(f"restore verified: {ok}  (rows={restored_count}, checksum_match={checksum_ok})")

# Simulate a corrupted restore (one row dropped) to show the check catching it
bad_backup = dict(backup)
bad_backup["rows"] = backup["rows"][:-1]
bad_restore = restore_backup(bad_backup)
ok2, restored_count2, checksum_ok2 = verify_restore(bad_restore, backup)
print(f"corrupted restore verified: {ok2}  (rows={restored_count2}, checksum_match={checksum_ok2})")
Output
backup taken: 3 rows, checksum_len=12
restore verified: True  (rows=3, checksum_match=True)
corrupted restore verified: False  (rows=2, checksum_match=False)
Scenario A

In your world

The situation

Your Postgres database backs up nightly via an RDS automated snapshot. The runbook says "backups are enabled" and the Jenkins dashboard has shown a green "snapshot complete" tile every morning for eight months.

What actually breaks

A bad migration script drops a column and backfills it incorrectly on production. Someone finally tries to restore last night's snapshot to check the pre-migration state—and discovers the restored instance takes 40 minutes to become available, nobody knows the correct subnet group and parameter group to attach it to, and the "restore" the team actually needed was to a specific point five hours before the migration ran, not to the nightly snapshot boundary, which was already after the bad migration.

What you'd do on Monday

Enable point-in-time recovery (available on RDS/Aurora) so you can restore to any second within the retention window, not just the nightly snapshot time. Document and periodically rehearse the exact restore procedure—including which subnet group, parameter group and security group the restored instance needs—so the first real restore isn't also the first rehearsal.

PythonRestore an RDS instance to a specific point in time (needs boto3)
import boto3

rds = boto3.client("rds", region_name="ap-south-1")
rds.restore_db_instance_to_point_in_time(
    SourceDBInstanceIdentifier="orders-prod",
    TargetDBInstanceIdentifier="orders-restore-test",
    RestoreTime="2026-01-15T02:30:00+05:30",   # 5 hours before the bad migration ran
    DBSubnetGroupName="orders-prod-subnet-group",
    VpcSecurityGroupIds=["sg-0123456789abcdef0"],
)
Scenario B

Imagine you work at Blinkit

1,200 databases across tenants, nightly backup window

The numbers

Illustrative: 1,200 database instances, each with a 7-day point-in-time retention window and a daily snapshot. A full restore-test rehearsal of one database averaging 40 minutes means testing all 1,200 sequentially would take 800 hours—clearly nobody is manually restore-testing every tenant database every month at this count.

What the Scenario-A answer does here

The Scenario A fix (enable PITR, document and rehearse restore) is correct per database, but "rehearse the restore procedure" doesn't scale to 1,200 instances if rehearsal means a human clicking through a runbook each time.

What flips at this scale

At this scale, restore testing has to be automated and sampled, not manual and exhaustive: a scheduled job restores a random sample (say, 5% of tenants per week) to a scratch environment, runs the same checksum-and-row-count verification as any other automated test, and alarms if a restore fails or exceeds a time budget—so every tenant gets tested on a rolling basis without a human doing it by hand.

The trade-off you'd defend

I would defend automated, sampled restore testing with alarms over either skipping restore testing at this scale or trying to rehearse every tenant manually, trading full per-tenant coverage on any given week for continuous, unattended coverage across the whole fleet over time.

Illustrative tenant counts and timings; not Blinkit-published figures.

Gotchas

Replication is mistaken for a backup strategy

Symptom

A post-incident review finds the "backup" everyone pointed to was actually a read replica, faithfully containing the same bad delete.

Cause

Replication has near-zero RPO but propagates corruption and deletion just as fast as good writes.

Fix

Maintain point-in-time backups (and, for insider threats, a locked vault) separately from any availability replication.

Nightly snapshot boundary doesn't match when the mistake happened

Symptom

The only restore point is already after the bad write, so "restoring from backup" doesn't undo the damage.

Cause

A single daily snapshot gives one point in time, not a window.

Fix

Enable point-in-time recovery where supported, so you can restore to any second in the retention window.

A green backup job with an unusable restore

Symptom

The nightly backup has shown "success" for months; the first real restore fails or produces an app that can't start.

Cause

A successful backup job proves data was written somewhere—nothing about whether the restored resource has the right network or key access to serve traffic.

Fix

Schedule automated restore-and-verify drills on a sample of resources, not just completion checks.

Tricky questions

Does S3 Cross-Region Replication count as your S3 backup strategy?
Most people say

Yes, it's a copy in another Region, so data is protected.

Actually

AWS Backup has natively supported S3 since 2022—continuous backups (35-day PITR) and periodic snapshots (up to 99 years), both requiring S3 Versioning. CRR is a separate, complementary pattern for region-level availability, not a substitute for an actual AWS Backup plan.

Does restoring an RDS snapshot restore the exact same instance configuration?
Most people say

Yes, a restore recreates the instance identically, including network and security settings.

Actually

A restore creates a new instance from the data; you must reattach the subnet group, security groups and parameter group yourself—those are restore-time choices, not data in the snapshot.

Scenario question

You're handed a Postgres service with nightly RDS snapshots and told "we're backup-compliant." Leadership wants a one-sentence answer to "can we recover from a bad deploy that silently corrupted data for six hours before anyone noticed?" What do you check, and what's your answer?

Model answer

I'd check three things: whether PITR is enabled (nightly-only means the restore boundary may already be after the corruption started), whether anyone has actually performed and verified a restore versus just seeing "snapshot complete," and how long a real restore takes end to end versus the six-hour window. If PITR is enabled and tested, my answer is we can likely recover to just before corruption with a known RTO; if either is missing, my answer is "not yet—we have backups, not a proven recovery," and that gap is next sprint's work.

Interview corner

CommonWhat is the difference between a backup and replication?
“I'd say replication gives a near-real-time copy for availability, but propagates mistakes just as fast as good writes. A backup is deliberately a point in the past, which is what lets you recover from corruption replication would have already carried over.”
TrickYour nightly backup job has shown green for eight months. Is your data safe?
“Not provably. Green tells me the backup ran, not that it restores into something usable. I'd want evidence of at least one actual restore-and-verify.”
Experience checkHow would you add confidence to a backup strategy without manually restoring everything?
“I'd automate restore testing on a rolling sample—a scheduled job restoring a few resources a week, checking row counts and a checksum, alarming on failure. That's continuous fleet coverage without anyone restoring a thousand databases by hand.”
DesignWhen would you reach for AWS Backup Vault Lock?
“I'd reach for it whenever the threat model includes someone with legitimate delete permissions doing damage—a compromised credential, an insider, a bad script. Vault Lock makes retention and backups immutable even against account root.”

Memory hook

A backup is a photocopy in another building; it only means something once you've walked over, unlocked the door, and read the page back.

If you remember only one thing:

A backup you've never restored is a hypothesis. Schedule the restore test, not just the backup job.

Self-check

  1. Why does a point-in-time backup protect against data corruption in a way replication doesn't?

    Replication mirrors changes near-instantly, so it propagates corruption or deletion along with good writes. A point-in-time backup is deliberately from before the mistake, giving you a clean version to restore to.

  2. What does AWS Backup Vault Lock prevent, and who can it block?

    It prevents anyone—including the account root—from deleting backups or shortening their retention period before they expire, enforcing a write-once-read-many model.

  3. What does a "green" backup job actually prove, and what doesn't it prove?

    It proves the backup process completed and wrote data somewhere. It does not prove the data is restorable, uncorrupted, or that the restored resource can actually come up and serve traffic.

Prove it

Extend the production example's restore-test harness to a second resource type (a key-value store represented as a dict) with its own checksum function, run both verifications in one script, and print a single pass/fail summary line. Then deliberately corrupt one row in the key-value backup before restoring and show the harness catching it.

aws-09-04Card 4 of 6

Disaster recovery

HardP3 · Role-dependentTalk about it11 min
rtorpopilot-lightwarm-standbyactive-activefailover

Why you care

"What's your RTO and RPO?" is one of the few interview questions where a wrong-sounding honest answer ("we don't have a tested one") beats a confident invented one, since the follow-up always exposes which is true. This is also where a Jenkins-era ETL developer's instincts mislead: "we have backups" and "we have a disaster recovery plan" sound like the same sentence and aren't—the gap is hours of downtime during a real outage, not a design doc nobody has run.

Explain like I'm 5

RTO is "how long can the shop stay closed before it matters." RPO is "how much of today's till can we afford to lose if the register breaks." A spare till never plugged in (pilot light) reopens slower than one already running next door (warm standby), slower than two shops already open and splitting customers (active/active). Each option costs more than the last; the right choice depends on how expensive "closed" is for your business.

Core concept

RTO is the max acceptable time between a disaster and the workload serving traffic again. RPO is the max acceptable data loss, measured in time. Both are business decisions dressed as engineering numbers.

AWS's four DR strategies, increasing cost, decreasing RTO/RPO:

  • Backup and restore. Data backed up (ideally cross-Region); on disaster, redeploy from IaC and restore. Nothing runs in the DR Region day to day. Cheapest, slowest—often hours.
  • Pilot light. Core data stores replicate continuously; everything else deployed but switched off, turned on and scaled up on disaster. Faster since the data layer is already warm; still needs action first.
  • Warm standby. A scaled-down but fully functional copy runs continuously; you only scale up. Pilot light can't serve a request without intervention; warm standby serves immediately at reduced capacity.
  • Multi-site active/active. Full capacity in more than one Region, splitting real traffic. No "failover"—losing one Region just means routing away. Fastest, most expensive, hardest due to write consistency.

Traffic failover: Route 53 health checks redirect DNS automatically or act as a manual switch via Application Recovery Controller; Global Accelerator and CloudFront are alternatives. Prefer data plane operations over control plane ones for failover—an Auto Scaling call during the exact outage stressing the Region depends on the thing most likely to also be struggling.

Decision rule: pick the cheapest strategy meeting a written, business-agreed RTO/RPO, proven by a timed drill. Backup and restore often suffices for data loss or a single-AZ disruption; the other three earn their cost only when "disaster" means losing a whole Region or a regulatory requirement.

Under the hood

Promoting an Aurora global database secondary, or flipping a Route 53 health check via Application Recovery Controller, are data-plane operations—built for the exact moment things are failing. Calling Auto Scaling to raise capacity, or redeploying a stack, are control-plane operations, dependent on the orchestration layer being healthy during the same event testing your resilience. A warm-standby design that quietly depends on an Auto Scaling call has taken on that dependency exactly when it's least reliable; a "hot standby" already at full capacity avoids it, at the cost of paying for that capacity continuously.

Minimal example RTO/RPO arithmetic for four strategies

PythonModeling relative recovery time and data loss across the four DR strategiesruns
STRATEGIES = [
    {"name": "backup-restore", "infra_deploy_min": 45, "restore_min": 30, "replication_lag_min": 1440},
    {"name": "pilot-light",    "infra_deploy_min": 10, "restore_min": 5,  "replication_lag_min": 5},
    {"name": "warm-standby",   "infra_deploy_min": 2,  "restore_min": 0,  "replication_lag_min": 1},
    {"name": "active-active",  "infra_deploy_min": 0,  "restore_min": 0,  "replication_lag_min": 0},
]

for s in STRATEGIES:
    rto = s["infra_deploy_min"] + s["restore_min"]
    rpo = s["replication_lag_min"]
    print(f"{s['name']:<16} estimated RTO={rto:>3} min   estimated RPO={rpo:>5} min")
Output
backup-restore   estimated RTO= 75 min   estimated RPO= 1440 min
pilot-light      estimated RTO= 15 min   estimated RPO=    5 min
warm-standby     estimated RTO=  2 min   estimated RPO=    1 min
active-active    estimated RTO=  0 min   estimated RPO=    0 min

In production Choosing a strategy against a business-stated RTO/RPO

PythonPick the cheapest strategy that meets stated targets, with the over-insurance costruns
STRATEGIES = [
    {"name": "backup-restore", "rto_min": 75,  "rpo_min": 1440, "relative_cost": 1},
    {"name": "pilot-light",    "rto_min": 15,  "rpo_min": 5,    "relative_cost": 3},
    {"name": "warm-standby",   "rto_min": 2,   "rpo_min": 1,    "relative_cost": 6},
    {"name": "active-active",  "rto_min": 0,   "rpo_min": 0,    "relative_cost": 12},
]

def meets(s, target_rto, target_rpo):
    return s["rto_min"] <= target_rto and s["rpo_min"] <= target_rpo

def cheapest_meeting(target_rto, target_rpo):
    candidates = [s for s in STRATEGIES if meets(s, target_rto, target_rpo)]
    return min(candidates, key=lambda s: s["relative_cost"]) if candidates else None

TARGETS = [(80, 1440), (20, 10), (5, 2), (0, 0), (3, 10)]
for target_rto, target_rpo in TARGETS:
    choice = cheapest_meeting(target_rto, target_rpo)
    if choice is None:
        print(f"target RTO<={target_rto:>3}min RPO<={target_rpo:>4}min -> NONE MEET TARGET")
        continue
    overpay = choice["relative_cost"] - 1  # relative cost above the cheapest strategy of all
    print(f"target RTO<={target_rto:>3}min RPO<={target_rpo:>4}min -> {choice['name']:<14} (+{overpay} relative cost over backup-restore)")
Output
target RTO<= 80min RPO<=1440min -> backup-restore (+0 relative cost over backup-restore)
target RTO<= 20min RPO<=  10min -> pilot-light    (+2 relative cost over backup-restore)
target RTO<=  5min RPO<=   2min -> warm-standby   (+5 relative cost over backup-restore)
target RTO<=  0min RPO<=   0min -> active-active  (+11 relative cost over backup-restore)
target RTO<=  3min RPO<=  10min -> warm-standby   (+5 relative cost over backup-restore)
Scenario A

In your world

The situation

Your team's Postgres database runs in one AWS Region with nightly RDS snapshots. There's no documented RTO or RPO; honestly it's "whatever restoring a snapshot and redeploying from Jenkins takes, which nobody has timed."

What actually breaks

During a scheduled DR tabletop (not a real outage, which is itself telling), the team walks through what a Region-level event would require: redeploy infrastructure from scratch since there's no IaC, find the latest snapshot, restore it, and reattach every security group from memory because none of it is written down. The walkthrough alone takes four hours, with no real outage pressure yet.

What you'd do on Monday

Treat backup-and-restore as the deliberate strategy for this risk profile, but make it real: put infrastructure in IaC, write and rehearse the restore runbook, and agree an actual RTO/RPO with the business so "whatever it takes" becomes a tested commitment.

PythonMinimal cross-Region backup copy as the first real DR investment (needs boto3)
import boto3

backup = boto3.client("backup", region_name="ap-south-1")
backup.start_copy_job(
    RecoveryPointArn="arn:aws:rds:ap-south-1:123456789012:snapshot:orders-prod-2026-01-15",
    SourceBackupVaultName="orders-prod-vault",
    DestinationBackupVaultArn="arn:aws:backup:eu-west-1:123456789012:backup-vault:orders-dr-vault",
    IamRoleArn="arn:aws:iam::123456789012:role/aws-backup-copy-role",
)
Scenario B

Imagine you work at Cult.fit

50,000 bookings/day, one Region

The numbers

Illustrative: a booking platform at roughly 50,000 bookings/day across one Region. A 75-minute backup-and-restore RTO, at a conservative 30 bookings/minute lost, is on the order of 2,250 missed bookings before service is back.

What the Scenario-A answer does here

The Scenario A fix (IaC, a rehearsed runbook, an agreed RTO/RPO) makes backup-and-restore real and tested instead of a hope. But at this rate, a 75-minute RTO may simply not meet what the business needs once asked directly.

What flips at this scale

Once the business states an RTO under roughly 15 minutes, backup-and-restore mathematically cannot meet it. The flip is strategic: move to pilot light at minimum, with the booking database replicating continuously to a second Region, removing the restore step from the critical path.

The trade-off you'd defend

I would defend pushing the business to accept a longer, truthful RTO or fund pilot light, over quietly promising a 15-minute RTO a backup-and-restore architecture can't deliver.

Illustrative booking volumes and RTO figures; not Cult.fit-published numbers.

The four DR strategies

StrategyWhat runs in DR Region normallyRelative RTORelative RPORelative costFailover action needed
Backup and restoreNothingHoursUp to backup intervalLowestDeploy infra from IaC, restore data
Pilot lightCore data stores only, replicatingTens of minutesMinutesLow–medium"Switch on" and scale application tier
Warm standbyFull stack, scaled downMinutesSeconds–minutesMedium–highScale up to full capacity
Multi-site active/activeFull stack, serving live trafficNear zeroNear zero (writes), nonzero (corruption)HighestRoute traffic away; no failover step

Gotchas

"We have a DR plan" means a document, not a tested capability

Symptom

A DR runbook exists and reads well, but the first time anyone executes it is during a real outage.

Cause

DR plans are written once and never rehearsed, so unstated assumptions only surface under pressure.

Fix

Schedule a recurring timed drill—even a tabletop counts as a start—and update the runbook with every gap found.

Control-plane dependency hides inside a "simple" failover step

Symptom

Failover works in testing but stalls during the actual regional event it was built for.

Cause

A failover step depends on a control-plane API with a lower availability target than the data-plane operation it should have used.

Fix

Prefer data-plane failover (a health-check switch via Application Recovery Controller, a database promotion) wherever the RTO is tight.

Multi-Region replication is confused with multi-Region DR

Symptom

A team believes they have regional DR because their database replicates cross-Region, with no plan for the application tier, DNS, or IAM there.

Cause

Replication is one ingredient of a DR strategy, not the strategy itself.

Fix

Walk every layer (data, compute, network, DNS, IAM) and confirm each has an explicit DR Region story.

Tricky questions

Does having a cross-Region read replica mean you already have disaster recovery?
Most people say

Yes, if the primary Region goes down, you can just use the replica.

Actually

A replica gives replicated data, not a recovery plan. Promoting it, redeploying the application tier, and repointing DNS/IAM are all separate work.

Is multi-site active/active always the "best" DR strategy since it has near-zero RTO/RPO?
Most people say

Yes, since it eliminates failover entirely, it's strictly better.

Actually

It's the most expensive, hardest strategy, with write-consistency problems simpler ones don't have. "Best" depends on what RTO/RPO the business needs and will pay for—most don't need it.

Scenario question

You're handed a payments-adjacent service informally stated as "can't be down more than an hour," running single-Region with nightly snapshots and no DR Region. How do you turn that into an actual plan?

Model answer

I'd first get that turned into explicit RTO and RPO numbers agreed with the business, since "down" conflates both. If RTO ≤ 60 minutes, I'd rule out backup-and-restore—redeploying and restoring rarely finishes inside an hour with margin. I'd propose pilot light: continuous replication to a second Region, infrastructure in IaC switched-off there, a tested runbook to scale up. I'd insist on a timed drill before calling the RTO "met," since an untested number is a guess with better formatting. The trade-off: pilot light costs more than today's setup, and that's the real price of the promise already made.

Interview corner

CommonWhat is the difference between RTO and RPO?
“I'd say RTO is how long you can be down before it matters. RPO is how much data you can afford to lose, measured in time. A strategy can be strong on one and weak on the other.”
TrickYour database has a cross-Region read replica. Do you have disaster recovery?
“I have replicated data, not disaster recovery. I still need a plan for promoting it, standing up the application tier, and verifying IAM there—the replica is one ingredient, not the whole dish.”
Experience checkWalk me through choosing a DR strategy for a real workload.
“I'd get an actual RTO/RPO number from the business, pick the cheapest of the four strategies that mathematically meets it, and insist on a timed drill before calling it done—an untested RTO is just a guess.”
DesignWhy would you prefer data-plane operations for failover over control-plane ones?
“Control planes and data planes have different availability design targets, and I'd rather not depend on the orchestration layer being healthy during the exact event testing my resilience. A database promotion or a Route 53 health-check switch is data-plane; redeploying a stack or calling Auto Scaling is control-plane, and I want that off the critical path for a tight RTO.”

Memory hook

A spare till in the back room, never plugged in, reopens the shop slower than one already running at the next counter—pick based on what "closed" actually costs you, not on which option sounds most resilient.

If you remember only one thing:

RTO and RPO are business numbers first; the strategy is just whichever of the four options is the cheapest one that provably meets them.

Self-check

  1. What's the practical difference between pilot light and warm standby?

    Pilot light can't serve a request without turning something on first; warm standby is already running and can serve traffic immediately, just at reduced capacity.

  2. Why does AWS recommend data-plane operations over control-plane operations during failover?

    Data planes are generally built to higher availability design goals than control planes, so depending on a control-plane call (like Auto Scaling) during the exact event stressing the Region adds a dependency on something less likely to be reliable at that moment.

  3. Why is a cross-Region replica alone not a complete DR strategy?

    It only covers the data layer. Application infrastructure, traffic routing, DNS and IAM in the second Region all need their own explicit, tested plan.

Prove it

Using the production example's cheapest_meeting function, add a fifth, hypothetical strategy between warm-standby and active-active with its own RTO/RPO/cost figures, and show at least one target RTO/RPO pair where it—not warm-standby or active-active—is now the cheapest strategy that qualifies. Print the full strategy table sorted by relative cost.

aws-09-05Card 5 of 6

Game days and failure injection

HardP4 · DifferentiatorTalk about it11 min
fischaos-engineeringgame-daystop-conditionsfault-injection

Why you care

A DR strategy never exercised against a real-ish failure is a theory. A game day is the deliberate, scoped version of "let's find out what breaks" instead of the accidental version where production finds out for you. Interviewers asking about chaos engineering aren't checking whether you've memorized a tool name—they're checking whether you understand the difference between breaking something safely, with a plan to stop, and breaking something by accident.

Explain like I'm 5

A fire drill works because everyone knows it's a drill, there's a planned route out, and someone is watching the clock to call it off. Randomly pulling the alarm during a workday to "see what happens" isn't a fire drill—it's chaos with no stop button. AWS Fault Injection Service is the fire-drill version: you write down exactly what you're going to break, for how long, and the exact condition that makes you call it off immediately.

Core concept

AWS Fault Injection Service (FIS) runs controlled experiments based on chaos-engineering principles: deliberately disrupt part of a real system to observe how it actually behaves, not how you assume it does.

  • An experiment template defines the actions (what fault to inject), the targets (resources, selected by tag or ID), and the stop conditions.
  • A stop condition is a CloudWatch alarm attached to the experiment; the moment it breaches, FIS automatically stops, wired to the same alarm infrastructure you already use for monitoring.
  • Actions run sequentially or in parallel, and FIS performs real actions on real resources—not a simulation layer—which is why stop conditions and a pre-production trial matter before running anything in production.

Game days follow a consistent shape: scope the failure precisely (not "break everything"), predict what you expect before running it, run with a stop condition armed and someone watching, measure the actual recovery clock against your stated RTO, and record evidence—the point is the gap between prediction and reality, useful only if written down.

Game days differ from genuine incidents in one way: you choose the blast radius. A permission-denial experiment against one non-critical IAM role teaches you about error handling safely; a network partition against your primary database's security group is far more dangerous if the stop condition fails to fire. Start narrow, in pre-production, and widen scope only once the stop-condition discipline is proven.

Decision rule: run game days against the specific failure modes your DR strategy (Disaster recovery) claims to handle—if pilot-light claims a 15-minute RTO for losing the primary database, test exactly that: fail it and time the promotion, not a generic "break something and see."

Under the hood

FIS stop conditions evaluate against the same CloudWatch alarm state machine as Alarms, SLOs and Well-Architected reviews, with higher stakes: a stop-condition alarm on the default missing treatment, watching a metric that goes quiet because the experiment just took down the thing emitting it, can sit in INSUFFICIENT_DATA instead of ALARM—and FIS only stops on ALARM. A game day's stop condition needs its missing-data behavior deliberately set to fail safe (usually breaching), not left on default, or the safety net goes quiet exactly when needed most.

Minimal example A stop condition that actually fires

PythonA simulated experiment that halts the instant its stop-condition alarm breachesruns
def alarm_state(value, threshold=5.0, treat_missing="breaching"):
    if value is None:
        return "ALARM" if treat_missing == "breaching" else "INSUFFICIENT_DATA"
    return "ALARM" if value > threshold else "OK"

ACTIONS = ["deny-s3-read", "inject-100ms-latency", "stop-one-instance", "partition-db-sg"]
METRICS = [0.5, 2.1, 7.8, None]  # metric goes missing once the DB is partitioned

for i, (action, value) in enumerate(zip(ACTIONS, METRICS), start=1):
    state = alarm_state(value)
    print(f"step {i}: {action} -> {state} (metric={value})")
    if state == "ALARM":
        print(f"STOPPED after step {i} of {len(ACTIONS)}")
        break
Output
step 1: deny-s3-read -> OK (metric=0.5)
step 2: inject-100ms-latency -> OK (metric=2.1)
step 3: stop-one-instance -> ALARM (metric=7.8)
STOPPED after step 3 of 4

In production Scoped experiment template with blast-radius checks

PythonA tiny FIS-shaped experiment runner that refuses to target more than its allowed scoperuns
class ExperimentTemplate:
    def __init__(self, name, actions, targets, stop_condition, max_targets):
        self.name, self.actions, self.targets = name, actions, targets
        self.stop_condition, self.max_targets = stop_condition, max_targets

    def validate(self):
        if len(self.targets) > self.max_targets:
            raise ValueError(
                f"{self.name}: {len(self.targets)} targets exceeds allowed blast radius of {self.max_targets}"
            )

    def run(self, metric_feed):
        self.validate()
        log = []
        for action, value in zip(self.actions, metric_feed):
            breached = value is not None and value > self.stop_condition["threshold"]
            log.append((action, value, "STOP" if breached else "ok"))
            if breached:
                break
        return log

narrow = ExperimentTemplate(
    name="deny-read-on-reporting-role",
    actions=["deny-iam-action"],
    targets=["role:reporting-readonly"],
    stop_condition={"threshold": 5.0},
    max_targets=1,
)
print(narrow.name, "->", narrow.run([1.2]))

too_wide = ExperimentTemplate(
    name="partition-every-db-security-group",
    actions=["network-partition"] * 12,
    targets=[f"sg-db-{i}" for i in range(12)],
    stop_condition={"threshold": 5.0},
    max_targets=1,
)
try:
    too_wide.run([0.0])
except ValueError as e:
    print("ValueError:", e)
Output
deny-read-on-reporting-role -> [('deny-iam-action', 1.2, 'ok')]
ValueError: partition-every-db-security-group: 12 targets exceeds allowed blast radius of 1
Scenario A

In your world

The situation

Your team wants to validate that "retry with backoff" in the loader Lambda actually survives a Postgres connection failure, instead of assuming it does because it's in a six-month-old code review.

What actually breaks

Nobody has ever broken the database on purpose to check. The only real test was a 3 a.m. incident, where the retry loop had no backoff cap and hammered the recovering database every 100ms, making the outage longer—discovered the hard way, with customers affected, instead of in a scoped drill.

What you'd do on Monday

Run a scoped game day: target one non-prod RDS security group, inject a network partition for a fixed short duration, with a stop-condition alarm on error rate set to breaching for missing data, and watch whether retry backs off correctly. Record what happened against what the code review assumed.

PythonFIS experiment template targeting one non-prod DB security group (needs boto3/AWS)
import boto3

fis = boto3.client("fis", region_name="ap-south-1")
fis.create_experiment_template(
    description="Game day: partition staging DB, verify Lambda retry/backoff behavior",
    actions={
        "partitionDb": {
            "actionId": "aws:network:disrupt-connectivity",
            "parameters": {"duration": "PT2M", "scope": "availability-zone"},
            "targets": {"Subnets": "staging-db-subnet"},
        }
    },
    targets={
        "staging-db-subnet": {
            "resourceType": "aws:ec2:subnet",
            "resourceTags": {"Environment": "staging", "Component": "orders-db"},
            "selectionMode": "ALL",
        }
    },
    stopConditions=[{"source": "aws:cloudwatch:alarm",
                      "value": "arn:aws:cloudwatch:ap-south-1:123456789012:alarm:staging-conn-errors-breaching"}],
    roleArn="arn:aws:iam::123456789012:role/fis-gameday-role",
)
Scenario B

Imagine you work at Darwinbox

40 microservices, quarterly game-day program

The numbers

Illustrative: 40 services × 2-3 plausible failure modes ≈ 100 candidate experiments. Running them manually at 2 hours each is 200 hours per quarter—over an engineer-month, which is why most teams give up after one round.

What the Scenario-A answer does here

The Scenario A fix (one scoped experiment with a stop condition) is correct per service, but running 100 by hand every quarter doesn't survive contact with real workloads; coverage degrades to "whichever three services someone remembered."

What flips at this scale

Game days need a program, not a one-off: experiment templates become reusable, version-controlled artifacts, a rotating subset runs automatically each week against a fixed budget, and stop-condition alarms are shared infrastructure reviewed like production alarms—coverage accumulates instead of depending on memory.

The trade-off you'd defend

I would defend a smaller number of experiments run consistently every week over a large manual quarterly push, trading the feeling of "we tested everything" for sustained coverage that doesn't collapse once the team gets busy.

Illustrative service counts and effort estimates; not Darwinbox-published figures.

Gotchas

Stop condition doesn't fire because its metric went missing, not breaching

Symptom

An experiment that should have auto-stopped kept running past the point where things clearly went wrong.

Cause

The stop-condition's missing-data treatment was left on the default, and the thing being broken stopped emitting the metric, so it went to INSUFFICIENT_DATA instead of ALARM.

Fix

Set the stop-condition's missing-data treatment to breaching whenever the experiment could plausibly silence the metric it watches.

Blast radius creeps during the experiment, not before it

Symptom

An experiment scoped to "one instance" ends up affecting a whole Availability Zone's traffic.

Cause

Tag-based target selection is a query, not a fixed list—resources matching at run time, not design time, get hit.

Fix

Validate the resolved target list against an expected count before the experiment runs, and fail closed if it doesn't match.

A game day becomes an actual incident because it ran in production untested

Symptom

An experiment fine in a trial run causes real customer impact at full production scope.

Cause

Fault-injection behavior doesn't always generalize from a smaller trial to full scope and load.

Fix

Trial every template in pre-production first, and widen scope incrementally rather than jumping to full scope.

Tricky questions

Is a stop condition a guarantee that an experiment can't cause an outage?
Most people say

Yes, that's the entire point of a stop condition.

Actually

It only stops once its alarm reaches ALARM. If missing-data treatment or threshold isn't tuned to the failure mode, it can fail to trip in time—it reduces risk, not eliminates it.

Does a successful game day mean the system is now resilient to that failure?
Most people say

Yes, if the experiment ran and nothing bad happened, resilience is proven.

Actually

It means the system was resilient to that failure, at that scope and load, on that day. Different time of day, traffic, or failure mode can behave differently—game days reduce uncertainty, not eliminate it.

Scenario question

You're handed a mandate to "start chaos engineering" on a system with no prior fault injection, no FIS experience on the team, and a production environment nobody wants to touch yet. Where do you start?

Model answer

Not in production, and not with FIS's most dramatic action. I'd pick one non-critical, non-production service, define a narrow experiment template with a stop condition tied to an alarm I've verified fires correctly (checking its missing-data treatment), and run it with someone watching live—something low-stakes like latency injection, not partitioning a database. I'd write the prediction before running it and treat the gap against reality as the deliverable, not whether anything broke. Only after that succeeds would I widen scope, and only much later consider production, once the stop-condition discipline has proven itself.

Interview corner

CommonWhat is the role of a stop condition in a chaos engineering experiment?
“I'd say it's a CloudWatch alarm wired into the experiment that automatically halts it the moment a threshold breaches—a real, automated off switch instead of relying on someone noticing.”
TrickA game day ran with a stop condition configured, but it didn't stop when things clearly went wrong. What likely happened?
“I'd check the alarm's missing-data treatment first. If the injected fault silenced the metric and treatment was left default, the alarm goes to INSUFFICIENT_DATA instead of ALARM, and FIS only stops on ALARM.”
Experience checkHow would you introduce game days to a team that's never run one?
“Small and non-production first—one narrow experiment, one low-stakes failure mode, a stop condition I've personally verified fires, and a written prediction to compare against the result. Early game days build the muscle and trust, not maximize coverage.”
DesignHow do you scale a game-day program across dozens of services without it collapsing into 'whoever remembers'?
“I'd make experiment templates reusable artifacts, run a rotating subset automatically on a fixed weekly budget rather than a quarterly push, and treat stop-condition alarms as shared infrastructure reviewed like production alarms. Coverage accumulates on a schedule, not memory.”

Memory hook

A fire drill works because someone announced it, planned the route out, and is watching the clock to call it off—chaos with no stop button is just an accident with better branding.

If you remember only one thing:

A stop condition is only as good as its missing-data treatment; verify it fires on the exact failure you're about to cause, not just in theory.

Self-check

  1. What three things does an AWS FIS experiment template define?

    Actions (what fault to inject), targets (which resources it applies to), and stop conditions (the CloudWatch alarm that halts the experiment).

  2. Why might a stop-condition alarm fail to stop an experiment even when something has clearly gone wrong?

    If its missing-data treatment is left on the default and the fault being injected silences the exact metric it watches, the alarm goes to INSUFFICIENT_DATA instead of ALARM, and FIS only halts on ALARM.

  3. Why should a new game-day program start in pre-production with a narrow scope?

    Because fault-injection behavior doesn't always generalize from a trial to full production scope and load, and the team's stop-condition discipline needs to be proven before the blast radius or environment gets riskier.

Prove it

Extend the production example's ExperimentTemplate class with a trial_run flag that, when True, forces max_targets to 1 regardless of the template's configured value. Show that the same too_wide template from the example now succeeds in trial mode (targeting only the first resource) and still raises in non-trial mode.

aws-09-06Card 6 of 6

DR and clone-release operations

HardP3 · Role-dependentTalk about it11 min
dr-drillrelease-clonerollbackaudit-evidencechange-management

Why you care

Every TCS-style shop has a "DR" or "clone" ticket process: clone a release config, point it at a different environment, run it, mark it done. That's real change-management discipline—and it isn't a disaster-recovery drill, even when the ticket says "DR." Confusing the two is how a team tells an auditor they've "tested DR quarterly" when what happened was a config clone with no measured recovery time, no failure injected, no proof the recovered system could serve traffic. This card exists so you can tell the two apart first.

Explain like I'm 5

A fire drill where everyone calmly walks to a marked exit while the building is fine is good practice, but it's not proof the building survives an actual fire—for that you'd need a contained fire and a timed exit. Cloning your release config into a second building while nothing is on fire is the first kind of drill, not the second, no matter what the sign-up sheet calls it.

Core concept

A release clone—duplicating a deployment's config, pointing it at a new target, and promoting it through the same pipeline—is a legitimate operation. It answers "can we repeat this deployment elsewhere." It does not answer "can we recover after an actual failure," because nothing failed and no clock measured recovery time.

A genuine DR drill has four things a release clone usually doesn't:

  • A real or simulated failure, not a clean start. It begins from "the primary is gone," not "deploy the same thing elsewhere while everything is fine."
  • A measured recovery clock, from simulated failure to verified traffic—against a stated RTO, as Disaster recovery defines it.
  • Data recovery, not just infrastructure. A clone ships fresh infra with no production data; a drill must prove the data comes back correct, using the backup/restore mechanics from Backup strategy and restore evidence, not a blank database.
  • Rollback and audit evidence as outcomes, not just pass/fail—what broke, the actual recovery time versus target, evidence an auditor can use.

Approvals and config parity matter for different reasons. A clone's approval gate stops an unreviewed change reaching a new environment; a drill's approval is a go/no-go on blast radius, scoped as deliberately as the game days in Game days and failure injection. Config parity is necessary and sufficient for a clone, but nowhere near sufficient for a drill—matching config says nothing about whether data recovery and the clock hold up.

Decision rule: before writing "DR tested" anywhere, ask whether the exercise included an actual failure, a measured clock against a stated RTO, and real data recovery. If any is missing, the honest label is "release/config clone verified"—still valuable, just not DR evidence.

Under the hood

A release-clone pipeline and a DR failover mechanism often share tooling, since both answer "deploy a known-good config to a target." That's exactly what makes them easy to conflate on a status report: pipeline logs look identical either way. The difference that never shows in logs is what happened to state: a clone's target typically starts empty, so success never proves a restore under time pressure. A genuine drill needs a second artifact the clone pipeline doesn't produce by default—a timed, verified data-recovery record.

Minimal example Same pipeline, different evidence

PythonA deployment event, scored for whether it is actually DR evidenceruns
def classify(failure, clock, data_recovered, cloned, deployed):
    if failure and clock is not None and data_recovered:
        return "DR_DRILL_EVIDENCE"
    if cloned and deployed:
        return "RELEASE_CLONE_ONLY"
    return "INSUFFICIENT_EVIDENCE"

EVENTS = [
    ("quarterly-dr-ticket", False, None, False, True, True),
    ("fis-db-partition-drill", True, 480, True, True, True),
]

for name, failure, clock, recovered, cloned, deployed in EVENTS:
    print(f"{name:<24} -> {classify(failure, clock, recovered, cloned, deployed)}")
Output
quarterly-dr-ticket      -> RELEASE_CLONE_ONLY
fis-db-partition-drill   -> DR_DRILL_EVIDENCE

In production Evidence log that distinguishes the two, with a rollback audit trail

PythonBuild an audit-ready evidence record for a DR drill vs a release cloneruns
import sqlite3
from datetime import datetime, timedelta

db = sqlite3.connect(":memory:")
db.execute("""
    CREATE TABLE ops_events (
        name TEXT, event_type TEXT, approved_by TEXT,
        started_at TEXT, recovered_at TEXT, rto_target_min INTEGER,
        data_verified INTEGER, rolled_back INTEGER
    )
""")

def log_event(name, event_type, approved_by, started_at, recovered_at,
              rto_target_min, data_verified, rolled_back=False):
    db.execute(
        "INSERT INTO ops_events VALUES (?,?,?,?,?,?,?,?)",
        (name, event_type, approved_by, started_at.isoformat(),
         recovered_at.isoformat() if recovered_at else None,
         rto_target_min, int(data_verified), int(rolled_back)),
    )
    db.commit()

start = datetime(2026, 1, 20, 2, 0)
log_event("prod-config-clone-eu", "release_clone", "release-mgr",
          start, start + timedelta(minutes=12), rto_target_min=None, data_verified=False)
log_event("rds-failover-drill", "dr_drill", "sre-lead",
          start, start + timedelta(minutes=18), rto_target_min=30, data_verified=True)
log_event("rds-failover-drill-rollback", "dr_drill", "sre-lead",
          start, start + timedelta(minutes=45), rto_target_min=30, data_verified=True, rolled_back=True)

for row in db.execute("SELECT name, event_type, started_at, recovered_at, rto_target_min, data_verified, rolled_back FROM ops_events"):
    name, etype, started, recovered, rto, verified, rolled = row
    minutes = (datetime.fromisoformat(recovered) - datetime.fromisoformat(started)).total_seconds() / 60
    within_rto = "n/a" if rto is None else ("YES" if minutes <= rto else "NO")
    print(f"{name:<28} type={etype:<13} minutes={minutes:>5.1f}  within_rto={within_rto}  data_verified={bool(verified)}  rolled_back={bool(rolled)}")
Output
prod-config-clone-eu         type=release_clone minutes= 12.0  within_rto=n/a  data_verified=False  rolled_back=False
rds-failover-drill           type=dr_drill      minutes= 18.0  within_rto=YES  data_verified=True  rolled_back=False
rds-failover-drill-rollback  type=dr_drill      minutes= 45.0  within_rto=NO  data_verified=True  rolled_back=True
Scenario A

In your world

The situation

Your team's quarterly "DR exercise" is a Jenkins job that clones the production release config, deploys it to a secondary account, and marks the ticket "DR tested — passed" once the pipeline shows green. Eight quarters straight, no failures recorded.

What actually breaks

A real regional outage happens. The runbook everyone points to is the DR ticket process—except it has never restored production data, never measured a clock against any RTO, and never run from a failure state. Mid-outage, the team discovers the "DR-tested" secondary account has no current copy of the database and nobody knows the real restore procedure.

What you'd do on Monday

Keep the release-clone process as-is—it's legitimate evidence. Separately, build an actual DR drill that starts from a simulated failure, measures a real recovery clock, restores or promotes real data, and records the gap against the stated RTO. Label each kind of evidence honestly.

PythonSeparate the two evidence types in the ticketing/audit system (needs an internal API, illustrative)
import requests  # stand-in for whatever ticketing/audit API the org actually uses

def record_ops_event(ticket_system_url, name, event_type, **evidence):
    payload = {"name": name, "event_type": event_type, **evidence}
    requests.post(f"{ticket_system_url}/ops-evidence", json=payload, timeout=10)

record_ops_event(
    "https://internal-tickets.example", "q1-release-clone-eu", "release_clone",
    config_cloned=True, deployment_succeeded=True, started_from_failure=False,
)
record_ops_event(
    "https://internal-tickets.example", "q1-dr-drill-rds-failover", "dr_drill",
    started_from_failure=True, recovery_minutes=18, rto_target_min=30, data_verified=True,
)
Scenario B

Imagine you work at Scaler

25 services, annual audit requiring documented DR evidence

The numbers

Illustrative: 25 services, each needing one audit-acceptable DR evidence record a year—zero of the 25 would hold up if an auditor asked for a failure, a clock, and verified data recovery.

What the Scenario-A answer does here

The Scenario A fix works for one service once caught, but retrofitting genuine drills across 25 release-clone-only services means building 25 drill procedures from scratch—far more than anyone budgeted for an audit that assumed the evidence already existed.

What flips at this scale

The fix is prioritized, not uniform: tier the 25 by business impact of an undetected gap, run genuine drills first for the highest tier, and treat the rest as a scheduled backlog rather than a scramble before the audit.

The trade-off you'd defend

I would defend an honestly tiered rollout over a last-minute scramble, or continuing to label clones as DR evidence, trading a harder conversation now for evidence that's actually true when tested.

Illustrative service counts; not Scaler-published figures.

Gotchas

"DR tested" on a status report means a release clone, not a drill

Symptom

A status report claims quarterly DR testing, but no one can produce a recorded recovery time or evidence real data was restored under failure.

Cause

A clone pipeline's "deployment succeeded" status got relabeled "DR tested" somewhere along the way, with nobody separating the two kinds of evidence.

Fix

Audit what evidence actually exists; relabel honestly as release-clone evidence where that's all it is, and schedule real drills to close the gap.

Config parity confirmed, data recovery never checked

Symptom

A cloned environment matches production config exactly, and the team treats that as proof the environment is DR-ready.

Cause

Config parity only proves the infrastructure definition is correct—nothing about whether production data can be restored under time pressure.

Fix

Add a data-recovery verification step as a required part of any exercise claiming DR readiness.

Rollback path only tested on the happy path

Symptom

A drill's rollback step has never been exercised, because every drill so far "succeeded" and nobody triggered it deliberately.

Cause

Rollback procedures only ever written down, never executed, carry the same untested-assumption risk as the forward recovery procedure.

Fix

Deliberately trigger rollback as part of at least some drills, not only when something goes wrong.

Tricky questions

If a cloned release environment's configuration exactly matches production, is that environment DR-ready?
Most people say

Yes, matching configuration is the hard part, and that's done.

Actually

Matching config proves the infrastructure definition is correct. It says nothing about whether real data can actually be restored under a measured clock, starting from simulated failure—the part a clone, by design, never has to prove.

Does a passing release-clone pipeline count as evidence for an audit asking about disaster recovery testing?
Most people say

Yes, the deployment succeeded, which is what the pipeline verifies.

Actually

It's legitimate change-management evidence, not disaster-recovery evidence—it never started from failure, never measured a clock, and typically never restored real data. Presenting it as DR evidence is the gap found during the next real incident.

Scenario question

You're handed a compliance requirement to "demonstrate tested disaster recovery capability" for a service whose only evidence is eight quarters of successful release-clone tickets, two weeks before the audit. What do you actually do?

Model answer

I would not retroactively relabel the clone tickets as DR evidence, since they genuinely aren't. In two weeks, I'd scope the smallest honest drill achievable: pick the single most critical data store, use the backup/restore mechanism from Backup strategy and restore evidence, simulate a failure, measure actual recovery time, and verify the restored data with a checksum. I'd present that one real drill alongside the eight quarters of clone evidence, labeled for what each actually is. The trade-off is admitting a real gap existed, but an auditor finding a mislabeled gap later is far worse than disclosing it now.

Interview corner

CommonWhat's the difference between a release/config clone and a disaster recovery drill?
“I'd say a release clone proves you can deploy a known configuration to a new target while everything is healthy. A DR drill starts from a simulated failure, measures a recovery clock against a stated RTO, and has to prove real data comes back. They often share a pipeline, which is why they get confused on a status report.”
TrickYour team has run the same 'DR exercise' successfully for eight quarters straight. Is that strong evidence of DR readiness?
“I'd want to know what the exercise actually does first. If it's a clone with no simulated failure and no real restoration, eight quarters of green tickets is evidence the pipeline works, not that the system can recover from a disaster.”
Experience checkHow would you present honest DR evidence if some of it is incomplete?
“I'd label each piece of evidence for what it demonstrates—clone success here, a timed drill with verified recovery there—rather than let a report imply uniform coverage. An honest partial picture holds up better than a complete-looking one that doesn't survive a follow-up question.”
DesignHow would you prioritize building real DR drills across many services with only release-clone evidence today?
“By business impact of an undetected gap, not uniformly. I'd drill the highest-impact services for real first and treat the rest as a scheduled backlog, rather than scrambling to produce the same depth everywhere before a deadline.”

Memory hook

A release clone proves you can deploy the same thing twice. A DR drill proves you can come back from being broken—they share a pipeline, not a meaning.

If you remember only one thing:

If the exercise never started from a failure, never measured a clock, and never restored real data, the honest label is "release clone," not "DR tested."

Self-check

  1. What three things does a genuine DR drill have that a release clone usually doesn't?

    A real or simulated failure as the starting state, a measured recovery clock against a stated RTO, and verified recovery of actual data (not just infrastructure).

  2. Why does config parity between a clone and production matter for a release clone but fail to prove DR readiness?

    It proves the infrastructure definition is correct, which is sufficient for a release clone's purpose. It says nothing about whether real data can be restored or promoted under time pressure, which is what a DR drill specifically has to prove.

  3. Why is it dangerous to label release-clone success as "DR tested" on a compliance report?

    Because it creates evidence of a capability that was never actually exercised—the gap only surfaces during a real incident or audit follow-up question, at a worse time than disclosing it honestly up front.

Prove it

Extend the production example's evidence log with a fourth event that is a release clone mislabeled with event_type="dr_drill" but has started_from_failure=False and data_verified=False. Write a validation function that flags any row where event_type="dr_drill" but the underlying fields don't actually support that classification, and print the flagged row.