Shelf · AWS · Chapter 6

Chapter 6: Event-driven AWS

Event-driven patterns let your ETL jobs react to S3 uploads, decouple services and survive partial failures without tight scheduling. This chapter builds the vocabulary—SQS, SNS, EventBridge, Step Functions—and drills the one cross-cutting rule that every card depends on: a message delivery and a business side effect are not the same transaction.

Difficulty
Priority
aws-06-01Card 1 of 6

SQS workers and DLQ recovery

MediumP3 · Role-dependentTalk about it12 min
sqsqueuesdlqvisibility-timeoutworkers

Why you care

Every Jenkins ETL job you have written runs on a schedule. SQS lets the upstream service trigger the job the moment data arrives rather than one hour later. The catch is that SQS gives you at-least-once delivery, not exactly-once processing: your worker can crash after writing to Postgres but before deleting the message, and SQS will hand that same message to another worker. Until you understand the visibility timeout lifecycle and the dead-letter queue, you will either silently duplicate rows or lose messages with no error. Interviewers ask about SQS not to test AWS trivia but to find out whether you have thought about what "acknowledgement" means.

Explain like I'm 5

Imagine a ticket office where customers leave job requests in a box. You pick one up, and a curtain drops around you for ten minutes—nobody else can see that ticket. If you finish and drop the ticket in the bin, it is gone. If the curtain rises before you finish—because you took too long or crashed—the ticket reappears in the box and the next worker picks it up. After a ticket has been picked up three times without ever reaching the bin, the office moves it to a lost-and-found tray (the DLQ) so an operator can inspect it. Every detail in SQS maps to this physical story.

Core concept

SQS has two queue types. A standard queue delivers at least once, in roughly-FIFO order; duplicates are possible. A FIFO queue delivers exactly once per message group, in strict order, at lower throughput (check the current quota page for transactions-per-second limits).

The visibility timeout is the curtain. When a worker calls ReceiveMessage, the message becomes invisible for the configured duration (default 30 seconds, max 12 hours). The worker must call DeleteMessage before that window closes. If it does not—because it crashed, timed out or raised an unhandled exception—the message reappears and another worker receives it. The receive count increments each time.

A dead-letter queue is a separate standard or FIFO queue attached via a redrive policy. You configure maxReceiveCount (1 to 1000). When a message's receive count exceeds that value, SQS moves it to the DLQ automatically. The DLQ has its own retention period (1 minute to 14 days; check the current quota page for current defaults).

Key rule: DeleteMessage is the acknowledgement. Call it only after your side effect—the DB write, the S3 put, the downstream API call—has fully committed. Never call it speculatively before the work succeeds. If your work is idempotent, a retry from a redelivered message is harmless; if it is not, see Idempotent consumers under failure.

Visibility timeout extension. If your processing takes longer than the configured timeout, call ChangeMessageVisibility with more time before the window closes. Failure to do this is the most common cause of duplicate processing.

DLQ redrive. After you fix the bug that caused failures, use the SQS console or StartMessageMoveTask API to move messages from the DLQ back to the source queue. Test this in staging: a DLQ full of messages is not alarming by itself, but a DLQ that grows every day is.

Use a standard queue when order does not matter and throughput is the priority. Use a FIFO queue when processing order within a group is a correctness requirement, not just a preference.

Minimal example Visibility timeout redelivers; DLQ catches repeat failures

PythonVisibility timeout: message redelivered after timeout window expiresruns
class SQSQueue:
    def __init__(self, vt=10, max_rc=3):
        self._msgs = {}   # id -> [body, rc, invisible_until]
        self._vt = vt
        self._max = max_rc
        self.dlq = []

    def send(self, mid, body):
        self._msgs[mid] = [body, 0, 0]

    def receive(self, now):
        for mid, m in self._msgs.items():
            if m[2] <= now:
                m[1] += 1
                m[2] = now + self._vt
                return mid, m[0], m[1]
        return None

    def delete(self, mid):
        self._msgs.pop(mid, None)

    def redrive(self, now):
        gone = [mid for mid, m in self._msgs.items()
                if m[1] >= self._max and m[2] <= now]
        for mid in gone:
            self.dlq.append(self._msgs.pop(mid)[0])


q = SQSQueue(vt=10, max_rc=3)
q.send("m1", "order:42")
t = 0
for _ in range(3):
    mid, body, rc = q.receive(t)
    print(f"t={t:2d} received {mid} rc={rc}")
    t += 11   # visibility expires at t+10, so next receive at t+11
q.redrive(t)
print(f"queue={len(q._msgs)}  dlq={len(q.dlq)}  dlq_body={q.dlq[0]}")
Output
t= 0 received m1 rc=1
t=11 received m1 rc=2
t=22 received m1 rc=3
queue=0  dlq=1  dlq_body=order:42

In production S3-event worker: commit first, then delete

PythonS3-event worker: commit record, then delete; failures go to DLQruns
import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE etl_log (s3_key TEXT PRIMARY KEY, rows INT, status TEXT)")
db.execute("CREATE TABLE dlq (msg_id TEXT, s3_key TEXT, error TEXT)")
db.commit()

QUEUE = [
    {"id": "m1", "s3_key": "raw/orders/2024-01-15/part-001.csv", "rows": 850},
    {"id": "m2", "s3_key": "raw/orders/2024-01-15/part-002.csv", "rows": 0},
    {"id": "m3", "s3_key": "raw/orders/2024-01-15/part-003.csv", "rows": 720},
]

deleted, dlq_count = [], 0
for msg in QUEUE:
    try:
        if msg["rows"] == 0:
            raise ValueError(f"empty file: {msg['s3_key']}")
        db.execute("INSERT OR REPLACE INTO etl_log VALUES (?,?,?)",
                   (msg["s3_key"], msg["rows"], "ok"))
        db.commit()
        deleted.append(msg["id"])
        print(f"ok   {msg['id']}  rows={msg['rows']}")
    except ValueError as e:
        db.execute("INSERT INTO dlq VALUES (?,?,?)",
                   (msg["id"], msg["s3_key"], str(e)))
        db.commit()
        dlq_count += 1
        print(f"dlq  {msg['id']}  {e}")

committed = db.execute("SELECT COUNT(*) FROM etl_log").fetchone()[0]
print(f"committed={committed}  deleted={len(deleted)}  dlq={dlq_count}")
Output
ok   m1  rows=850
dlq  m2  empty file: raw/orders/2024-01-15/part-002.csv
ok   m3  rows=720
committed=2  deleted=2  dlq=1

See it happen SQS message lifecycle

producerSQS queueworkerPostgresDLQSendMessageReceiveMessagemessage + receiptHandleINSERT rowsDeleteMessage(receiptHandle)timeout expires → redeliveredredrive when receiveCount > maxReceiveCount
  1. Producer sends the S3-event notification. Message is visible immediately.
  2. Worker polls and receives the message. Visibility timeout clock starts (default 30 s).
  3. Worker holds the receiptHandle. Message is invisible to all other pollers.
  4. Worker writes rows to Postgres and commits the transaction.
  5. Only after the commit succeeds does the worker acknowledge. Message is gone.
  6. If DeleteMessage is never called (worker crash), the message reappears. receiveCount increments.
  7. After maxReceiveCount deliveries with no delete, SQS moves the message to the DLQ for inspection.
Notice: DeleteMessage is the only acknowledgement SQS accepts. A commit to your database and a delete from the queue are two separate operations; the window between them is where duplicates are born.
Scenario A

In your world

The situation

Your Jenkins job processes order-events SQS messages that reference S3 keys. The Lambda-equivalent worker uses boto3 to receive messages, reads the S3 object, counts rows and writes to Postgres, then calls delete_message. The visibility timeout is the SQS default of 30 seconds.

What actually breaks

A file with 480,000 rows takes 45 seconds to parse and load. The visibility timeout expires during the load. A second worker picks up the same message, parses the same file, and inserts the same rows. Both workers eventually delete the message. The orders table now has duplicate rows—primary key violations if you have a natural key, silent duplicates if you do not. Jenkins shows green for both runs.

What you'd do on Monday

Before or during processing, extend the visibility timeout. Call ChangeMessageVisibility with a comfortable budget (say, 5 minutes) once you have confirmed the file size. Better still, make the upsert idempotent (ON CONFLICT DO NOTHING with a natural key) so that a duplicate delivery has no effect even if the extension fails. The visibility extension needs the receipt handle from ReceiveMessage.

PythonExtend visibility before the load starts (needs boto3 and SQS queue URL)
import boto3

sqs = boto3.client("sqs", region_name="ap-south-1")

msg = sqs.receive_message(
    QueueUrl=QUEUE_URL,
    MaxNumberOfMessages=1,
    WaitTimeSeconds=20,         # long polling
)["Messages"][0]

# Extend before touching the DB
sqs.change_message_visibility(
    QueueUrl=QUEUE_URL,
    ReceiptHandle=msg["ReceiptHandle"],
    VisibilityTimeout=300,      # 5 minutes
)

try:
    load_to_postgres(msg["Body"])
    sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=msg["ReceiptHandle"])
except Exception:
    pass  # message reappears after 5 minutes; DLQ catches persistent failures
Scenario B

Imagine you work at Swiggy

12,000 order events per minute

The numbers

Illustrative: 12,000 events/min = 200/s. A standard queue handles this easily (check current throughput quota). Each message represents one S3 object with ~1,000 rows. At 200 objects/s × 1,000 rows = 200,000 rows/s to Postgres. A 30-second visibility timeout means a worker has 30 seconds to process one file. At 200 ms/file (small CSV) that is fine; at 6 s/file (slow Postgres) a queue of 30 files backlog will all expire simultaneously, causing a thundering re-delivery.

What the Scenario-A answer does here

The Scenario A fix (extend visibility) still works per message, but with 200 concurrent workers each calling ChangeMessageVisibility every 60 seconds, that is 200 API calls/minute just for heartbeats.

What flips at this scale

At this scale the flip is to match the visibility timeout to the 99th-percentile processing time (measured, not guessed), so extensions are rarely needed. You also instrument ApproximateAgeOfOldestMessage as a lag alarm: if it climbs above two visibility-timeout windows, workers are falling behind and you scale out.

The trade-off you'd defend

I would defend a generous visibility timeout matched to real processing time, plus idempotent upserts, over a short timeout with aggressive heartbeats. The short-timeout + heartbeat approach wastes API calls and introduces a new failure mode (heartbeat thread crashes, message redelivers mid-write). The cost is that dead workers hold messages invisibly for longer before they reappear.

Illustrative arithmetic; not Swiggy-published figures.

Gotchas

Worker processes the message but never deletes it

Symptom

Row counts in Postgres grow after every Jenkins run; your audit table shows repeated S3 keys being processed.

Cause

An exception after the DB commit but before DeleteMessage lets the message reappear after the visibility timeout expires.

Fix

Use a try/finally or context manager that calls DeleteMessage only on success, and ensure your upsert is idempotent so redeliveries are harmless.

Visibility timeout expires mid-load

Symptom

Two workers finish successfully; the DB shows duplicate or conflicting rows; both report success to Jenkins.

Cause

Processing took longer than the visibility timeout; the message became visible and a second worker picked it up.

Fix

Extend the timeout before processing starts, or match the timeout to measured 99th-percentile load time.

DLQ messages are not alarmed

Symptom

Bugs silently drop messages into the DLQ for days; nobody notices until downstream data is stale.

Cause

SQS does not alert by default when the DLQ depth climbs.

Fix

Add a CloudWatch alarm on ApproximateNumberOfMessagesVisible for the DLQ with a threshold of 1 or more, and page the on-call.

Tricky questions

Does DeleteMessage guarantee the message is gone from the queue?
Most people say

Yes; once you call it, the message is deleted.

Actually

DeleteMessage is idempotent but not atomic across replicas. In rare cases on a standard queue, a copy of the message that was already in flight to another poller before your delete can still be delivered once. FIFO queues have stronger guarantees. Always write idempotent consumers.

Does a short visibility timeout make my system more responsive to failures?
Most people say

Yes; the shorter the timeout, the faster a failed message reappears.

Actually

A short timeout also causes spurious redeliveries for slow-but-correct workers. Use a timeout matched to the actual processing budget, not the shortest possible value.

Scenario question

You're handed an SQS-driven ETL job that loads S3 files into Postgres. The DLQ depth is 3,000 after a weekend. The source files are fine. Your predecessor set maxReceiveCount to 2 and visibility timeout to 10 seconds. The average file load takes 12 seconds. What happened and how do you fix it?

Model answer

The visibility timeout of 10 seconds is shorter than the 12-second average load time. Almost every message expired before the worker could delete it, incremented receive count on the second delivery, and moved to the DLQ when it hit maxReceiveCount of 2. No data loss—the files are fine—but no data was loaded either. I would: first, increase visibility timeout to at least 60 seconds (five times the average, to cover the tail); second, increase maxReceiveCount to at least 5 to tolerate genuine transient failures; third, redrive the DLQ back to the source queue and monitor the DLQ alarm. The trade-off is that a genuinely failing message now takes longer to reach the DLQ, but that is acceptable compared to discarding all messages.

Interview corner

CommonWhat is the SQS visibility timeout and why does it matter?
“I'd describe it as the window during which a received message is hidden from other consumers. The worker must delete the message inside that window. If it crashes or takes too long, the message reappears and gets redelivered. It matters because that redelivery is what makes consumers need to be idempotent.”
TrickYou call DeleteMessage immediately after receiving, before the load. What breaks?
“I'd say you've acknowledged a message you haven't processed yet. If the load fails, the message is gone and you've silently dropped data. DeleteMessage has to come after the side effect commits, not before.”
Experience checkWhen would you use a FIFO queue over a standard queue?
“I'd use FIFO when the order of processing within a group is a correctness requirement—for example, updating account balance events that must apply in the order they were generated. The cost is lower throughput, so I'd only pay that cost when ordering is genuinely required, not just preferred.”
DesignHow do you detect that your SQS workers are falling behind?
“I'd alarm on ApproximateAgeOfOldestMessage. If that climbs above one or two visibility timeout windows, workers are processing slower than messages arrive and I need to scale out. I'd also alarm on DLQ depth to catch persistent failures separately.”

Memory hook

The visibility timeout is the curtain. If the worker drops the ticket before the curtain falls, it is gone. If not, the ticket reappears.

If you remember only one thing:

DeleteMessage is the only acknowledgement. Call it after the side effect commits, never before.

Self-check

  1. What happens to a message if the worker calls DeleteMessage twice with the same receipt handle?

    The second call succeeds silently; DeleteMessage is idempotent. No error is raised.

  2. What is the maximum visibility timeout you can set on an SQS message?

    12 hours (check the current SQS documentation for the exact value; this is the documented maximum as of mid-2025).

  3. Why should DLQ retention be set longer than source queue retention?

    Messages in the DLQ need time for an operator to inspect and redrive them. If the DLQ retention is shorter than the time it takes to notice and respond, messages expire before they can be fixed.

Prove it

Extend the minimal example to show the full redrive workflow: after messages land in the DLQ, "fix the bug" (set a flag), move them back to the source queue, and process them successfully. Print the state of the source queue and DLQ after each phase.

aws-06-02Card 2 of 6

SNS fan-out and subscriptions

MediumP3 · Role-dependentTalk about it11 min
snsfan-outsubscriptionsfilter-policypub-sub

Why you care

You have one S3-event SQS consumer today. Tomorrow the payments team wants the same events, and so does the audit team. Copy-pasting the SQS queue and teaching the producer about every consumer does not scale, and it couples every new consumer to a producer code change. SNS is the fix: one publish, many deliveries. What trips engineers up is the subscription filter policy—it looks like a minor config detail but it determines whether a consumer gets ten events a day or ten million—and the fact that SNS delivery to an SQS subscriber is still at-least-once, not exactly-once. Interviewers ask about SNS to check whether you understand isolation: one subscriber failing should not block delivery to another.

Explain like I'm 5

SNS is a public address system in a large office. When someone makes an announcement, everyone with a matching preference hears it. You can tell the system "only send me announcements tagged URGENT", and everyone else keeps receiving all announcements. If the fire-alarm speaker breaks, the desk phone still rings. The speakers are isolated—one broken speaker does not stop the others.

Core concept

An SNS topic is the address. A subscription is a registered endpoint—SQS queue, Lambda function, HTTP/S endpoint, email, SMS or mobile push. When you publish a message to the topic, SNS attempts delivery to every active subscription.

Subscription filter policies let each subscription declare which messages it wants. You add a JSON policy to the subscription that lists allowed values for message attributes. SNS evaluates the policy before delivery; messages that do not match are simply dropped for that subscriber. A subscription with no filter policy receives every message. (As of 2023, SNS also supports filtering on the message body using the FilterPolicyScope setting; check the current documentation.)

Isolation. Delivery to each subscription is independent. A subscriber that is unavailable or returning errors does not prevent delivery to other subscribers. For AWS-managed endpoints (SQS, Lambda), SNS's own delivery policy retries up to 100,015 times over about 23 days. HTTP/S endpoints get a shorter, customer-configurable policy with a hard cap of 3,600 seconds (1 hour) total retry time — AWS's default HTTP/S policy makes 50 attempts inside that hour (check the current documentation for the exact schedule).

Message size. SNS messages are limited to 256 KB. For large payloads, publish a reference (S3 key) and let subscribers fetch the content.

SNS-to-SQS fan-out is the most reliable pattern. The SQS queue provides a durable buffer so the subscriber can process at its own rate. Lambda subscribers are invoked synchronously; if Lambda throttles, SNS retries. Combining SNS with SQS queues per subscriber gives each team an independent buffer, backpressure and DLQ.

When not to use SNS: if you need ordered delivery or exactly-once semantics, use SQS FIFO directly or route through EventBridge. SNS standard topics have no ordering guarantees and may deliver duplicates.

Minimal example Subscription filter routes each event to matching subscribers

PythonSNS fan-out: filter policy routes events to matching subscribers onlyruns
def match_filter(policy, attrs):
    return all(attrs.get(k) in v for k, v in policy.items())

SUBSCRIBERS = [
    {"name": "payments-sqs",  "filter": {"event_type": ["ORDER_PAID"]}},
    {"name": "logistics-sqs", "filter": {"event_type": ["ORDER_SHIPPED", "ORDER_DELIVERED"]}},
    {"name": "audit-lambda",  "filter": {}},
]

MESSAGES = [
    {"attrs": {"event_type": "ORDER_PAID"},      "body": "order-101"},
    {"attrs": {"event_type": "ORDER_SHIPPED"},   "body": "order-101"},
    {"attrs": {"event_type": "ORDER_CANCELLED"}, "body": "order-102"},
]

for msg in MESSAGES:
    delivered = [s["name"] for s in SUBSCRIBERS
                 if match_filter(s["filter"], msg["attrs"])]
    print(f"{msg['attrs']['event_type']:<20} -> {delivered}")
Output
ORDER_PAID           -> ['payments-sqs', 'audit-lambda']
ORDER_SHIPPED        -> ['logistics-sqs', 'audit-lambda']
ORDER_CANCELLED      -> ['audit-lambda']

In production Shipment fan-out: one subscriber failure does not block others

PythonSubscriber failure is isolated; other subscribers still receiveruns
def fan_out(subscribers, message, attrs):
    for sub in subscribers:
        if not all(attrs.get(k) in v for k, v in sub.get("filter", {}).items()):
            continue
        try:
            sub["fn"](message)
            print(f"  [ok]  {sub['name']}")
        except Exception as e:
            print(f"  [err] {sub['name']}: {e}")

def notify_sms(msg):
    if msg["carrier"] == "unknown":
        raise RuntimeError("carrier lookup failed")
    print(f"       sms sent for {msg['id']}")

def update_wh(msg):
    print(f"       warehouse updated for {msg['id']}")

def charge_cod(msg):
    print(f"       COD Rs {msg['cod']} charged for {msg['id']}")

SUBS = [
    {"name": "sms",       "filter": {"status": ["OUT_FOR_DELIVERY"]}, "fn": notify_sms},
    {"name": "warehouse", "filter": {},                                "fn": update_wh},
    {"name": "billing",   "filter": {"mode": ["COD"]},                "fn": charge_cod},
]

print("--- DL-1001 (prepaid, known carrier) ---")
fan_out(SUBS,
        {"id": "DL-1001", "carrier": "delhivery", "cod": 0},
        {"status": "OUT_FOR_DELIVERY", "mode": "PREPAID"})

print("--- DL-1002 (COD, unknown carrier) ---")
fan_out(SUBS,
        {"id": "DL-1002", "carrier": "unknown", "cod": 250},
        {"status": "OUT_FOR_DELIVERY", "mode": "COD"})
Output
--- DL-1001 (prepaid, known carrier) ---
       sms sent for DL-1001
  [ok]  sms
       warehouse updated for DL-1001
  [ok]  warehouse
--- DL-1002 (COD, unknown carrier) ---
  [err] sms: carrier lookup failed
       warehouse updated for DL-1002
  [ok]  warehouse
       COD Rs 250 charged for DL-1002
  [ok]  billing
Scenario A

In your world

The situation

Your ETL team publishes order-completed events to an SNS topic. Three subscribers: your SQS queue for DB loading, the payments team's Lambda for reconciliation, and the audit team's HTTP endpoint. All three use the same topic with no filter policies.

What actually breaks

The audit team's HTTP endpoint goes down for two hours. SNS's default HTTP/S retry policy caps total retry time at one hour, so every event published during the outage exhausts its retries and is discarded — silently, since the audit subscription has no DLQ. Your SQS queue and the payments Lambda keep receiving events without interruption, because those AWS-managed endpoints get SNS's much longer (about 23-day) retry window instead. The audit team ends up missing roughly two hours of events with no error on your side and no record of the gap on theirs.

What you'd do on Monday

Give the audit team an SQS queue as their subscriber instead of the raw HTTP endpoint. Their service polls the queue at their own rate, and the queue buffers during the outage without affecting your team. Add a DLQ to their queue so it can never silently lose events again. Switch the endpoint subscriber to the SQS-to-HTTP pattern and remove the raw HTTP subscription.

PythonPublish to SNS topic with message attributes for filter routing (needs boto3)
import boto3, json

sns = boto3.client("sns", region_name="ap-south-1")
sns.publish(
    TopicArn="arn:aws:sns:ap-south-1:123456789012:order-events",
    Message=json.dumps({"order_id": "ORD-42", "amount": 499}),
    MessageAttributes={
        "event_type": {"DataType": "String", "StringValue": "ORDER_COMPLETED"},
        "region":     {"DataType": "String", "StringValue": "south"},
    },
)
Scenario B

Imagine you work at Delhivery

2 million shipment events per day

The numbers

Illustrative: 2M events/day ≈ 23 events/s average, with peaks around 10× at COD delivery windows. SNS standard topics handle this throughput easily (check the current publish TPS quota for your region). Each event fans out to four subscribers (SMS, warehouse, billing, audit). Total deliveries: 8M/day.

What the Scenario-A answer does here

The fan-out itself scales fine at SNS. The problem is the billing subscriber: like every subscriber, it receives one delivery per event (2M/day), then filters in code and discards 60% (PREPAID orders). That is 1.2M wasted deliveries/day to a Lambda that immediately exits, which runs up Lambda invocation costs and adds latency for the billing team.

What flips at this scale

Add a filter policy {"mode": ["COD"]} to the billing subscription. SNS evaluates the filter before invoking Lambda, dropping 1.2M daily invocations at no cost. The filter lives on the subscription, not in the Lambda code, so the Lambda simplifies and the billing team no longer needs to handle irrelevant event types.

The trade-off you'd defend

I would defend filter policies on every subscription except the audit subscription, accepting slightly more SNS config to maintain over time in exchange for lower downstream costs and simpler subscriber code. Audit needs all events by definition.

Illustrative event volume; not Delhivery-published figures.

Gotchas

No subscription filter means all messages

Symptom

A new subscriber's Lambda is invoked for every event on the topic, including events it does not care about, and costs accumulate.

Cause

A subscription with no filter policy receives everything.

Fix

Add a filter policy to any subscription that should receive a subset of messages.

SNS to HTTP/S: retries expire within an hour, silently

Symptom

A subscriber's HTTP endpoint is down for longer than about an hour. No error surfaces on the publishing side, but events published during the outage never arrive.

Cause

SNS's default HTTP/S retry policy caps total retry time at 3,600 seconds (1 hour) — a hard AWS limit that cannot be raised. Once that window is exhausted, SNS discards the message unless a DLQ is attached to the subscription. (SQS and Lambda subscribers get a much longer, ~23-day SNS-managed retry window instead, so this specific failure is HTTP/S-only.)

Fix

Use an SQS queue as the subscriber instead of a raw HTTP endpoint, so the queue — not a 1-hour retry clock — buffers during the outage. If you must keep a raw HTTP/S subscriber, attach a DLQ so exhausted retries are captured instead of dropped.

SNS message size limit

Symptom

InvalidParameter error when publishing a large payload.

Cause

SNS messages are limited to 256 KB.

Fix

Publish a reference (S3 key or DynamoDB key) in the SNS message and let subscribers fetch the full payload independently.

Tricky questions

Does publishing to SNS guarantee delivery to all subscribers?
Most people say

Yes; SNS is a fully managed service, so it will eventually deliver to all subscribers.

Actually

SNS guarantees at-least-once delivery to active, reachable subscribers. A subscriber that is misconfigured, throttled or persistently unavailable may miss messages. HTTP/S endpoints that fail after all retries are exhausted permanently lose those messages unless you have a DLQ on an SNS subscription (supported for HTTP/S, SQS and Lambda subscribers as of 2023; check current documentation).

Is an SNS topic with one subscriber the same as using SQS directly?
Most people say

Yes; the result is identical.

Actually

SNS adds a fan-out capability you cannot get from SQS alone. The overhead is minimal, but if you genuinely have only one consumer and never will, SQS directly is simpler. The common mistake is adding SNS "in case we need fan-out later" before the second consumer actually exists, which adds operational complexity for no current benefit.

Scenario question

You're handed a system where an EC2-based order service publishes directly to three separate SQS queues—one per downstream team. Adding a fourth consumer requires a code change and redeploy of the order service. How do you redesign without changing the downstream consumers?

Model answer

I would introduce an SNS topic between the order service and the queues. The order service publishes once to the topic. Each team's SQS queue becomes an SNS subscription. Adding a fourth consumer is now a subscription configuration change with zero impact on the order service. The trade-off is that the order service loses direct visibility into per-consumer queue depth; I would add CloudWatch alarms on each queue's ApproximateAgeOfOldestMessage to each team's runbook so they own their own lag monitoring. I would also add filter policies per subscription so that each team's queue only receives the event types it actually needs.

Interview corner

CommonWhat is SNS fan-out and when would you use it?
“I'd describe it as publishing once and delivering to many subscribers simultaneously. I use it when multiple independent services need the same event—order completed, shipment status change—and I don't want the producer to know about every consumer. Each subscriber gets its own copy.”
TrickA subscriber is slow. Does that slow down other subscribers?
“I'd say no. SNS delivers to each subscription independently. A slow or failing SQS subscriber doesn't block delivery to a Lambda or HTTP endpoint on the same topic.”
Experience checkWhen would you put an SQS queue behind an SNS subscription?
“Whenever the subscriber needs to process at its own rate or survive downtime. The queue buffers messages during outages and lets me tune consumer concurrency independently. I almost always prefer SQS as the SNS subscriber over a raw HTTP endpoint.”

Memory hook

SNS is the office PA system: one announcement, every speaker at once. One broken speaker does not silence the others.

If you remember only one thing:

Subscriber failure is isolated. Add a filter policy to every subscription that does not need all messages.

Self-check

  1. What is the SNS message size limit?

    256 KB. For larger payloads, publish a reference to S3 and let subscribers fetch the content.

  2. If an SNS subscription has no filter policy, which messages does it receive?

    Every message published to the topic.

  3. Why prefer an SQS queue over a raw HTTP endpoint as an SNS subscriber?

    The queue buffers during downtime, lets the consumer process at its own rate, and provides a DLQ. A raw HTTP subscriber can permanently lose messages if retries are exhausted.

Prove it

Extend the production example to add a fourth subscriber that raises a random error 50% of the time (use random.seed(42) for reproducibility). Run 10 messages through the fan-out and show that the other three subscribers receive all 10 messages regardless of how many times the fourth fails.

aws-06-03Card 3 of 6

EventBridge routing and scheduling

MediumP3 · Role-dependentTalk about it11 min
eventbridgerulestargetsschedulerevent-patterns

Why you care

SQS needs a producer to explicitly send a message. SNS needs a producer to explicitly publish. EventBridge sits between your services and reacts to events they already emit—EC2 state changes, S3 PutObject notifications, CodePipeline state transitions—without those services knowing about your consumers. It also replaces Jenkins cron: EventBridge Scheduler triggers Lambda or Step Functions on a rate or cron expression without you managing a scheduler host. For ETL work, this means your nightly Glue job or Lambda-triggered S3 load can be an EventBridge rule, not a Jenkins pipeline running sleep. Interviewers use EventBridge questions to check whether you understand event contracts: the shape of an event is a public API that breaks consumers when it changes.

Explain like I'm 5

EventBridge is an air traffic controller. Planes (events) land from many airlines (AWS services, your apps, SaaS partners). The controller matches each plane against a set of rules: "all planes from DoorDash, terminal 3" or "all emergency landings, gate 1 immediately". The rules live in the controller, not in the planes or the terminals. A new terminal can start receiving planes by adding a rule, without any change to the airline.

Core concept

The three objects you configure:

Event bus. The channel events flow through. The default bus carries AWS service events and your custom events. A custom bus isolates domains (e.g., orders-bus, payments-bus). A partner bus receives events from SaaS providers (Salesforce, Zendesk, etc.).

Rule. A rule lives on a bus and has either an event pattern (JSON-based filter) or a schedule (rate or cron expression). An event pattern matches against the event's top-level fields (source, detail-type) and the detail object. EventBridge supports exact match, prefix match, numeric ranges, anything-but, exists and $or operators.

Target. Each rule can have up to five targets: Lambda, SQS, SNS, Step Functions, Kinesis, API Gateway, another EventBridge bus, etc. EventBridge retries failed target invocations with exponential backoff for up to 24 hours (185 retries; check the current documentation for the exact schedule). You can attach a dead-letter SQS queue to a rule target to capture events that exhausted all retries.

EventBridge Scheduler is a separate but related service that creates one-time or recurring schedules. It supports rate expressions (rate(5 minutes)), cron expressions (cron(0 9 * * ? *)) and one-time ISO 8601 timestamps. It targets the same set of services. The key advantage over a rule-based schedule on the default bus is that Scheduler handles missed invocations during a maintenance window and delivers them once the target is available.

Event contracts. The detail field schema is your service's public API to all EventBridge consumers. Adding fields is safe (additive). Removing or renaming fields breaks every consumer silently—their pattern stops matching and they receive no error. Version your events ("version": "1" in the detail) and treat schema changes with the same care as REST API changes.

Replay. You can archive events from a bus and replay them at a later time, filtered by event pattern and time range. This is invaluable for recovering from a bug in a consumer that silently dropped events.

Use EventBridge when the producer already emits events (S3, EC2, CodePipeline) and you want to route without changing the producer. Use SQS directly when the producer is your own code and you need durable, buffered delivery to one consumer.

Minimal example Rate-schedule to next invocation times

PythonEventBridge Scheduler: rate expression to next invocationsruns
from datetime import datetime, timedelta

UNITS = {
    "minute": timedelta(minutes=1),
    "hour": timedelta(hours=1),
    "day": timedelta(days=1),
}

def next_runs(expr, start, n):
    value, unit = expr.lstrip("rate(").rstrip(")").split()
    delta = UNITS[unit.rstrip("s")] * int(value)
    t = start
    for _ in range(n):
        t += delta
        yield t.strftime("%Y-%m-%d %H:%M")

START = datetime(2024, 1, 15, 0, 0)
for name, expr in [
    ("etl-hourly",   "rate(1 hour)"),
    ("reconcile",    "rate(6 hours)"),
    ("daily-report", "rate(1 day)"),
]:
    runs = list(next_runs(expr, START, 3))
    print(f"{name:<16} {expr:<20} -> {runs[0]}, {runs[1]}, ...")
Output
etl-hourly       rate(1 hour)         -> 2024-01-15 01:00, 2024-01-15 02:00, ...
reconcile        rate(6 hours)        -> 2024-01-15 06:00, 2024-01-15 12:00, ...
daily-report     rate(1 day)          -> 2024-01-16 00:00, 2024-01-17 00:00, ...

In production EventBridge event pattern matcher with retry simulation

PythonEvent pattern matching with exact, prefix and anything-but operatorsruns
def match(pattern, event):
    """Simulate EventBridge event pattern evaluation."""
    for key, condition in pattern.items():
        val = event.get(key)
        if isinstance(condition, dict):
            if not isinstance(val, dict) or not match(condition, val):
                return False
        elif isinstance(condition, list):
            ok = False
            for c in condition:
                if isinstance(c, dict):
                    if "prefix" in c and isinstance(val, str) and val.startswith(c["prefix"]):
                        ok = True; break
                    if "anything-but" in c and val != c["anything-but"]:
                        ok = True; break
                elif val == c:
                    ok = True; break
            if not ok:
                return False
    return True

RULES = [
    {"name": "etl-trigger",
     "pattern": {"source": ["orders.s3"],
                 "detail": {"status": ["OBJECT_CREATED"],
                             "key":    [{"prefix": "raw/orders/"}]}}},
    {"name": "failure-alert",
     "pattern": {"source": ["orders.s3"],
                 "detail": {"status": [{"anything-but": "OBJECT_CREATED"}]}}},
    {"name": "archive-skip",
     "pattern": {"detail": {"key": [{"prefix": "archive/"}]}}},
]

EVENTS = [
    {"source": "orders.s3", "detail": {"status": "OBJECT_CREATED",
                                        "key": "raw/orders/2024-01-15/001.csv"}},
    {"source": "orders.s3", "detail": {"status": "OBJECT_REMOVED",
                                        "key": "raw/orders/2024-01-15/001.csv"}},
    {"source": "orders.s3", "detail": {"status": "OBJECT_CREATED",
                                        "key": "archive/2024/backup.zip"}},
]

for ev in EVENTS:
    hits = [r["name"] for r in RULES if match(r["pattern"], ev)]
    print(f"{ev['detail']['status']:<20} {ev['detail']['key']:<36} -> {hits}")
Output
OBJECT_CREATED       raw/orders/2024-01-15/001.csv        -> ['etl-trigger']
OBJECT_REMOVED       raw/orders/2024-01-15/001.csv        -> ['failure-alert']
OBJECT_CREATED       archive/2024/backup.zip              -> ['archive-skip']
Scenario A

In your world

The situation

Your nightly Glue ETL job runs via a Jenkins cron at 02:00 IST. The job reads from S3 and writes to Redshift. Jenkins sometimes misses the 02:00 window when the EC2 host reboots for a patch, and nobody notices until 09:00 when the dashboard is stale.

What actually breaks

The Jenkins cron fires a job only if the host is up at exactly 02:00. A reboot at 01:55 means the job never runs. There is no retry, no alarm and no missed-invocation record.

What you'd do on Monday

Replace the Jenkins cron trigger with an EventBridge Scheduler rule targeting the Glue job or a Lambda that starts it. EventBridge Scheduler is serverless—no host to reboot—and supports a flexible window that will invoke within a window if the exact time is missed. Set a CloudWatch alarm on FailedInvocations for the rule.

PythonEventBridge Scheduler rule targeting a Glue job (needs boto3 and AWS)
import boto3, json

scheduler = boto3.client("scheduler", region_name="ap-south-1")
scheduler.create_schedule(
    Name="nightly-orders-etl",
    ScheduleExpression="cron(0 20 * * ? *)",   # 20:30 UTC = 02:00 IST
    FlexibleTimeWindow={"Mode": "FLEXIBLE", "MaximumWindowInMinutes": 15},
    Target={
        "Arn": "arn:aws:glue:ap-south-1:123456789012:job/orders-etl",
        "RoleArn": "arn:aws:iam::123456789012:role/scheduler-glue-role",
        "Input": json.dumps({"env": "prod", "date": "$(aws:ScheduledTime)"}),
    },
)
Scenario B

Imagine you work at DoorDash

300,000 delivery zone routing events per hour

The numbers

Illustrative: 300,000 events/hour ≈ 83/s. Each event hits a custom EventBridge bus with 12 rules, each rule with up to 5 targets = up to 60 target invocations per event in the worst case. Check the current EventBridge invocations-per-second quota for your region before relying on this pattern at high throughput.

What the Scenario-A answer does here

The Scenario A fix (replace Jenkins cron with EventBridge Scheduler) is correct, but at 83 events/s the default EventBridge quota for PutEvents may be a ceiling (check the current quota page). More critically: 12 rules with overlapping patterns all evaluating every event means the pattern matching cost multiplies. A rule with a broad source match fires for every event on the bus even if it has no targets for most of them.

What flips at this scale

At this scale the flip is to use specific source and detail-type patterns so that rules are as narrow as possible, reducing unnecessary evaluation. You also consider splitting into multiple custom buses by domain (delivery-events bus, restaurant-events bus) so each bus carries only the events its rules care about.

The trade-off you'd defend

I would defend narrow event patterns and domain-scoped buses over a single wide default bus, accepting more buses and rules to manage in exchange for predictable quota headroom and lower per-event evaluation cost. Single-bus simplicity is fine up to a few hundred events/s; beyond that, quota pages become required reading before designing.

Illustrative arithmetic; not DoorDash-published figures.

Gotchas

Event contract breaks silently

Symptom

A consumer's Lambda stops processing events after a producer renames a field in the event detail. No error is raised; the rule's pattern simply stops matching.

Cause

EventBridge event patterns do not validate against a schema by default; a non-matching event is silently dropped.

Fix

Enable EventBridge Schema Registry for your custom bus. Treat any detail field removal or rename as a breaking change. Version your events ("version" field) and maintain backward-compatible schemas.

Schedule fires at UTC, not IST

Symptom

The ETL job runs at 08:00 IST instead of 02:00 IST.

Cause

EventBridge Scheduler cron expressions are evaluated in UTC by default (or a time zone you specify; check current Scheduler documentation for time zone support).

Fix

Convert your local time to UTC before writing the cron expression, or specify the time zone explicitly in the Scheduler API.

Rule target retries exhaust silently

Symptom

Events are lost during a downstream outage; after recovery, the consumer never receives them.

Cause

EventBridge retries a failed target for up to 24 hours. After that, the event is dropped unless you configured a DLQ on the target.

Fix

Add a dead-letter SQS queue to every critical rule target. Enable EventBridge archive on the bus so you can replay if needed.

Tricky questions

Is EventBridge Scheduler the same as a CloudWatch Events schedule?
Most people say

Yes; they are the same service.

Actually

CloudWatch Events and EventBridge are the same underlying service (EventBridge is the newer name), but EventBridge Scheduler is a newer, separate service with its own API, richer time-zone support and flexible-window delivery. Prefer Scheduler for new scheduled invocations; CloudWatch Events-style schedules on rules still work but are the older pattern.

Does an EventBridge rule fire if no targets are configured?
Most people say

Yes; the rule still evaluates and records a match.

Actually

A rule with no targets evaluates patterns but does nothing; no invocation occurs and no cost is incurred for target invocation. The rule itself has a small cost per million events evaluated.

Scenario question

You're handed a microservices system where Service A calls Service B's REST API synchronously to trigger downstream processing. When B is slow, A times out and returns 500. How would you redesign using EventBridge?

Model answer

I would decouple the trigger from the response. Service A publishes an event to a custom EventBridge bus when its work is done, rather than calling B directly. An EventBridge rule routes that event to B's SQS queue. B processes asynchronously at its own rate. A gets an immediate 200 after publishing; B's queue absorbs backpressure. The trade-off is that A can no longer get B's response synchronously. If A needs a result from B, I would have B publish a completion event that A subscribes to, or use a correlation ID to poll a status table. This pattern also makes adding a third consumer C trivial: add a rule, no change to A.

Interview corner

CommonWhat is the difference between an EventBridge rule and a schedule?
“I'd say a rule evaluates an event pattern against events on the bus and triggers targets when it matches. A schedule fires on a time expression—rate or cron—with no incoming event needed. Both can target the same set of services.”
TrickAn EventBridge rule stops triggering a Lambda after a schema change. No errors appear. Why?
“I'd suspect the event pattern no longer matches the new event shape. EventBridge silently drops events that don't match any rule. I'd check the Schema Registry, compare the event detail to the rule pattern, and enable a DLQ so I can inspect missed events going forward.”
Experience checkHow would you replace a Jenkins cron trigger with EventBridge?
“I'd create an EventBridge Scheduler rule with a rate or cron expression targeting the Lambda or Glue job. Scheduler is serverless, so there's no host to go down. I'd set a flexible-time window for missed invocations and alarm on FailedInvocations.”

Memory hook

EventBridge is the air traffic controller: events arrive from all directions, and rules route them to the right gate without the planes knowing about the gates.

If you remember only one thing:

Event patterns are silent—they drop non-matching events with no error. Version your event schema.

Self-check

  1. How many targets can a single EventBridge rule have?

    Up to five targets per rule (check the current documentation for the exact limit).

  2. What happens to an event that matches no rule on a bus?

    It is silently discarded. No error is raised to the publisher.

  3. What is the advantage of EventBridge Scheduler over a CloudWatch Events schedule rule?

    Scheduler supports richer time zones, flexible-time windows for missed invocations, and one-time schedules with ISO 8601 timestamps.

Prove it

Using the production pattern-matcher, add a fourth rule that uses a numeric range operator ({"detail": {"size_bytes": [{"numeric": [">", 1048576]}]}}) and verify it matches events with size_bytes > 1_048_576 and does not match smaller ones. Show the output for three events with sizes 512000, 1048577 and 2097152.

aws-06-04Card 4 of 6

Step Functions orchestration

HardP3 · Role-dependentTalk about it14 min
step-functionsstate-machinesorchestrationretrycatchworkflows

Why you care

Lambda functions are stateless. A five-step ETL pipeline—validate, transform, load, verify, notify—needs state shared somewhere between steps. You can pass it through SQS or DynamoDB, but then you own the retry logic, the error branches, the timeout tracking and the audit trail. Step Functions gives you all of that as a managed state machine: each state is a JSON definition, retries and catches are declarative, and the execution history is queryable. For an interviewer, the Step Functions question tests whether you can articulate the difference between a workflow you can inspect and debug versus a chain of Lambda calls stitched together by hand.

Explain like I'm 5

Step Functions is a flowchart you can actually run. Each box in the flowchart is a state. When you draw an arrow to the next box, that is a transition. When you draw a diamond (decision), that is a Choice state. You can draw a circular arrow (retry) on any box. You can draw a detour to an error-handling box (catch). The flowchart runs once per execution, and you can open the AWS console and see exactly which box it is stuck in right now.

Core concept

A state machine is a JSON document describing states and transitions. An execution is one run of that machine with an input JSON payload.

State types:

  • Task: calls an AWS service (Lambda, ECS, DynamoDB, SQS, etc.) or an HTTP endpoint. This is where your business logic lives.
  • Choice: branches based on conditions on the current state input. No retry or catch.
  • Wait: pauses for a fixed duration or until a timestamp in the input.
  • Parallel: runs multiple branches simultaneously; waits for all branches before proceeding.
  • Map: iterates over an array in the input, running a sub-workflow per item.
  • Pass: transforms the input without calling anything. Useful for injecting defaults.
  • Succeed / Fail: terminal states.

Retry and Catch sit on Task and Parallel states. Retry is a list of rules; each rule matches on a list of error names, has MaxAttempts, IntervalSeconds and BackoffRate. Catch is a list of error handlers; when retries are exhausted or the error is not retried, the first matching catch fires and transitions to its Next state. This is where you put compensation (un-reserve stock, refund the charge).

Workflow types:

  • Standard workflow: durable execution history stored for 90 days. Exactly-once Task execution semantics—each Task is started at most once per execution step. Max duration: 1 year. Suitable for long-running ETL, order processing, human-approval workflows.
  • Express workflow: high-volume, short-duration (max 5 minutes). At-least-once execution semantics—a Task may be called more than once during retries. Suitable for streaming ingestion pipelines, IoT processing. Execution history goes to CloudWatch Logs, not the console.

Use Standard for workflows where exactly-once task execution matters (payments, data loads). Use Express for high-volume, idempotent, short-lived processing.

Service integrations. Step Functions can call Lambda (synchronous or fire-and-forget), ECS (run task and wait), DynamoDB, SQS, Glue, Athena and more without writing a Lambda wrapper. This reduces Lambda cold-start overhead and simplifies pipelines.

Under the hood

Step Functions Standard workflows store execution state durably in the service's own managed storage before calling each task. If a Task call is in flight and the Step Functions control plane restarts, the execution resumes from the last durable checkpoint—the state input and the task token are preserved. This is why the maximum execution duration is one year and why Standard workflows are billed per state transition, not by wall-clock time. Express workflows write state to CloudWatch Logs asynchronously and do not guarantee exactly-once Task execution; they are billed by duration and memory, not transitions. That billing model difference is not cosmetic: at 1M short executions per day, Standard workflows at $0.025 per 1,000 state transitions would cost dramatically more than Express at per-second pricing (check the current Step Functions pricing page before choosing).

Minimal example Task retry and catch in a booking flow

PythonStep Functions retry exhausts, then catch fires compensationruns
class PaymentError(Exception): pass


def flaky_validate(data):
    data["_attempts"] = data.get("_attempts", 0) + 1
    if data["_attempts"] < 3:
        raise RuntimeError(f"transient #{data['_attempts']}")
    return {**data, "validated": True}


def compensate(data):
    return {**data, "status": "compensated"}


def run_task(fn, data, max_retries=2, catch=None):
    for attempt in range(1, max_retries + 2):
        try:
            result = fn(data)
            print(f"  attempt {attempt}: ok")
            return result
        except Exception as e:
            if attempt <= max_retries:
                print(f"  attempt {attempt}: {e}")
            elif catch:
                print(f"  attempt {attempt}: {e} -> {catch.__name__}")
                return catch(data)
            else:
                raise


out = run_task(flaky_validate, {"order": 1}, max_retries=2, catch=compensate)
print(out)
Output
  attempt 1: transient #1
  attempt 2: transient #2
  attempt 3: ok
{'order': 1, '_attempts': 3, 'validated': True}

In production Multi-step ETL state machine with Choice and Catch

PythonETL state machine: validate → load → verify, with branching and compensationruns
import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE orders (id INT PRIMARY KEY, amt REAL)")
db.execute("CREATE TABLE audit  (step TEXT, status TEXT, detail TEXT)")
db.commit()

def log(step, status, detail=""):
    db.execute("INSERT INTO audit VALUES (?,?,?)", (step, status, detail))
    db.commit()

def validate_step(data):
    if not data.get("rows"):
        raise ValueError("no rows in payload")
    log("validate", "ok", f"rows={data['rows']}")
    return {**data, "validated": True}

def load_step(data):
    for i in range(data["rows"]):
        db.execute("INSERT OR IGNORE INTO orders VALUES (?,?)", (i + 1, (i + 1) * 10.0))
    db.commit()
    log("load", "ok", f"inserted={data['rows']}")
    return {**data, "loaded": True}

def verify_step(data):
    count = db.execute("SELECT COUNT(*) FROM orders").fetchone()[0]
    if count != data["rows"]:
        raise AssertionError(f"expected {data['rows']} rows, found {count}")
    log("verify", "ok", f"count={count}")
    return {**data, "verified": True}

def compensate_step(data, error):
    db.execute("DELETE FROM orders")
    db.commit()
    log("compensate", "ok", str(error))
    return {**data, "status": "rolled_back"}

STEPS = [("validate", validate_step), ("load", load_step), ("verify", verify_step)]

def run_machine(payload):
    data = dict(payload)
    for name, fn in STEPS:
        try:
            data = fn(data)
            print(f"  {name}: ok")
        except (ValueError, AssertionError) as e:
            print(f"  {name}: FAIL {e}")
            data = compensate_step(data, e)
            print(f"  compensate: {data['status']}")
            return data
    return data

result = run_machine({"rows": 3})
print(f"final: verified={result.get('verified')}  "
      f"audit={db.execute('SELECT step,status FROM audit').fetchall()}")
Output
  validate: ok
  load: ok
  verify: ok
final: verified=True  audit=[('validate', 'ok'), ('load', 'ok'), ('verify', 'ok')]

See it happen Step Functions execution flow

okokokCatchCatchSTARTValidate(Task)Load(Task)Verify(Task)Compensate(Task)FAILSUCCEED
  1. Execution starts with the input payload. Each state receives the output of the previous state.
  2. Validate runs. If it fails after all retries, Catch fires immediately and skips Load.
  3. Load runs. Step Functions stores the execution state durably before invoking the task.
  4. Verify runs. A failure here catches to Compensate, which deletes the loaded data.
  5. Happy path: Verify passes and the execution reaches SUCCEED with the full output.
  6. Failure path: Load raises after retries. Catch routes to Compensate, then to FAIL.
Notice: the execution history records every state transition. After a failure you can open the console, find the exact failing state, inspect its input and the error, and fix the bug before redriving.
Scenario A

In your world

The situation

Your nightly ETL pipeline is three Lambda functions chained by SQS: Extract → Transform → Load. When Transform fails, Extract has already completed and its output sits in the Transform SQS queue. When Load fails, Transform output sits in the Load queue. Your on-call process is: look at CloudWatch Logs, figure out which Lambda failed, manually replay the SQS message.

What actually breaks

A schema change in the source file breaks Transform. The Extract output is correct and is in the Transform queue, but the Transform Lambda fails on every message and moves them to the DLQ. You spend 20 minutes finding the failure in logs across three Lambda functions and two queues.

What you'd do on Monday

Move the chain into a Step Functions Standard workflow. Extract, Transform and Load are Task states. Transform has a Retry for transient errors (connection timeouts) and a Catch for schema errors that routes to a notification state. When Transform fails after retries, the console shows the failing state, the input payload and the exact error. You redrive the execution from the Transform state with the corrected payload.

PythonStep Functions state machine definition for ETL pipeline (needs AWS SDK)
# ASL (Amazon States Language) definition - deploy via CloudFormation or CDK
STATE_MACHINE = {
    "Comment": "Nightly orders ETL",
    "StartAt": "Extract",
    "States": {
        "Extract": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:ap-south-1:123456789012:function:orders-extract",
            "Next": "Transform",
            "Retry": [{"ErrorEquals": ["Lambda.ServiceException"], "MaxAttempts": 3,
                        "IntervalSeconds": 5, "BackoffRate": 2}],
        },
        "Transform": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:ap-south-1:123456789012:function:orders-transform",
            "Next": "Load",
            "Retry": [{"ErrorEquals": ["States.TaskFailed"], "MaxAttempts": 2}],
            "Catch": [{"ErrorEquals": ["SchemaError"], "Next": "NotifySchemaFailure"}],
        },
        "Load": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:ap-south-1:123456789012:function:orders-load",
            "End": True,
        },
        "NotifySchemaFailure": {"Type": "Task",
            "Resource": "arn:aws:states:::sns:publish",
            "Parameters": {"TopicArn": "arn:aws:sns:ap-south-1:123456789012:etl-alerts",
                           "Message.$": "$.error"},
            "End": True},
    },
}
Scenario B

Imagine you work at Expedia

400,000 booking orchestration executions per day

The numbers

Illustrative: 400,000 executions/day ≈ 4.6/s. Each execution has 8 states (search, hold, validate, charge, confirm, notify, audit, end) = 8 state transitions × 400,000 executions = 3.2M transitions/day. At $0.025 per 1,000 Standard workflow transitions (check the current pricing page), that is $80/day. At Express workflow pricing (check current page), a 2-second execution would be substantially cheaper.

What the Scenario-A answer does here

Standard workflow is correct for payment execution—exactly-once Task semantics matters when calling the card processor. The scenario A fix (Step Functions for the ETL chain) works but the billing model becomes expensive if you also run Express-suitable pipelines through Standard workflows.

What flips at this scale

The flip is to use Standard for the customer-facing booking flow (charge, confirm, audit—where exactly-once matters) and Express for the high-volume search and price-check flows (where idempotency lets you tolerate at-least-once). The decision is not aesthetic; it is driven by whether a Task being called twice causes an incorrect business outcome.

The trade-off you'd defend

I would defend Standard for anything involving payment or audit, Express for internal high-volume idempotent work, accepting more workflow definitions and more careful routing in exchange for correct billing and correct execution semantics.

Illustrative execution counts; not Expedia-published figures. Step Functions pricing as of mid-2025; check the current pricing page.

Gotchas

Express workflow calls a Task more than once

Symptom

A Lambda inside an Express workflow is invoked twice for the same execution step; a database row is inserted twice.

Cause

Express workflows have at-least-once execution semantics. A transient failure during the task invocation may cause Step Functions to retry the task call itself, not just the state transition.

Fix

Use Standard workflows for non-idempotent tasks. If you must use Express, make every Task idempotent (upsert, conditional write, idempotency key).

Catch without compensation leaves partial state

Symptom

The workflow catches a Load error and transitions to FAIL, but the previously-loaded rows from a partial batch are never cleaned up.

Cause

Catch transitions to a new state; it does not automatically undo previous states.

Fix

Add a Compensation task state that runs in the Catch path before the Fail state. The compensation must be idempotent (calling it twice should be safe).

Execution history quota for Standard workflows

Symptom

A long-running workflow with many Map iterations fails with ExecutionHistorySizeExceeded.

Cause

Standard workflow history stores every event. Each Map iteration adds events; a Map over 10,000 items can hit the 25,000-event history limit (check the current quota).

Fix

Use the Map state's maxConcurrency to limit parallelism, or use Express workflows for the Map sub-workflow.

Tricky questions

Does Standard workflow guarantee exactly-once business effect?
Most people say

Yes; Step Functions guarantees exactly-once execution.

Actually

Step Functions guarantees that each Task state is started at most once per execution step—not that the downstream service produces exactly one business effect. If the Lambda writes to a database and returns success, but Step Functions does not receive the response (network partition), it may start the Task again. Your Lambda still needs an idempotency key.

Can you restart a failed Step Functions execution from the failing state?
Most people say

No; you must restart from the beginning with the original input.

Actually

Step Functions supports "redrive" for failed Standard workflow executions (as of 2023; check current documentation). You can restart from the failed state using the last known input, without rerunning the states that already succeeded.

Scenario question

You're handed a three-Lambda ETL chain: Extract reads from S3, Transform normalises fields, Load writes to Redshift. The chain runs 200 times a day on different S3 prefixes. When Load fails, the on-call manually identifies the S3 prefix, reruns Transform by hand and retries Load. How would you redesign?

Model answer

I would wrap the chain in a Step Functions Standard workflow with one execution per S3 prefix. Extract, Transform and Load are Task states. Load gets a Retry of three attempts with exponential backoff for transient Redshift errors. A Catch on Load routes to a Notify state that posts to SNS with the S3 prefix and error. The execution history in the console shows the exact failing state and its input: no log archaeology. For failed executions, I can redrive from Load with the same input once the Redshift issue is resolved. The trade-off is that I now manage a state machine definition alongside the Lambda code, but the debugging and retry automation pay for that operational cost many times over at 200 executions per day.

Interview corner

CommonWhat is the difference between Standard and Express Step Functions workflows?
“I'd say Standard is durable, stores history for 90 days, runs up to a year, and gives exactly-once Task execution—use it for payments and critical data loads. Express is high-volume, short (max 5 minutes), at-least-once—use it for idempotent pipelines where you need throughput over durability guarantees.”
TrickStep Functions guarantees exactly-once—so your Lambda doesn't need idempotency, right?
“I'd call that the common misread. Step Functions guarantees the state transition happens at most once, but the underlying task—a Lambda or API call—can be called more than once due to network issues. The Lambda still needs idempotency. Step Functions takes care of the workflow state; you take care of the effect.”
DesignHow would you add a human-approval step to an ETL pipeline?
“I'd use a Task state with a callback pattern: the state publishes a message to SNS, then waits for a taskToken to be sent back via SendTaskSuccess or SendTaskFailure. The reviewer gets an email with a link that calls the API, and the workflow resumes. The wait can be up to a year in a Standard workflow.”
Experience checkWhen have you seen state-machine thinking in your ETL work?
“Every time I manually rebuilt retry logic and branch points in Python—if load fails, un-reserve S3, notify Slack, stop—that was an implicit state machine. Step Functions makes that explicit and debuggable. The main benefit I'd sell to my team is not needing to look through four log groups to find where a nightly job failed.”

Memory hook

Step Functions is the flowchart you can actually run. Every box, arrow and decision is visible in the console, and the execution pauses exactly where it failed.

If you remember only one thing:

Standard workflow = exactly-once Task start, up to 1 year. Express = at-least-once, max 5 minutes. Pick based on whether a duplicate Task call corrupts data.

Self-check

  1. What is the maximum execution duration for a Standard Step Functions workflow?

    1 year (check the current documentation; this is the documented value as of mid-2025).

  2. What is the maximum execution duration for an Express workflow?

    5 minutes (check the current documentation).

  3. Why does a Catch state still require compensation logic from the developer?

    Catch only transitions the execution to a new state. It does not undo previous states automatically. You must write a Compensation Task that reverses the effects of prior steps.

Prove it

Write a Python state machine executor that runs a five-state pipeline (extract → validate → transform → load → notify) where the load step fails on the first attempt. The Retry fires once and the second attempt succeeds. Print the execution log showing every state transition and retry attempt. Confirm that the final state is SUCCEED.

aws-06-05Card 5 of 6

Idempotent consumers under failure

HardP2 · Very likelyKnow cold14 min
idempotencyconsumersduplicate-detectiondlqpoison-messages

Why you care

This is the highest-probability card in this chapter. The question "how do you handle duplicate messages" appears in almost every backend interview because the answer reveals whether you understand distributed systems at a practical level. SQS at-least-once, SNS fan-out, EventBridge retries, Step Functions task reruns—they all share one assumption: your consumer handles the same input twice without producing a duplicate business effect. If you cannot explain an idempotency key, you will either silently double-charge customers or lose data trying to avoid it. At TCS, your ETL jobs run on Jenkins and each run touches Postgres; the same correctness requirement applies at the load step.

Explain like I'm 5

Idempotent means "doing it twice has the same effect as doing it once". A toggle button is idempotent: clicking "turn on" when the light is already on changes nothing. A light switch with a plus button (add one unit of brightness) is not idempotent: pressing it twice is different from pressing it once. Your consumer needs to be a toggle, not a plus button.

Core concept

Three ingredients make a consumer idempotent:

  1. Idempotency key: a stable, unique identifier derived from the message—the SQS message ID, the order ID, a combination of fields that uniquely identifies this business event. Do not use a random UUID generated at receive time; it changes with each delivery.
  1. Key store: a place to record which keys have already been processed. The store must be durable and visible to all consumers (a Postgres table, DynamoDB item, or Redis key). A local in-process dict does not survive a worker restart.
  1. Atomic write: the idempotency key record and the business effect must be committed in the same transaction, or one must be a conditional write that acts as both. The classic mistake is writing the key after the business effect in a separate commit; a crash between the two leaves the effect applied but the key unrecorded, so the next delivery sees no key and applies the effect again.

Timeout after side effect. The most dangerous case: the Lambda writes to Postgres, commits, then times out before returning success to Step Functions or before calling DeleteMessage. The caller retries. Your consumer receives the same message again with no key recorded (because the key write and the DB write were in separate transactions). The fix is to write the key in the same transaction as the business effect.

Out-of-order messages. Standard SQS delivers in approximate order. An ORDER_CANCELLED event can arrive before ORDER_SHIPPED. Write your state transitions defensively: an ORDER_CANCELLED that arrives when the order is already CANCELLED is a no-op.

Poison messages. A message that always causes your consumer to crash (corrupt JSON, schema mismatch, value that triggers a bug) will exhaust maxReceiveCount and move to the DLQ. Do not redrive it until you understand why it failed. Add a structured parser that catches JSON and schema errors and routes them to a quarantine table instead of crashing; those messages never increment receive count and never clog the DLQ.

Under the hood

The "exactly-once effect" is a lie at the infrastructure level. No distributed queue guarantees it. What you can build is effectively-once: the effect is applied if and only if the idempotency key is not already recorded. In Postgres, this is an INSERT ... ON CONFLICT DO NOTHING combined with a SELECT to check whether the row was inserted in this call. In DynamoDB, it is a conditional write (condition_expression="attribute_not_exists(pk)"). The subtlety is that between the SELECT (key absent) and the INSERT (key written), a concurrent duplicate delivery can also find the key absent and proceed. On a Postgres table with a primary key constraint on the idempotency key column, the second INSERT will raise a unique violation; catch it and return the stored result. This is a serialised race that Postgres handles correctly for you.

Minimal example Idempotency key store: same key returns cached result; different params raises

PythonIdempotency store: same key returns cached result; different params raisesruns
import sqlite3, hashlib, json

db = sqlite3.connect(":memory:")
db.execute("""
    CREATE TABLE idem (key TEXT PRIMARY KEY, params_hash TEXT,
                       result TEXT, status TEXT)
""")
db.commit()

def idempotent(key, params, fn):
    h = hashlib.sha256(json.dumps(params, sort_keys=True).encode()).hexdigest()[:8]
    row = db.execute(
        "SELECT params_hash, result FROM idem WHERE key=?", (key,)
    ).fetchone()
    if row:
        if row[0] != h:
            raise ValueError(f"{key!r} already used with different params")
        print(f"  [{key}] cached")
        return json.loads(row[1])
    result = fn(params)
    db.execute("INSERT INTO idem VALUES (?,?,?,?)",
               (key, h, json.dumps(result), "ok"))
    db.commit()
    print(f"  [{key}] processed")
    return result

def charge(p): return {"charged": True, "amount": p["amount"]}

r1 = idempotent("txn-001", {"amount": 500}, charge)
r2 = idempotent("txn-001", {"amount": 500}, charge)   # duplicate
print(f"r1={r1}  r2={r2}  same={r1 == r2}")
try:
    idempotent("txn-001", {"amount": 999}, charge)     # different params
except ValueError as e:
    print(f"ValueError: {e}")
Output
  [txn-001] processed
  [txn-001] cached
r1={'charged': True, 'amount': 500}  r2={'charged': True, 'amount': 500}  same=True
ValueError: 'txn-001' already used with different params

In production Consumer with idempotency, poison-message quarantine and DLQ

PythonConsumer: idempotent upsert, bad-JSON quarantine, duplicate skipruns
import sqlite3, hashlib, json

db = sqlite3.connect(":memory:")
db.executescript("""
    CREATE TABLE orders  (id TEXT PRIMARY KEY, amt REAL);
    CREATE TABLE idem    (key TEXT PRIMARY KEY, status TEXT);
    CREATE TABLE poison  (msg_id TEXT, reason TEXT);
""")

def consume(msg_id, payload_str):
    key = f"sqs:{msg_id}"
    if db.execute("SELECT 1 FROM idem WHERE key=?", (key,)).fetchone():
        return "skip"
    try:
        p = json.loads(payload_str)
        order_id, amount = p["order_id"], float(p["amount"])
        if amount <= 0:
            raise ValueError()
    except json.JSONDecodeError:
        with db:
            db.execute("INSERT INTO poison VALUES (?,?)", (msg_id, "bad-json"))
        return "poison:bad-json"
    except (KeyError, ValueError):
        with db:
            db.execute("INSERT INTO poison VALUES (?,?)", (msg_id, "bad-schema"))
        return "poison:bad-schema"
    with db:
        db.execute("INSERT OR IGNORE INTO orders VALUES (?,?)", (order_id, amount))
        db.execute("INSERT INTO idem VALUES (?,?)", (key, "ok"))
    return "ok"

MSGS = [
    ("m1", '{"order_id":"ORD-1","amount":"499.00"}'),
    ("m2", '{"order_id":"ORD-2","amount":"-1"}'),
    ("m3", "not-json"),
    ("m1", '{"order_id":"ORD-1","amount":"499.00"}'),  # duplicate
    ("m4", '{"order_id":"ORD-3","amount":"250.00"}'),
]

for mid, payload in MSGS:
    print(f"  {mid}: {consume(mid, payload)}")

print(f"orders={db.execute('SELECT COUNT(*) FROM orders').fetchone()[0]}  "
      f"poison={db.execute('SELECT COUNT(*) FROM poison').fetchone()[0]}")
Output
  m1: ok
  m2: poison:bad-schema
  m3: poison:bad-json
  m1: skip
  m4: ok
orders=2  poison=2

See it happen Idempotency key check on every delivery

SQSconsumerPostgresdelivery 1 (msg m1)SELECT key 'sqs:m1'not foundBEGIN; INSERT order; INSERT key; COMMITDeleteMessagedelivery 2 (msg m1, timeout redelivery)SELECT key 'sqs:m1'found, status=okDeleteMessage
  1. First delivery of message m1. Consumer receives it and starts processing.
  2. Consumer checks the idempotency key table. Key is absent.
  3. Key absent: consumer proceeds to apply the business effect.
  4. Both the order row and the idempotency key are written in the same transaction.
  5. Only after the commit does the consumer acknowledge the message.
  6. Visibility timeout expired before a previous DeleteMessage (or SQS redelivered). Same message arrives again.
  7. Consumer checks again. This time the key is present.
  8. Key found: consumer returns the stored result without re-applying the effect.
  9. Consumer acknowledges safely. Zero duplicate writes.
Notice: the idempotency key check happens on every delivery, and both writes go in the same transaction. Splitting them into two commits creates a gap where a retry can slip through.
Scenario A

In your world

The situation

Your ETL Lambda reads order files from S3, writes to Postgres and returns success to the SQS consumer. The Lambda timeout is set to 30 seconds; the Postgres INSERT ... SELECT occasionally takes 28 seconds on large files. The SQS visibility timeout is also 30 seconds.

What actually breaks

Postgres commits at t=28 seconds. The Lambda tries to call DeleteMessage and times out at t=30 seconds before the SQS API call returns. SQS redelivers the message. Your Lambda runs again, inserts the same rows (Postgres has no unique constraint on the natural key), and logs 0 errors. The orders table now has duplicate rows. Finance notices doubled revenue figures the next morning.

What you'd do on Monday

Two changes. First, add a unique constraint on the natural key (order_id, source_date) and use INSERT ... ON CONFLICT DO NOTHING. Second, write an idempotency key (using the SQS message ID) in the same Postgres transaction as the insert. On redelivery, the SELECT on the key table returns a row and the Lambda returns immediately without touching orders.

PythonIdempotent SQS consumer in Lambda (needs boto3 + psycopg)
import json, hashlib
import boto3
import psycopg2

sqs = boto3.client("sqs", region_name="ap-south-1")

def handler(event, context):
    for record in event["Records"]:
        msg_id = record["messageId"]
        receipt = record["receiptHandle"]
        payload = json.loads(record["body"])

        with psycopg2.connect(DSN) as conn, conn.cursor() as cur:
            cur.execute("SELECT 1 FROM idem WHERE key=%s", (f"sqs:{msg_id}",))
            if cur.fetchone():
                continue  # already processed

            cur.execute(
                "INSERT INTO orders (order_id, amount) VALUES (%s, %s) "
                "ON CONFLICT (order_id) DO NOTHING",
                (payload["order_id"], payload["amount"]),
            )
            cur.execute("INSERT INTO idem (key, status) VALUES (%s, 'ok')",
                        (f"sqs:{msg_id}",))
            # Both writes commit together; no gap for a retry
Scenario B

Imagine you work at Lenskart

800,000 order events per day with 0.1% SQS redelivery rate

The numbers

Illustrative: 800,000 events/day × 0.1% redelivery ≈ 800 duplicates/day. Without idempotency: 800 duplicate order inserts silently → wrong inventory counts → wrong daily revenue report. With idempotency key check (one SELECT per message): 800,000 SELECTs/day ≈ 9.3/s on the Postgres idem table. At 0.1 ms per lookup that is negligible overhead.

What the Scenario-A answer does here

The Scenario A fix (unique constraint + idempotency key in the same transaction) handles all 800 duplicates correctly. The risk at scale is key table growth: 800,000 rows/day × 365 days = 292M rows/year without cleanup. The idem table starts adding query plan cost after a few million rows without an index.

What flips at this scale

At this scale you add two things. An index on key (it is the primary key, so it already exists, but also consider a covering index if you query by status). A retention job that deletes keys older than your SQS message retention window (14 days max), because you will never see a redelivery of a message older than that. 292M → 11.2M live rows at any time (14 days × 800,000/day).

The trade-off you'd defend

I would defend idem table retention at one full SQS retention window plus a safety margin, accepting a background delete job in exchange for a bounded table. Keeping all keys forever buys nothing; SQS cannot redeliver a message after its retention expires.

Illustrative event rates; not Lenskart-published figures.

Gotchas

Idempotency key written after the business effect

Symptom

Duplicate rows appear in the DB even though idempotency keys exist; there is no key for some transactions.

Cause

The key write and the business effect are in separate transactions. A crash between commit 1 (effect) and commit 2 (key) leaves the effect applied but the key absent; the next delivery sees no key and applies the effect again.

Fix

Always write the key in the same transaction as the effect. If you cannot (different stores), write the key first; an unprocessed-but-keyed record is far easier to clean up than a duplicate effect.

Reusing an idempotency key with different parameters

Symptom

A second caller with a different order amount reuses a payment idempotency key (e.g., the client retried with a corrected amount but the same token). The consumer silently applies the first amount again and returns success.

Cause

The idempotency key check only verifies the key exists; it does not check whether the parameters match.

Fix

Hash the key parameters alongside the key (as in the minimal example) and raise an error if the hash differs. Return the stored result only when parameters also match.

DLQ misread as a consumer bug when it is a poison message

Symptom

DLQ depth grows steadily; every consumed message eventually lands there; the consumer code has no bug.

Cause

The messages themselves are corrupt or schema-invalid. The consumer raises on parse, the receive count increments, and messages eventually move to the DLQ.

Fix

Add a parser that catches JSON and schema errors before the business logic and routes those messages to a quarantine table. Poison messages never reach the business logic and never increment receive count.

Tricky questions

Is a UUID generated at receive time a valid idempotency key?
Most people say

Yes; it is unique for each invocation.

Actually

It is unique but unstable across deliveries. On the second delivery, a new UUID is generated, the key store has no match, and the effect is applied again. The key must be derived from the message content, not from the processing context.

Does INSERT ... ON CONFLICT DO NOTHING make a consumer idempotent?
Most people say

Yes; the DB constraint prevents duplicates.

Actually

A unique DB constraint prevents duplicate rows for the same natural key, but it does not prevent other side effects (API calls, S3 writes, downstream events) from happening again. True idempotency requires the key store pattern, not just a DB constraint.

Scenario question

You're handed a payment Lambda triggered by SQS. It calls an external payment gateway API, then writes the result to Postgres. The gateway charges the customer on every call with the same parameters. How do you make it idempotent?

Model answer

I would add an idempotency key to the gateway API call. Most payment gateways (Stripe, Razorpay) support an Idempotency-Key header that causes the gateway to return the cached response if the same key is replayed within a time window. I would use the SQS message ID as the key. On the Postgres side, I would also write a key record in the same transaction as the payment result row, so a retry that the gateway deduplicates (returning 200) does not re-write the Postgres row either. The trade-off is that if I use a random UUID as the idempotency key header (the naive approach), each retry generates a new key and the gateway charges again. The fix is stable key derivation from stable inputs.

Interview corner

CommonWhat is an idempotency key and why do you need one with SQS?
“I'd describe it as a stable, unique identifier for a business operation that the consumer checks before processing. I need it with SQS because SQS delivers at-least-once: the same message can arrive twice. Without the key, processing twice means two DB rows, two API calls, two charges.”
TrickYou write the idempotency key after the DB insert. What's the failure window?
“I'd explain: if the process crashes between the insert commit and the key commit, the insert is applied but the key is absent. The next delivery finds no key and inserts again. The fix is to write both in the same transaction.”
Experience checkHow do you handle a message that always makes the consumer crash?
“I'd add a structured parser before the business logic that catches JSON decode errors and schema violations and routes them to a quarantine table. Those messages never hit the business logic, never increment the receive count and never reach the DLQ. The DLQ then only contains messages that parsed correctly but failed for a business reason—much easier to diagnose.”
DesignHow do you size the idempotency key table?
“I'd set retention to one full SQS message retention window—up to 14 days—plus a safety margin. A message older than that can never be redelivered, so the key buys nothing. A daily delete job keeps the table bounded. At 800,000 events/day and 14-day retention, that is about 11 million live rows, easily managed with a primary-key index.”

Memory hook

Idempotent means toggle, not plus button. Check the key before you act, write it in the same transaction as the act.

If you remember only one thing:

Idempotency key and business effect must commit together. Any other order has a gap.

Self-check

  1. What makes a message a "poison message" distinct from a message that fails due to a transient error?

    A poison message consistently fails every delivery (corrupt JSON, invalid schema, a bug triggered by specific values). A transient failure resolves on retry. Poison messages need a parser-level catch that routes them to quarantine before the business logic ever sees them.

  2. Why does the idempotency key derivation matter?

    The key must be stable across redeliveries. A key derived from the SQS message ID (which does not change between deliveries) is correct. A key generated from a UUID at receive time changes on each delivery and provides no protection.

  3. What is the risk of not cleaning up old idempotency keys?

    The key table grows unboundedly. Query time and storage cost increase. Old keys for messages that can never be redelivered (beyond SQS retention) provide no benefit.

Prove it

Extend the production consumer to simulate a worker crash: process message m5 (valid order), commit the order row, then raise an exception before the idempotency key commit. Show that on the next delivery, the order row already exists and the consumer handles the race (unique constraint + key check) without inserting a duplicate. Print the order and key table counts after both deliveries.

aws-06-06Card 6 of 6

Orchestration versus choreography

HardP3 · Role-dependentTalk about it13 min
orchestrationchoreographyarchitecturestep-functionseventbridge

Why you care

Every time a new team asks "can we hook into the order flow?", you are making a choreography-versus-orchestration decision. Get it wrong and you either end up with a tangled event chain where a failure in step 4 is invisible until step 7, or a central Step Functions orchestrator that every team waits on. There is no universal right answer, but there is a framework: orchestration owns the business outcome and can compensate; choreography scales horizontally but loses failure visibility. Interviewers ask this question at the design level because it reveals whether you think about who owns the failure.

Explain like I'm 5

Orchestration is a conductor directing an orchestra. One conductor knows the full score. If the violins come in late, the conductor stops everyone and restarts from bar 12. Choreography is a jazz band: each musician listens to the others and responds, with no conductor. If the bassist misses a cue, nobody stops the band—the bassist just catches up, or the piece goes on without the bass line. Jazz is more flexible; orchestral failure recovery is more explicit.

Core concept

Orchestration uses a central coordinator—Step Functions, or a dedicated orchestrator Lambda—that calls each service in sequence (or parallel), tracks what completed, and runs compensation on failure. Key properties: the orchestrator knows the full execution graph, failure is detected at the coordinator, and rollback is explicit and auditable.

Choreography uses events. Each service subscribes to events it cares about and emits new events when it completes. There is no central coordinator; the flow emerges from event subscriptions. Key properties: services are decoupled and can be added or removed by adding or removing subscriptions; failure detection requires correlating distributed logs or a separate monitoring service.

Decision rule: Use orchestration when a business transaction has a defined outcome, must be compensated atomically if it fails, or requires audit of every step (payments, bookings, regulated data loads). Use choreography when services are truly independent, the flow can tolerate partial failures gracefully, and adding new consumers is a self-service operation for other teams.

Hybrid. The common production pattern is both: a Step Functions workflow orchestrates the core business transaction (reserve, charge, confirm), and EventBridge emits events from the terminal states of that workflow for downstream consumers (notifications, analytics, audit) to react to choreographically. The outcome is owned by the orchestrator; the downstream reactions are choreographed.

Failure tracing. Orchestration failure is visible in the Step Functions execution console with the exact failing state, input and error. Choreography failure requires correlation IDs in every event and a log aggregation query across services. If your team is not set up to trace across services, choreography failures become "I don't know why the customer never got the SMS".

When not to use orchestration: if the number of subscribers changes frequently, requiring orchestrator updates on every addition, you lose the self-service property that makes choreography attractive. Each new subscriber to an SNS-EventBridge choreography adds zero code to the publisher.

Under the hood

The fundamental difference is where state lives during execution. In orchestration, state is held by the coordinator (Step Functions execution input/output, or a Lambda's local variables). If the coordinator fails mid-execution, the state is durable in Step Functions or must be rebuilt from a checkpoint. In choreography, state is implicit: it is the set of events that have been emitted and consumed. Tracing the "state" of an order requires querying every event store (SQS, Kinesis, EventBridge archive) for a given correlation ID. This is why distributed tracing tools (AWS X-Ray, OpenTelemetry) become mandatory in choreographed systems at scale: without them, "where is my order?" is unanswerable from a single console.

Minimal example Orchestration tracks completed steps; choreography has no undo list

PythonOrchestration vs choreography: failure surfaces differentlyruns
DONE = []

def orchestrate(steps, fail_at=""):
    for name, fn in steps:
        try:
            fn()
            DONE.append(name)
        except Exception as e:
            print(f"  orch FAIL at {name!r}: {e}")
            print(f"  orch can undo: {DONE[::-1]}")
            return "failed"
    return "ok"

def choreograph(events, fail_at=""):
    for event, handler in events:
        try:
            handler()
            print(f"  choreo {event!r}: ok")
        except Exception as e:
            print(f"  choreo {event!r}: FAIL {e} (no undo list)")
            return "failed"
    return "ok"

ok = lambda: None
fail = lambda: (_ for _ in ()).throw(RuntimeError("gateway timeout"))

print("--- orchestration (fails at charge) ---")
DONE.clear()
r = orchestrate(
    [("reserve", ok), ("charge", fail), ("notify", ok)], fail_at="charge"
)
print(f"  result: {r}")

print("--- choreography (fails at RESERVED event) ---")
r = choreograph(
    [("ORDER_PLACED", ok), ("RESERVED", fail), ("CHARGED", ok)],
    fail_at="RESERVED",
)
print(f"  result: {r}")
Output
--- orchestration (fails at charge) ---
  orch FAIL at 'charge': gateway timeout
  orch can undo: ['reserve']
  result: failed
--- choreography (fails at RESERVED event) ---
  choreo 'ORDER_PLACED': ok
  choreo 'RESERVED': FAIL gateway timeout (no undo list)
  result: failed

In production Hybrid: orchestrated core, choreographed downstream reactions

PythonHybrid pattern: orchestrator owns the transaction; events drive optional reactionsruns
import sqlite3

db = sqlite3.connect(":memory:")
db.executescript("""
    CREATE TABLE orders   (id TEXT PRIMARY KEY, status TEXT);
    CREATE TABLE event_log(event TEXT, order_id TEXT);
""")

# --- orchestrated core (owns the outcome) ---
def reserve(order_id):
    db.execute("INSERT OR IGNORE INTO orders VALUES (?,?)", (order_id, "reserved"))
    db.commit()

def charge(order_id):
    db.execute("UPDATE orders SET status='charged' WHERE id=?", (order_id,))
    db.commit()

def orchestrate(order_id, fail_at=""):
    completed = []
    for name, fn in [("reserve", reserve), ("charge", charge)]:
        if name == fail_at:
            print(f"  orch FAIL at {name}")
            # compensate: undo in reverse
            for past in reversed(completed):
                print(f"  orch compensate: undo {past}")
            return "failed"
        fn(order_id)
        completed.append(name)
        print(f"  orch {name}: ok")
    return "confirmed"

# --- choreographed downstream (reacts to terminal events) ---
def on_order_confirmed(order_id):
    db.execute("INSERT INTO event_log VALUES (?,?)", ("ORDER_CONFIRMED", order_id))
    db.commit()
    print(f"  choreo analytics: logged ORDER_CONFIRMED for {order_id}")

def on_order_confirmed_notify(order_id):
    print(f"  choreo notify: SMS sent for {order_id}")

DOWNSTREAM = [on_order_confirmed, on_order_confirmed_notify]

def run(order_id, fail_at=""):
    status = orchestrate(order_id, fail_at=fail_at)
    if status == "confirmed":
        print(f"  emitting ORDER_CONFIRMED -> {len(DOWNSTREAM)} downstream handlers")
        for handler in DOWNSTREAM:
            try:
                handler(order_id)
            except Exception as e:
                print(f"  choreo handler error (non-fatal): {e}")
    return status

print("=== happy path ===")
print(f"result: {run('ORD-1')}")
print("=== failure path ===")
print(f"result: {run('ORD-2', fail_at='charge')}")
events = db.execute("SELECT event, order_id FROM event_log").fetchall()
print(f"event_log: {events}")
Output
=== happy path ===
  orch reserve: ok
  orch charge: ok
  emitting ORDER_CONFIRMED -> 2 downstream handlers
  choreo analytics: logged ORDER_CONFIRMED for ORD-1
  choreo notify: SMS sent for ORD-1
result: confirmed
=== failure path ===
  orch reserve: ok
  orch FAIL at charge
  orch compensate: undo reserve
result: failed
event_log: [('ORDER_CONFIRMED', 'ORD-1')]
Scenario A

In your world

The situation

Your team's order service calls five downstream services via REST in sequence: inventory, payments, notifications, analytics and audit. Each call is a synchronous HTTP call. When payments fails, inventory has already reserved stock. There is no compensation and no audit of what happened.

What actually breaks

Payments goes down. Every order attempt partially reserves inventory, fails at payments and returns 500. After 30 minutes you have 4,000 phantom reservations in inventory that are never released, and finance sees 0 successful payments for the afternoon. Nobody has a list of which inventory items to un-reserve.

What you'd do on Monday

Wrap the core transaction (reserve + charge + confirm) in a Step Functions Standard workflow with Catch states that fire an un-reserve compensation step. Notifications, analytics and audit react to the ORDER_CONFIRMED EventBridge event from the terminal state; they do not block the core transaction. The Step Functions execution history gives you the exact list of orders that failed after reserve but before charge.

PythonHybrid wiring: Step Functions for core, EventBridge for downstream (needs AWS)
import boto3, json

sfn = boto3.client("stepfunctions", region_name="ap-south-1")
events = boto3.client("events", region_name="ap-south-1")

# Start the orchestrated core
execution = sfn.start_execution(
    stateMachineArn="arn:aws:states:ap-south-1:123456789012:stateMachine:order-core",
    input=json.dumps({"order_id": "ORD-42", "amount": 499}),
)

# Terminal success state in the Step Functions definition publishes to EventBridge:
# EventBridgePutEvents task -> sends {"source": "orders.core", "detail-type": "ORDER_CONFIRMED"}
# Downstream subscribers (analytics Lambda, SMS Lambda, audit SQS) react independently.
Scenario B

Imagine you work at Rapido

1.5 million ride bookings per day

The numbers

Illustrative: 1.5M bookings/day ≈ 17/s average, 100/s peak. Core booking flow: match driver, charge wallet, confirm ride = 3 states in a Standard Step Functions workflow. 1.5M × 3 transitions = 4.5M transitions/day (check current Step Functions pricing). Downstream reactions (SMS, analytics, driver payout calculation) react to RIDE_CONFIRMED events on EventBridge: adding a new downstream consumer (e.g., a carbon-offset calculation service) requires one new EventBridge subscription, zero changes to the orchestrator.

What the Scenario-A answer does here

A fully orchestrated approach adds SMS, analytics and payout as states in the Step Functions workflow. The workflow now has 8 states. Adding a ninth consumer (carbon offset) requires a Step Functions state machine update, a deployment, and an on-call change window. At 17/s executions, a bug in the carbon-offset Lambda blocks every booking until the Lambda is fixed and the workflow is redeployed.

What flips at this scale

The flip is the hybrid. Core business states (match, charge, confirm) stay in Step Functions because they must compensate atomically. Terminal-state events go to EventBridge. Each downstream team subscribes independently. A bug in carbon-offset never blocks a booking; the consumer just fails and lands in its own DLQ. Adding the tenth consumer is self-service.

The trade-off you'd defend

I would defend the hybrid over pure choreography for the core booking flow, because I need atomic compensation when charge fails. I would defend the hybrid over pure orchestration for downstream reactions, because I do not want to redeploy the booking workflow every time a new analytics team wants ride events. The trade-off is maintaining both patterns and ensuring every event carries a correlation ID for end-to-end tracing.

Illustrative booking volumes; not Rapido-published figures.

Gotchas

Choreography failure is invisible without correlation IDs

Symptom

A customer reports they never received an order confirmation SMS. Logs show the order succeeded. You spend an hour tracing across four Lambda log groups to find that the SNS delivery to the SMS gateway timed out.

Cause

Choreography has no central execution record. Without correlation IDs in every event and log line, tracing a failure requires manual log archaeology.

Fix

Inject a correlation ID (the order ID or a trace ID) into every event at the publisher and propagate it through every handler. Use AWS X-Ray or OpenTelemetry to build end-to-end traces.

Orchestrator becomes a bottleneck

Symptom

All downstream teams wait for the orchestrator team to deploy before their feature goes live. Every new consumer requires an orchestrator change.

Cause

A fully orchestrated flow treats every downstream action as a workflow step that the orchestrator owns.

Fix

Use the hybrid: the orchestrator owns only the business-critical transaction. Terminal-state events trigger choreographed downstream reactions without orchestrator changes.

Compensation is not automatic in choreography

Symptom

A downstream service fails mid-chain. The upstream service already committed its write. No rollback happens.

Cause

In choreography, there is no coordinator to fire compensation. Each service owns its own failure handling.

Fix

Either design compensating events (ORDER_CANCEL undoes ORDER_PLACED) that subscribers must handle, or move the transaction to an orchestrated flow with explicit Catch → Compensate states.

Tricky questions

Does choreography mean you do not need idempotency?
Most people say

Yes; each service reacts to a specific event, so it only fires once.

Actually

EventBridge and SNS deliver at-least-once. A service reacting to ORDER_CONFIRMED may receive the event twice. Every choreography handler still needs idempotency.

Is orchestration always more observable than choreography?
Most people say

Yes; the orchestrator has a complete execution record.

Actually

Only for the flow the orchestrator controls. Downstream choreographed reactions are not in the orchestrator's execution history. True end-to-end observability requires correlation IDs and distributed tracing regardless of pattern.

Scenario question

You're handed a monolith that processes orders synchronously: validate → charge → notify → analytics → audit. The analytics team wants to add three new event types next quarter. The notify step occasionally times out and blocks the charge result from reaching analytics. How would you redesign?

Model answer

I would separate the business-critical path from the downstream reactions. The orchestrated core becomes: validate → charge → confirm (three Step Functions states, with Catch on each for compensation). On confirmation, the workflow emits an ORDER_CONFIRMED event to EventBridge. Notify, analytics and audit each subscribe to that event via their own SQS queues. The analytics team adds new event types by creating new EventBridge rules and subscriptions, with zero changes to the order service or the orchestrator. Notify's timeout no longer blocks charge or analytics. The trade-off is that I lose the synchronous response that analytics currently reads from the notify result; analytics must now operate from the event, which may arrive up to a few seconds later. If analytics needs a synchronous response, I would reconsider, but for async reporting that lag is usually acceptable.

Interview corner

CommonWhat is the difference between orchestration and choreography?
“I'd say orchestration has a central coordinator that calls each service and tracks completion; it can compensate explicitly. Choreography has no central coordinator—each service reacts to events. Orchestration is better when the outcome must be auditable and reversible. Choreography is better when services are independent and adding new consumers should be self-service.”
TrickYou use EventBridge for everything, so your system is choreography, right?
“Not necessarily. I can use EventBridge as a target router for Step Functions terminal states, keeping the core transaction orchestrated. The downstream reactions are choreographed. EventBridge is infrastructure; the pattern depends on whether there is a coordinator tracking the flow.”
DesignWhen would you use a hybrid pattern?
“I'd use a hybrid when the core business transaction—charge, confirm, reserve—needs atomic compensation and an audit trail, but downstream reactions like notifications, analytics and audit should be independent. The orchestrator owns the outcome; EventBridge delivers the terminal-state event to downstream subscribers who add themselves without touching the orchestrator.”
Experience checkHow do you debug a failure in a choreographed system?
“I start with the correlation ID, which should be in every event and every log line. I query the log aggregation tool for all log lines with that order ID, sorted by time. If we have X-Ray or OpenTelemetry, the trace view shows the full call chain. Without correlation IDs this is painful; I'd treat adding them as a P0 item before going to production.”

Memory hook

The conductor knows who missed their cue; the jazz band has to figure it out from the recording.

If you remember only one thing:

Orchestration owns the failure and can compensate. Choreography scales with subscribers but requires correlation IDs to trace failures.

Self-check

  1. In a hybrid pattern, which part is orchestrated and which is choreographed?

    The core business transaction (reserve, charge, confirm) is orchestrated by Step Functions or a coordinator. Downstream reactions (notify, analytics, audit) react choreographically to terminal-state events from the orchestrator.

  2. Why does choreography still require idempotency in consumers?

    EventBridge and SNS deliver at-least-once. A consumer reacting to ORDER_CONFIRMED may receive the event twice and must not apply the effect twice.

  3. What makes orchestration harder to extend than choreography?

    Every new consumer that needs to react to the workflow must be added as a state in the orchestrator, which requires a workflow update and deployment. In choreography, new consumers subscribe to existing events without touching the publisher.

Prove it

Extend the production example to simulate a downstream choreography handler that fails on the first attempt but succeeds on the second (use a counter). Show that the orchestrator's result is unaffected by the handler failure, and that the handler eventually succeeds on retry. Print the event_log table after both attempts.