SQS workers and DLQ recovery
Why you care
Every Jenkins ETL job you have written runs on a schedule. SQS lets the upstream service trigger the job the moment data arrives rather than one hour later. The catch is that SQS gives you at-least-once delivery, not exactly-once processing: your worker can crash after writing to Postgres but before deleting the message, and SQS will hand that same message to another worker. Until you understand the visibility timeout lifecycle and the dead-letter queue, you will either silently duplicate rows or lose messages with no error. Interviewers ask about SQS not to test AWS trivia but to find out whether you have thought about what "acknowledgement" means.
Explain like I'm 5
Imagine a ticket office where customers leave job requests in a box. You pick one up, and a curtain drops around you for ten minutes—nobody else can see that ticket. If you finish and drop the ticket in the bin, it is gone. If the curtain rises before you finish—because you took too long or crashed—the ticket reappears in the box and the next worker picks it up. After a ticket has been picked up three times without ever reaching the bin, the office moves it to a lost-and-found tray (the DLQ) so an operator can inspect it. Every detail in SQS maps to this physical story.
Core concept
SQS has two queue types. A standard queue delivers at least once, in roughly-FIFO order; duplicates are possible. A FIFO queue delivers exactly once per message group, in strict order, at lower throughput (check the current quota page for transactions-per-second limits).
The visibility timeout is the curtain. When a worker calls ReceiveMessage, the message becomes invisible for the configured duration (default 30 seconds, max 12 hours). The worker must call DeleteMessage before that window closes. If it does not—because it crashed, timed out or raised an unhandled exception—the message reappears and another worker receives it. The receive count increments each time.
A dead-letter queue is a separate standard or FIFO queue attached via a redrive policy. You configure maxReceiveCount (1 to 1000). When a message's receive count exceeds that value, SQS moves it to the DLQ automatically. The DLQ has its own retention period (1 minute to 14 days; check the current quota page for current defaults).
Key rule: DeleteMessage is the acknowledgement. Call it only after your side effect—the DB write, the S3 put, the downstream API call—has fully committed. Never call it speculatively before the work succeeds. If your work is idempotent, a retry from a redelivered message is harmless; if it is not, see Idempotent consumers under failure.
Visibility timeout extension. If your processing takes longer than the configured timeout, call ChangeMessageVisibility with more time before the window closes. Failure to do this is the most common cause of duplicate processing.
DLQ redrive. After you fix the bug that caused failures, use the SQS console or StartMessageMoveTask API to move messages from the DLQ back to the source queue. Test this in staging: a DLQ full of messages is not alarming by itself, but a DLQ that grows every day is.
Use a standard queue when order does not matter and throughput is the priority. Use a FIFO queue when processing order within a group is a correctness requirement, not just a preference.
Minimal example Visibility timeout redelivers; DLQ catches repeat failures
class SQSQueue:
def __init__(self, vt=10, max_rc=3):
self._msgs = {} # id -> [body, rc, invisible_until]
self._vt = vt
self._max = max_rc
self.dlq = []
def send(self, mid, body):
self._msgs[mid] = [body, 0, 0]
def receive(self, now):
for mid, m in self._msgs.items():
if m[2] <= now:
m[1] += 1
m[2] = now + self._vt
return mid, m[0], m[1]
return None
def delete(self, mid):
self._msgs.pop(mid, None)
def redrive(self, now):
gone = [mid for mid, m in self._msgs.items()
if m[1] >= self._max and m[2] <= now]
for mid in gone:
self.dlq.append(self._msgs.pop(mid)[0])
q = SQSQueue(vt=10, max_rc=3)
q.send("m1", "order:42")
t = 0
for _ in range(3):
mid, body, rc = q.receive(t)
print(f"t={t:2d} received {mid} rc={rc}")
t += 11 # visibility expires at t+10, so next receive at t+11
q.redrive(t)
print(f"queue={len(q._msgs)} dlq={len(q.dlq)} dlq_body={q.dlq[0]}")t= 0 received m1 rc=1
t=11 received m1 rc=2
t=22 received m1 rc=3
queue=0 dlq=1 dlq_body=order:42In production S3-event worker: commit first, then delete
import sqlite3
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE etl_log (s3_key TEXT PRIMARY KEY, rows INT, status TEXT)")
db.execute("CREATE TABLE dlq (msg_id TEXT, s3_key TEXT, error TEXT)")
db.commit()
QUEUE = [
{"id": "m1", "s3_key": "raw/orders/2024-01-15/part-001.csv", "rows": 850},
{"id": "m2", "s3_key": "raw/orders/2024-01-15/part-002.csv", "rows": 0},
{"id": "m3", "s3_key": "raw/orders/2024-01-15/part-003.csv", "rows": 720},
]
deleted, dlq_count = [], 0
for msg in QUEUE:
try:
if msg["rows"] == 0:
raise ValueError(f"empty file: {msg['s3_key']}")
db.execute("INSERT OR REPLACE INTO etl_log VALUES (?,?,?)",
(msg["s3_key"], msg["rows"], "ok"))
db.commit()
deleted.append(msg["id"])
print(f"ok {msg['id']} rows={msg['rows']}")
except ValueError as e:
db.execute("INSERT INTO dlq VALUES (?,?,?)",
(msg["id"], msg["s3_key"], str(e)))
db.commit()
dlq_count += 1
print(f"dlq {msg['id']} {e}")
committed = db.execute("SELECT COUNT(*) FROM etl_log").fetchone()[0]
print(f"committed={committed} deleted={len(deleted)} dlq={dlq_count}")ok m1 rows=850
dlq m2 empty file: raw/orders/2024-01-15/part-002.csv
ok m3 rows=720
committed=2 deleted=2 dlq=1See it happen SQS message lifecycle
- Producer sends the S3-event notification. Message is visible immediately.
- Worker polls and receives the message. Visibility timeout clock starts (default 30 s).
- Worker holds the receiptHandle. Message is invisible to all other pollers.
- Worker writes rows to Postgres and commits the transaction.
- Only after the commit succeeds does the worker acknowledge. Message is gone.
- If DeleteMessage is never called (worker crash), the message reappears. receiveCount increments.
- After maxReceiveCount deliveries with no delete, SQS moves the message to the DLQ for inspection.
In your world
The situation
Your Jenkins job processes order-events SQS messages that reference S3 keys. The Lambda-equivalent worker uses boto3 to receive messages, reads the S3 object, counts rows and writes to Postgres, then calls delete_message. The visibility timeout is the SQS default of 30 seconds.
What actually breaks
A file with 480,000 rows takes 45 seconds to parse and load. The visibility timeout expires during the load. A second worker picks up the same message, parses the same file, and inserts the same rows. Both workers eventually delete the message. The orders table now has duplicate rows—primary key violations if you have a natural key, silent duplicates if you do not. Jenkins shows green for both runs.
What you'd do on Monday
Before or during processing, extend the visibility timeout. Call ChangeMessageVisibility with a comfortable budget (say, 5 minutes) once you have confirmed the file size. Better still, make the upsert idempotent (ON CONFLICT DO NOTHING with a natural key) so that a duplicate delivery has no effect even if the extension fails. The visibility extension needs the receipt handle from ReceiveMessage.
import boto3
sqs = boto3.client("sqs", region_name="ap-south-1")
msg = sqs.receive_message(
QueueUrl=QUEUE_URL,
MaxNumberOfMessages=1,
WaitTimeSeconds=20, # long polling
)["Messages"][0]
# Extend before touching the DB
sqs.change_message_visibility(
QueueUrl=QUEUE_URL,
ReceiptHandle=msg["ReceiptHandle"],
VisibilityTimeout=300, # 5 minutes
)
try:
load_to_postgres(msg["Body"])
sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=msg["ReceiptHandle"])
except Exception:
pass # message reappears after 5 minutes; DLQ catches persistent failuresImagine you work at Swiggy
The numbers
Illustrative: 12,000 events/min = 200/s. A standard queue handles this easily (check current throughput quota). Each message represents one S3 object with ~1,000 rows. At 200 objects/s × 1,000 rows = 200,000 rows/s to Postgres. A 30-second visibility timeout means a worker has 30 seconds to process one file. At 200 ms/file (small CSV) that is fine; at 6 s/file (slow Postgres) a queue of 30 files backlog will all expire simultaneously, causing a thundering re-delivery.
What the Scenario-A answer does here
The Scenario A fix (extend visibility) still works per message, but with 200 concurrent workers each calling ChangeMessageVisibility every 60 seconds, that is 200 API calls/minute just for heartbeats.
What flips at this scale
At this scale the flip is to match the visibility timeout to the 99th-percentile processing time (measured, not guessed), so extensions are rarely needed. You also instrument ApproximateAgeOfOldestMessage as a lag alarm: if it climbs above two visibility-timeout windows, workers are falling behind and you scale out.
The trade-off you'd defend
I would defend a generous visibility timeout matched to real processing time, plus idempotent upserts, over a short timeout with aggressive heartbeats. The short-timeout + heartbeat approach wastes API calls and introduces a new failure mode (heartbeat thread crashes, message redelivers mid-write). The cost is that dead workers hold messages invisibly for longer before they reappear.
Illustrative arithmetic; not Swiggy-published figures.
Gotchas
Worker processes the message but never deletes it
Row counts in Postgres grow after every Jenkins run; your audit table shows repeated S3 keys being processed.
An exception after the DB commit but before DeleteMessage lets the message reappear after the visibility timeout expires.
Use a try/finally or context manager that calls DeleteMessage only on success, and ensure your upsert is idempotent so redeliveries are harmless.
Visibility timeout expires mid-load
Two workers finish successfully; the DB shows duplicate or conflicting rows; both report success to Jenkins.
Processing took longer than the visibility timeout; the message became visible and a second worker picked it up.
Extend the timeout before processing starts, or match the timeout to measured 99th-percentile load time.
DLQ messages are not alarmed
Bugs silently drop messages into the DLQ for days; nobody notices until downstream data is stale.
SQS does not alert by default when the DLQ depth climbs.
Add a CloudWatch alarm on ApproximateNumberOfMessagesVisible for the DLQ with a threshold of 1 or more, and page the on-call.
Tricky questions
Yes; once you call it, the message is deleted.
DeleteMessage is idempotent but not atomic across replicas. In rare cases on a standard queue, a copy of the message that was already in flight to another poller before your delete can still be delivered once. FIFO queues have stronger guarantees. Always write idempotent consumers.
Yes; the shorter the timeout, the faster a failed message reappears.
A short timeout also causes spurious redeliveries for slow-but-correct workers. Use a timeout matched to the actual processing budget, not the shortest possible value.
Scenario question
You're handed an SQS-driven ETL job that loads S3 files into Postgres. The DLQ depth is 3,000 after a weekend. The source files are fine. Your predecessor set maxReceiveCount to 2 and visibility timeout to 10 seconds. The average file load takes 12 seconds. What happened and how do you fix it?
Model answer
The visibility timeout of 10 seconds is shorter than the 12-second average load time. Almost every message expired before the worker could delete it, incremented receive count on the second delivery, and moved to the DLQ when it hit maxReceiveCount of 2. No data loss—the files are fine—but no data was loaded either. I would: first, increase visibility timeout to at least 60 seconds (five times the average, to cover the tail); second, increase maxReceiveCount to at least 5 to tolerate genuine transient failures; third, redrive the DLQ back to the source queue and monitor the DLQ alarm. The trade-off is that a genuinely failing message now takes longer to reach the DLQ, but that is acceptable compared to discarding all messages.
Interview corner
CommonWhat is the SQS visibility timeout and why does it matter?
TrickYou call DeleteMessage immediately after receiving, before the load. What breaks?
Experience checkWhen would you use a FIFO queue over a standard queue?
DesignHow do you detect that your SQS workers are falling behind?
Memory hook
The visibility timeout is the curtain. If the worker drops the ticket before the curtain falls, it is gone. If not, the ticket reappears.
DeleteMessage is the only acknowledgement. Call it after the side effect commits, never before.
Self-check
What happens to a message if the worker calls DeleteMessage twice with the same receipt handle?
The second call succeeds silently; DeleteMessage is idempotent. No error is raised.
What is the maximum visibility timeout you can set on an SQS message?
12 hours (check the current SQS documentation for the exact value; this is the documented maximum as of mid-2025).
Why should DLQ retention be set longer than source queue retention?
Messages in the DLQ need time for an operator to inspect and redrive them. If the DLQ retention is shorter than the time it takes to notice and respond, messages expire before they can be fixed.
Extend the minimal example to show the full redrive workflow: after messages land in the DLQ, "fix the bug" (set a flag), move them back to the source queue, and process them successfully. Print the state of the source queue and DLQ after each phase.