Shelf · Core Eng · Chapter 3

Chapter 3: HTTP & the web contract

Every API call, webhook and browser request your services handle rides on this contract - methods, status codes, headers, cookies, CORS, TLS and the protocol version underneath. By the end you can read a request/response pair like a native, explain exactly why POST is never idempotent, and debug a slow or failing call with curl instead of guessing.

Difficulty
Priority
ce-03-01Card 1 of 6

HTTP wire contract: messages and observable outcomes

MediumP1 · Must knowKnow cold14 min
httpstatus-codesrequest-responsewebhooks
Needs: nothing. Start here.

Why you care

Every backend bug report eventually reduces to "what did the server actually say back," and the answer is a status code plus headers, not a feeling. Interviewers ask "what's the difference between 401 and 403" or "what does 202 mean" precisely because those answers reveal whether you have only ever glanced at a 200/404/500 and guessed at everything else. In ETL and integration work specifically, webhooks and partner callbacks are just HTTP requests arriving instead of leaving, and treating them differently from a client call you make is the source of half the "why did we double-process this" incidents. If you cannot tell "accepted" apart from "completed" at a glance, you will build something that lies about its own state.

Explain like I'm 5

An HTTP request is a filled-out courier form: the request line says where it's going and what kind of job it is (deliver, replace, remove), the headers are the special instructions taped to the box (fragile, cash on delivery, my account number), and the body is what's actually inside. The response is the courier's receipt: a status code that says what happened (delivered, refused, come back later), headers with details, and sometimes a body with more information. A "202 Accepted" is the courier taking the box and handing you a tracking slip - it says nothing about whether the parcel has arrived yet. A "200 OK" on a status check is the confirmation that it actually got there.

Core concept

An HTTP message has three parts on each side: a request line (method, path, version) or status line (version, status code, reason phrase), a set of headers (case-insensitive name: value pairs, repeatable), and an optional body. Headers describe the message; the body carries the payload. A connection carries one or more of these request/response pairs, one at a time per HTTP/1.1 connection (Trace TLS and HTTP/1.1, HTTP/2 and HTTP/3 covers what changes for HTTP/2 and HTTP/3). A webhook is not a different protocol - it's an ordinary HTTP request your service receives instead of sends, which is why everything in this card applies to writing a receiver, not just a client (Receive authenticated, replay-aware webhooks).

Status codes are grouped by their first digit, and the exact code inside each group matters:

  • 2xx - it worked. 200 the response body carries the result. 201 a resource was created; expect a Location header pointing at it. 204 it worked and there is deliberately no body (a delete, a health check). 202 is the odd one out: the request was accepted, not completed - the work happens later, out of band, and HTTP gives you no built-in mechanism to be told when it finishes; you poll or you get a separate callback.
  • 3xx - go elsewhere. 301/302 mean "moved" (permanently/temporarily); historically, some clients turned a redirected POST into a GET, which is exactly the kind of surprise that makes redirects risky for anything but GET/HEAD. 307/308 are the same permanent/temporary pair but promise to preserve the method and body - use these when a POST must stay a POST after the redirect.
  • 4xx - your request, your problem. 400 malformed or unparsable. 401 means "I don't know who you are" (no or invalid credentials). 403 means "I know who you are, and no." 404 resource not found - or deliberately used instead of 403 to avoid confirming something exists. 409 the request conflicts with the current state of the resource (two people editing the same row). 422 the request is well-formed JSON/XML but semantically invalid (a negative amount, a missing required field). 429 you're being rate-limited; a well-behaved server sends Retry-After.
  • 5xx - the server's problem. 500 generic, unhandled failure. 502 a gateway/proxy got an invalid response from whatever is behind it. 503 temporarily unavailable (overloaded, draining, maintenance) - responses here should not be cached. 504 a gateway timed out waiting on whatever is behind it.

The decision rule: pick the status that describes what actually happened, not the closest one you remember. If work is deferred, that's 202, not 200 with a note in the body. If the client is wrong, use the most specific 4xx you have; if the client did nothing wrong and it's you, use the most specific 5xx. Never encode the real outcome only in the body while returning 200 - anything downstream that reads status codes (caches, load balancers, retry logic, monitoring) will believe the status line, not your JSON.

Minimal example One server, three outcomes you can read off the status line

PythonStatus and headers tell the whole storyruns
import http.server, threading, http.client

class Handler(http.server.BaseHTTPRequestHandler):
    def log_message(self, *a): pass
    def do_GET(self):
        if self.path == "/health":
            self.send_response(204)
            self.end_headers()
        else:
            self.send_response(404)
            self.send_header("Content-Type", "text/plain")
            self.end_headers()
            self.wfile.write(b"not found")
    def do_POST(self):
        self.send_response(201)
        self.send_header("Location", "/orders/1")
        self.end_headers()

server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()

conn = http.client.HTTPConnection("127.0.0.1", port)
conn.request("GET", "/health")
r = conn.getresponse(); print(r.status, r.reason); r.read()

conn.request("POST", "/orders", body=b'{"amount":100}', headers={"Content-Type":"application/json"})
r = conn.getresponse(); print(r.status, r.getheader("Location")); r.read()

conn.request("GET", "/missing")
r = conn.getresponse(); print(r.status, r.read())

server.shutdown()
Output
204 No Content
201 /orders/1
404 b'not found'

In production Accepted now, complete later - and a conflict in between

Python202 now, 200 later: acknowledge is not completeruns
import http.server, threading, http.client, time

orders = {}  # id -> "processing" | "done"

class Handler(http.server.BaseHTTPRequestHandler):
    def log_message(self, *a): pass

    def do_POST(self):
        if self.path == "/orders":
            length = int(self.headers.get("Content-Length", 0))
            self.rfile.read(length)
            order_id = str(len(orders) + 1)
            orders[order_id] = "processing"
            self.send_response(202)
            self.send_header("Location", f"/orders/{order_id}")
            self.end_headers()
        elif self.path.endswith("/cancel"):
            order_id = self.path.split("/")[2]
            if orders.get(order_id) == "done":
                self.send_response(409)
                self.end_headers()
            else:
                orders[order_id] = "cancelled"
                self.send_response(200)
                self.end_headers()
        else:
            self.send_response(404)
            self.end_headers()

    def do_GET(self):
        order_id = self.path.split("/")[-1]
        status = orders.get(order_id)
        if status is None:
            self.send_response(404)
            self.end_headers()
        elif status == "processing":
            self.send_response(202)
            self.end_headers()
        else:
            self.send_response(200)
            self.end_headers()

server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()
conn = http.client.HTTPConnection("127.0.0.1", port)

conn.request("POST", "/orders", body=b"{}")
r = conn.getresponse(); loc = r.getheader("Location"); print("create:", r.status, loc); r.read()

conn.request("GET", loc)
r = conn.getresponse(); print("poll while processing:", r.status); r.read()

orders["1"] = "done"          # stand-in for the worker finishing the job
conn.request("GET", loc)
r = conn.getresponse(); print("poll after worker finished:", r.status); r.read()

conn.request("POST", "/orders/1/cancel")
r = conn.getresponse(); print("cancel after done:", r.status); r.read()

server.shutdown()
Output
create: 202 /orders/1
poll while processing: 202
poll after worker finished: 200
cancel after done: 409

See it happen Accept is not complete

clientserverPOST /orders202 Accepted, Location: /orders/1GET /orders/1202 Acceptedworker finishes the orderGET /orders/1200 OK
  1. Client sends the request line, headers and body in one message.
  2. The server has only accepted the work; nothing about the order is guaranteed done yet.
  3. Client polls the status endpoint named in Location.
  4. Still processing: same status family, no new information beyond 'not yet'.
  5. The actual completion happens out of band, driven by a worker, not by this HTTP exchange.
  6. Client polls again.
  7. Only now does the status code say the work is actually complete.
Notice: 202 tells you the request was accepted, not that the order exists yet; only the later 200 says the work is done.
Scenario A

In your world

The situation

Your nightly job posts each of the day's 800 refunds to an internal ledger API and logs 200 OK for every call, treating any 2xx as "done."

What actually breaks

The ledger API actually returns 202 Accepted while it queues refunds for a fraud check, then 200 once approved and 409 if a refund was already reversed. Your job's audit log shows 800 successes; finance later finds only 640 refunds actually posted, because 160 are still sitting in the fraud queue and your code never checked which 2xx it got.

What you'd do on Monday

Branch on the exact code instead of 2xx in general, and poll or subscribe for the ones that were only accepted:

PythonBranch on the exact status, not just "2xx" (needs the real ledger API + network)
resp = post_refund(payload)
if resp.status_code == 201:
    mark_done(payload["id"])
elif resp.status_code == 202:
    queue_for_followup(payload["id"], resp.headers["Location"])
elif resp.status_code == 409:
    log.warning("refund %s already reversed, skipping", payload["id"])
else:
    raise RuntimeError(f"unexpected {resp.status_code} for refund {payload['id']}")
Scenario B

Imagine you work at Akamai

2M requests/minute through one edge PoP

The numbers

Illustrative: one edge point of presence handling 2,000,000 requests/minute is about 33,333 requests/second. If the origin behind it returns 502/504 for even 0.1% of that traffic under load, that's ~33 failed requests every second, or roughly 2.9 million failed edge-to-origin calls in a single day - small percentages become large absolute counts once you multiply by request volume, not request count per script run.

What the Scenario-A answer does here

The Scenario A fix (branch on the exact status) still holds for a single service, but "just return 200 with an {"ok": false} field so nothing downstream has to think about status codes" breaks badly here: intermediate caches, retry logic and monitoring at the edge make automated decisions off the status line itself, and a 200 body they never parse looks identical to a real success.

What flips at this scale

At CDN scale, the status code is a machine-readable contract other systems act on without reading your body: a cache decides whether to store the response, a load balancer decides whether to retry against another origin, and an alert decides whether to page someone - all based on the exact code and headers like Retry-After, not on inspecting JSON.

The trade-off you'd defend

I would defend precise status codes plus Retry-After on every error path over a uniform 200 with an error flag in the body, trading a slightly more complex client-side branch for correct caching, retry and alerting behavior at a scale where nobody is reading bodies by hand.

Illustrative traffic and error-rate figures chosen to make the arithmetic concrete; not Akamai-published numbers.

Gotchas

Returning 200 with the real outcome buried in the body

Symptom

Dashboards and logs show 100% success while the business operation actually failed.

Cause

The status line is what caches, proxies, retry logic and monitoring read; a 200 says "success" to all of them regardless of what the JSON body says.

Fix

Return the status that matches the outcome (4xx/5xx/202) and keep the body for detail, not as the only source of truth.

Confusing 401 with 403

Symptom

A client shows "please log in" when the real problem is a permissions gap, or shows "access denied" for someone who was never authenticated at all.

Cause

401 means the server doesn't know who's asking (missing/invalid credentials); 403 means it does know, and says no anyway.

Fix

Return 401 only when authentication itself failed or is missing; return 403 once identity is established but the action is disallowed.

A 302 quietly turns your POST into a GET

Symptom

An automated client following a redirect after a POST arrives at the next URL with no body and a GET, silently dropping data.

Cause

302/301 have historically been treated by clients as license to switch the method to GET.

Fix

Use 307/308 when the method and body must be preserved across the redirect.

429 without Retry-After

Symptom

A rate-limited client backs off for an arbitrary guessed duration, either hammering the server again too soon or waiting far longer than necessary.

Cause

429 alone tells the client it was throttled but not for how long.

Fix

Always send Retry-After on 429 (and on 503), and have clients honor it instead of guessing.

Tricky questions

Is 404 always "you're not allowed to see this"?
Most people say

No - 404 means "not found," full stop, so it's only used when the resource genuinely doesn't exist.

Actually

Servers deliberately return 404 instead of 403 when they don't want to confirm a resource exists at all (a private repo, another user's record). 403 reveals existence and denies access; 404 denies existence itself. Which one you choose is a security decision, not just an accuracy one.

Does a 500 mean my code has a bug?
Most people say

Yes, 500 always means the application threw an unhandled exception.

Actually

500 is the generic catch-all for "something went wrong and I have no better code for it." A well-designed service maps known failure modes to specific codes - 502/503/504 for upstream problems, 409/422 for a bad-but-well-formed request - and reserves 500 for genuinely unexpected failures, so a spike in 500s specifically (not 4xx) is the signal worth paging on.

Scenario question

You're handed a partner integration spec that says "call our endpoint; a 200 response means the operation completed." Your own client needs to know definitively whether money moved before it updates its own ledger. How do you build the client?

Model answer

I would not trust "200 means done" as the whole story without checking what the partner actually documents for their async paths - if any endpoint can return 202, I treat that as "accepted, not complete" and poll a status URL (or subscribe to a webhook) until I see a terminal code. Even where they claim synchronous 200 completion, I would still handle 409 (already applied) and 5xx (unknown outcome, safe to retry only if idempotent) explicitly rather than assuming every non-200 is the same kind of failure. The trade-off is more branches in the client than "if 200, done" - but the alternative is a ledger that's wrong the first time the partner's "always synchronous" claim turns out to have an exception.

Interview corner

CommonWalk me through what happens, at the message level, when a client calls an API.
“I'd describe the client sending a request line, headers and an optional body, and the server replying with a status line, headers and an optional body. The status code is the one field everything downstream - caches, retries, alerting - reads without looking at the body, which is why picking the right one matters.”
TrickA client gets 200 back from a POST, but the operation actually failed halfway through. What's wrong?
“I'd say the server is lying with its status line - a 200 should mean the request fully succeeded. I'd fix the server to return the status that matches reality, like a 5xx for a partial failure or a 409 for a conflict, instead of always saying 200 and putting the truth only in the body.”
Experience checkWhere have status codes bitten you in real ETL or integration work?
“I've seen a job treat every 2xx as 'loaded' when the API actually used 202 for async acceptance and 200 only once approved, which meant the audit log said more rows landed than actually did. Now I branch on the exact code, not the status family.”
CommonWhat's the practical difference between 202 and 200?
“I'd say 200 means the work described by the request is done and here's the result, while 202 means the request was only accepted and the actual work happens later. HTTP gives me no built-in way to be notified when that finishes, so I poll a status resource or wait for a separate callback.”

Memory hook

A courier's tracking slip (202) is not the same as a signed delivery receipt (200) - one says "I took it," the other says "it arrived."

If you remember only one thing:

The status code, not the body, is what the rest of the system trusts - make it tell the truth.

Self-check

  1. What's the difference between 401 and 403?

    401 means the server doesn't know who you are (no/invalid credentials); 403 means it knows and refuses anyway.

  2. Why is 202 different from every other 2xx code?

    It means the request was accepted, not completed - the work happens later, out of band, with no HTTP-native way to be notified when it finishes.

  3. Why should 301/302 redirects be treated carefully for a POST?

    Some clients historically turn the redirected request into a GET, silently dropping the method and body; use 307/308 when the method must be preserved.

Prove it

Extend the minimal example's server with a /orders/<id>/cancel route that returns 409 if the order is already done, and a client script that creates an order, polls it twice, marks it done, then tries to cancel it. Paste the four status codes your script prints, in order, and explain in one sentence why the last one is a 409 and not a 200.

ce-03-02Card 2 of 6

Apply REST semantics without promising exactly once

MediumP1 · Must knowKnow cold12 min
restidempotencysafe-methodsretries

Why you care

"Is PUT idempotent?" and "why is POST never idempotent?" are knockout questions for a reason: retries are the single most common failure-recovery mechanism in distributed systems, and retrying the wrong kind of request duplicates the side effect instead of fixing the problem. In ETL and payments-adjacent work, this is the difference between a Jenkins job that safely resumes after a timeout and one that double-inserts, double-charges or double-ships. Knowing which methods are safe, which are idempotent, and that those are two different properties, is what lets you design a retry policy instead of hoping one works.

Explain like I'm 5

Ringing a doorbell (GET) never changes anything no matter how many times you press it - that's safe. Setting a thermostat to exactly 20 degrees (PUT) can be done any number of times and the room ends up at 20 degrees either way - that's idempotent, even though it does change something. Placing an order by shouting it to a waiter (POST) is neither: shout it twice and you get two dinners, because the waiter has no way to know your second shout was a repeat of the first, not a new order.

Core concept

Two independent properties describe an HTTP method's effect, and confusing them is the source of most retry bugs:

  • Safe: the request has no side effects on the server's state; it only reads. GET and HEAD are safe.
  • Idempotent: making the same request N times has the same effect on server state as making it once (N ≥ 1). GET, HEAD, PUT and DELETE are idempotent. PUT is idempotent because it replaces a resource with the given representation - do it once or five times, the end state is the same. DELETE is idempotent because deleting an already-deleted resource leaves the same end state (though the status code on the second call may differ, see the gotchas below). POST and PATCH are not idempotent by definition: POST typically creates a new resource each time it's called (two identical POSTs can mean "please create two things"), and PATCH applies a delta that can produce a different result depending on the resource's current state.

The decision rule: idempotency is about the property of the method as specified, not a guarantee your specific implementation honors it - a PUT handler that applies a delta instead of replacing state is not really idempotent no matter what the spec says it should be. And critically, idempotent is not the same as "safe to retry blindly forever." A client can't always tell whether a POST that timed out actually committed on the server before the response was lost - the request was sent, but the acknowledgment never arrived. If you retry that blindly, you may create the resource twice. The fix isn't "make POST idempotent" (you can't, by definition); it's giving the operation an explicit deduplication key so a retried POST is recognized as a repeat, not a new request - see the production example.

Minimal example Three POSTs make three orders; three PUTs make one profile

PythonPOST creates every time; PUT replaces every timeruns
import http.server, threading, http.client

count = {"posts": 0}
resource = {"name": None}

class Handler(http.server.BaseHTTPRequestHandler):
    def log_message(self, *a): pass
    def do_POST(self):
        count["posts"] += 1
        self.send_response(201); self.end_headers()
    def do_PUT(self):
        length = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(length).decode()
        resource["name"] = body
        self.send_response(200); self.end_headers()

server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()
conn = http.client.HTTPConnection("127.0.0.1", port)

for _ in range(3):
    conn.request("POST", "/orders"); r = conn.getresponse(); r.read()
print("orders created by 3 identical POSTs:", count["posts"])

for _ in range(3):
    conn.request("PUT", "/profile", body=b"Asha")
    r = conn.getresponse(); r.read()
print("profile after 3 identical PUTs:", resource["name"])
server.shutdown()
Output
orders created by 3 identical POSTs: 3
profile after 3 identical PUTs: Asha

In production An Idempotency-Key turns a retried POST into a no-op

PythonSame key, same charge - the second and third attempts are replays, not new workruns
import http.server, threading, http.client, json

charges = []              # ledger: what actually got charged
seen_keys = {}             # Idempotency-Key -> response already sent

class Handler(http.server.BaseHTTPRequestHandler):
    def log_message(self, *a): pass
    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        payload = json.loads(self.rfile.read(length) or b"{}")
        key = self.headers.get("Idempotency-Key")

        if key and key in seen_keys:
            charge_id = seen_keys[key]
            self.send_response(200)          # replayed, not re-executed
        else:
            charge_id = f"ch_{len(charges) + 1}"
            charges.append((charge_id, payload["amount"]))
            if key:
                seen_keys[key] = charge_id
            self.send_response(201)

        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps({"charge_id": charge_id}).encode())


server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()
conn = http.client.HTTPConnection("127.0.0.1", port)

body = json.dumps({"amount": 499}).encode()
headers = {"Content-Type": "application/json", "Idempotency-Key": "order-77-attempt"}

for attempt in range(3):                     # simulate a client retrying after a timeout
    conn.request("POST", "/payments", body=body, headers=headers)
    r = conn.getresponse()
    print(f"attempt {attempt}: status={r.status} body={r.read().decode()}")

print("charges actually recorded:", charges)
server.shutdown()
Output
attempt 0: status=201 body={"charge_id": "ch_1"}
attempt 1: status=200 body={"charge_id": "ch_1"}
attempt 2: status=200 body={"charge_id": "ch_1"}
charges actually recorded: [('ch_1', 499)]
Scenario A

In your world

The situation

Your Jenkins job posts each settlement to an internal API with requests.post(url, json=row, timeout=30) inside a for row in rows loop, and retries the whole row on any exception, including requests.exceptions.Timeout.

What actually breaks

One night, the API takes 31 seconds to commit settlement id 88214 - the insert succeeds server-side, but the response never makes it back before your 30-second timeout fires. Your retry loop re-posts the same row. The API has no way to know this is a retry, not a new settlement, so it creates a second row. Finance finds settlement 88214 credited twice, for ₹42,000, three weeks later during reconciliation.

What you'd do on Monday

Give every settlement a natural, stable key and let the database reject the duplicate instead of trusting the client to only send it once:

PythonInsert with a natural key so a retried settlement can't duplicate (needs psycopg + a real connection)
cur.execute(
    """
    INSERT INTO settlements (event_id, amount, status)
    VALUES (%s, %s, 'applied')
    ON CONFLICT (event_id) DO NOTHING
    """,
    (event_id, amount),
)
if cur.rowcount == 0:
    log.info("settlement %s already applied, retry ignored", event_id)
Scenario B

Imagine you work at Shopify

5,000 checkout POSTs/second during a flash sale

The numbers

Illustrative: a flash sale pushes checkout submissions to 5,000 POSTs/second across storefronts. If a brief network blip makes just 0.5% of client connections time out after the server already committed, that's 25 requests/second whose client believes it failed and may retry - roughly 1,500 potential duplicate orders in a single minute-long blip, entirely from retries of requests that had actually already succeeded.

What the Scenario-A answer does here

The Scenario A fix (an idempotency key on the side-effecting call) still applies, but doing it only for the payment step and not the order-creation step leaves a gap: at this volume, "rare" double-submits during a network hiccup become a routine, multiple-times-per-minute occurrence, not an edge case you can shrug off.

What flips at this scale

At this scale, every endpoint that commits money, inventory or a shippable order needs the same treatment, and the idempotency-key store itself has to be a shared, durable service (a keyed table or cache with a TTL) that every checkout replica checks - not a dict living in one process's memory, which forgets everything on restart or deploy.

The trade-off you'd defend

I would defend a centralized, durable idempotency-key store across every side-effecting endpoint over "the client shouldn't retry so aggressively," trading the storage and lookup cost of that store for a guarantee that a network blip during a flash sale costs latency, not duplicate orders.

Illustrative traffic and duplication figures chosen to make the arithmetic concrete; not Shopify-published numbers.

Gotchas

A timeout after a committed side effect makes a blind retry duplicate work

Symptom

The same logical operation (a payment, an insert, an email) happens twice after exactly one client-visible timeout.

Cause

POST is not idempotent by definition - the server may have already committed the write before the response was lost in transit, and the client can't distinguish "never received" from "received but the reply got lost."

Fix

Attach an idempotency key (or a natural key the database can enforce uniqueness on) to every side-effecting call, and have the retry replay against that key instead of blindly resubmitting.

"PUT is idempotent" assumed for a handler that applies a delta

Symptom

Calling the same PUT twice produces a different result each time - a counter that keeps incrementing, a balance that keeps changing.

Cause

The handler applies a relative change (balance += amount) instead of setting an absolute state, which breaks the idempotency the method's contract promises.

Fix

Make PUT set the full, absolute representation of a resource; use POST/PATCH for deltas, with their own explicit deduplication if retried.

DELETE returns 404 on the second call and gets treated as an error

Symptom

A retried DELETE returns 404 Not Found instead of the 204 the first call got, and monitoring flags it as a failure.

Cause

The resource is genuinely gone after the first DELETE; the end state is idempotent (deleted either way), but the status code differs because there's nothing left to describe.

Fix

Treat 404 on a DELETE retry as success in the client, or have the server return 204 for both the first and any subsequent delete of the same id if you control that behavior.

Tricky questions

Is a PUT idempotent because it returns the same status code every time?
Most people say

Yes - if the status code repeats, the request is idempotent.

Actually

Idempotency is about the resulting state, not the status code. A PUT might legitimately return 201 on the first call (it created the resource) and 200 on every call after (it replaced an existing one) - different codes, same guarantee: the resource ends up identical regardless of how many times you send it.

Is it always safe to retry a GET without thinking about it?
Most people say

Yes, GET is safe by definition, so retrying it can never cause a problem.

Actually

The HTTP contract says GET should have no side effects, but that's a promise your implementation has to keep - a GET that increments an analytics counter or triggers a rate-limit check on every hit has a side effect the spec doesn't forbid at the transport level. "Safe" is a contract you write to, not a guarantee enforced for you.

Scenario question

You're handed a payments endpoint that returns 201 on success with no way for a client to retry safely after a network blip. Product wants retries to be safe without changing how clients call the endpoint's URL or method. How do you change the contract?

Model answer

I would add support for an Idempotency-Key header that clients can optionally send. On the first request with a given key, I process normally and store the key alongside the result. On any later request with the same key, I skip re-executing the side effect and just replay the stored result, returning 200 instead of 201 to signal "this already existed." The trade-off is a bit of storage (the key-to-result mapping needs a TTL so it doesn't grow forever) and a decision about how long a key stays valid for replay - but that's cheap next to the cost of a duplicate charge, and clients that don't send a key just keep the old, non-idempotent behavior.

Interview corner

CommonWhat is the difference between safe and idempotent?
“I'd say safe means no side effects at all - it's a read. Idempotent means side effects are allowed, but repeating the request any number of times leaves the same end state as doing it once. GET is both; PUT and DELETE are idempotent but not safe; POST is neither.”
TrickWhy is POST never idempotent, but a payment API can still be made safe to retry?
“I'd say POST's whole point is usually 'create something,' so by definition two identical POSTs can mean two things should exist - that's not something the method itself can fix. What makes retries safe is a deduplication mechanism layered on top, like an idempotency key, not the method's own semantics.”
Experience checkHave you been bitten by an unsafe retry in ETL work?
“Yes - a job retried a settlement POST after a 30-second timeout, and the original request had actually already committed server-side; the retry created a duplicate row that took weeks to surface in reconciliation. I fixed it with a unique natural key and an ON CONFLICT DO NOTHING insert.”
DesignHow would you design an idempotency-key mechanism for a payments API?
“I'd accept a client-supplied key per logical operation, store it with the operation's result and a TTL, and on a repeated key return the stored result instead of re-executing anything. I'd scope keys per endpoint so a key reused across two different operations doesn't collide, and I'd reject a reused key with a different payload as a client error rather than silently replaying the wrong result.”

Memory hook

Idempotent means doing it twice looks the same as doing it once - it doesn't mean nothing happened twice.

If you remember only one thing:

POST is never idempotent by the spec; retries need your own dedup key, not hope.

Self-check

  1. Name the four idempotent HTTP methods.

    GET, HEAD, PUT, DELETE.

  2. Why can a client not always tell whether a timed-out POST actually succeeded?

    The request may have been received and committed server-side before the response was lost in transit; a timeout only means the client didn't get an answer, not that nothing happened.

  3. What makes a retried DELETE still count as idempotent even though its status code changes?

    The resulting state (the resource is gone) is the same after one delete or five; only the status code (204 then 404) differs because there's nothing left to describe.

Prove it

Extend the production example so that calling /payments with the same Idempotency-Key but a different amount returns 409 Conflict instead of silently replaying the first charge. Write a script that sends the same key twice with different amounts and paste both responses, showing the second is rejected rather than replayed.

ce-03-03Card 3 of 6

Separate cookies, CORS and authorization

MediumP2 · Very likelyKnow cold12 min
cookiescorssame-sitebrowser-security

Why you care

"My frontend gets a CORS error but curl/Postman works fine" is one of the most common early-career support tickets, and it wastes hours precisely when someone treats CORS as a security feature of the server instead of a policy the browser enforces on itself. Interviewers ask about CORS because the wrong mental model - "CORS protects my API" - leads directly to shipping APIs that are wide open to any non-browser client while everyone feels safe because the browser complained once. Cookie attributes sit right next to this: getting SameSite or Secure wrong either breaks a login flow or silently weakens it.

Explain like I'm 5

CORS is a sticky note the server hands to the browser: "only these sites are allowed to read this reply." The browser is the one who actually enforces the note - it reads it after the response comes back and decides whether to let the page's JavaScript see the data. Nobody else in the room is checking that note: curl, Postman, a server calling your API directly, none of them care what the sticky note says, because they aren't the ones the note is written for. Before a request with unusual headers or methods, the browser also sends a preflight: it politely asks in advance, "if I send this exact request, will you let my page read the answer?" - and only sends the real request if the answer is yes.

Core concept

Cookies. The server sets a cookie with Set-Cookie in a response; the browser sends it back with Cookie on later requests to the same site. Three attributes matter: Secure (only sent over HTTPS), HttpOnly (invisible to JavaScript, document.cookie never sees it - your main defense against a cookie being stolen via XSS), and SameSite (Strict, Lax, or None, controlling whether the cookie is sent on cross-site requests; the default when unset is Lax). SameSite=None requires Secure - browsers reject a None cookie that isn't also marked secure.

CORS. The browser's default is the same-origin policy: JavaScript on a.example cannot read a response from b.example unless b.example's response says it's allowed to, via Access-Control-Allow-Origin. For a "non-simple" request - a method other than GET/HEAD/POST, a custom header, or a Content-Type outside a small safelist - the browser sends an OPTIONS preflight first, and only sends the real request if the preflight response allows the method and headers. When a request carries credentials (cookies, Authorization), the server must echo back the exact requesting origin in Access-Control-Allow-Origin and add Access-Control-Allow-Credentials: true - a wildcard * is not allowed together with credentials.

The decision rule, and the one fact worth repeating until it's automatic: CORS is enforced by the browser, not the server, and it is not authorization. It stops a victim's browser from letting attacker-controlled JavaScript quietly read your API on the victim's behalf using the victim's cookies. It does nothing to stop a direct call from curl, Postman, a mobile app, or another server - those never run the browser's same-origin check at all. Configuring CORS controls which frontends a browser will let read your responses; it is never a substitute for authenticating and authorizing every request server-side, which is what actually decides who is allowed to do what (Authentication and authorization: sessions and JWT).

Minimal example A non-browser client always sees the body, whatever the Access-Control-Allow-Origin says

Pythoncurl/http.client always reads the body; only a browser enforces the ACAO headerruns
import http.server, threading, http.client

class Handler(http.server.BaseHTTPRequestHandler):
    def log_message(self, *a): pass
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Access-Control-Allow-Origin", "https://allowed.example")
        self.end_headers()
        self.wfile.write(b'{"balance": 4200}')

server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()

conn = http.client.HTTPConnection("127.0.0.1", port)
conn.request("GET", "/account", headers={"Origin": "https://evil.example"})
r = conn.getresponse()
print("status:", r.status)
print("ACAO header:", r.getheader("Access-Control-Allow-Origin"))
print("body a non-browser client still reads:", r.read())
server.shutdown()
Output
status: 200
ACAO header: https://allowed.example
body a non-browser client still reads: b'{"balance": 4200}'

In production Answering a preflight and setting a cookie correctly

PythonOPTIONS preflight, then the real request with credentials and a locked-down cookieruns
import http.server, threading, http.client

class Handler(http.server.BaseHTTPRequestHandler):
    def log_message(self, *a): pass

    def do_OPTIONS(self):
        self.send_response(204)
        self.send_header("Access-Control-Allow-Origin", "https://app.example")
        self.send_header("Access-Control-Allow-Methods", "POST, GET, OPTIONS")
        self.send_header("Access-Control-Allow-Headers", "Content-Type, X-Request-Id")
        self.send_header("Access-Control-Max-Age", "600")
        self.end_headers()

    def do_POST(self):
        self.send_response(201)
        self.send_header("Access-Control-Allow-Origin", "https://app.example")
        self.send_header("Access-Control-Allow-Credentials", "true")
        self.send_header(
            "Set-Cookie",
            "session=abc123; Secure; HttpOnly; SameSite=Strict",
        )
        self.end_headers()


server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()
conn = http.client.HTTPConnection("127.0.0.1", port)

conn.request(
    "OPTIONS", "/login",
    headers={
        "Origin": "https://app.example",
        "Access-Control-Request-Method": "POST",
        "Access-Control-Request-Headers": "content-type",
    },
)
pre = conn.getresponse()
print("preflight:", pre.status, pre.getheader("Access-Control-Allow-Methods"))
pre.read()

conn.request("POST", "/login", body=b"{}", headers={"Origin": "https://app.example", "Content-Type": "application/json"})
r = conn.getresponse()
print("real request:", r.status, r.getheader("Set-Cookie"))
r.read()
server.shutdown()
Output
preflight: 204 POST, GET, OPTIONS
real request: 201 session=abc123; Secure; HttpOnly; SameSite=Strict
Scenario A

In your world

The situation

The frontend team reports that a POST from their React app at portal.company.com to your API at api.company.com fails in every browser with a console error, but works fine when they test the exact same call in Postman.

What actually breaks

You spend most of a day assuming it's an auth bug and rotating credentials, because "the request works" in Postman. It doesn't reach your application logs at all during the browser failures - the preflight OPTIONS request is being rejected before your handler ever runs, because your API returns Access-Control-Allow-Origin: * while the request carries cookies, and browsers refuse a wildcard origin combined with credentials.

What you'd do on Monday

Echo back the real origin from an explicit allow-list instead of a wildcard whenever the response involves credentials:

PythonEcho the real origin from an allow-list, never a wildcard, when cookies are involvedruns
import http.server, threading, http.client

ALLOWED_ORIGINS = {"https://app.example", "https://admin.example"}

class Handler(http.server.BaseHTTPRequestHandler):
    def log_message(self, *a): pass
    def do_GET(self):
        origin = self.headers.get("Origin", "")
        self.send_response(200)
        if origin in ALLOWED_ORIGINS:
            self.send_header("Access-Control-Allow-Origin", origin)
            self.send_header("Access-Control-Allow-Credentials", "true")
        self.end_headers()

server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()
conn = http.client.HTTPConnection("127.0.0.1", port)

conn.request("GET", "/data", headers={"Origin": "https://app.example"})
r = conn.getresponse(); print("known frontend:", r.getheader("Access-Control-Allow-Origin")); r.read()

conn.request("GET", "/data", headers={"Origin": "https://random-site.example"})
r = conn.getresponse(); print("unknown origin:", r.getheader("Access-Control-Allow-Origin")); r.read()
server.shutdown()
Output
known frontend: https://app.example
unknown origin: None
Scenario B

Imagine you work at Postman

10,000 dev-time requests/day that never trigger a CORS check

The numbers

Illustrative: 50 engineers across 12 teams each run about 200 Postman requests/day while building against your APIs - roughly 10,000 requests/day that verify status codes and payloads but never once exercise a browser's same-origin or CORS check, because Postman doesn't run one. If even 5% of endpoints ship with a subtly wrong Access-Control-Allow-Origin, that's dozens of endpoints "verified working" by Postman and broken for every real browser user the moment they ship.

What the Scenario-A answer does here

The Scenario A fix (an explicit origin allow-list) still holds, but "test it in Postman/curl before shipping" doesn't generalize as a verification strategy at all, because neither tool ever enforces CORS - a broken configuration is invisible in exactly the tool everyone uses to sign off on a change.

What flips at this scale

At organization scale, you can't rely on a person remembering to open a real browser and check devtools for every endpoint. The flip is an automated check - in CI or at the gateway - that sends a request with an Origin header and asserts the response carries the expected Access-Control-Allow-Origin/Access-Control-Allow-Credentials pair, catching the class of bug before a human ever needs to notice a browser error.

The trade-off you'd defend

I would defend an automated header-assertion check over trusting manual Postman verification, trading a small amount of pipeline complexity for catching an entire category of "works for me, breaks in the browser" bugs before they reach production.

Illustrative development-time request volumes; not Postman-published usage data.

Gotchas

Wildcard origin combined with credentials

Symptom

The browser rejects the response with a CORS error even though the server returned 200 and an Access-Control-Allow-Origin: * header.

Cause

Browsers disallow the combination of a wildcard Access-Control-Allow-Origin with Access-Control-Allow-Credentials: true - a wildcard cannot be paired with credentialed requests.

Fix

Echo back the exact requesting origin from an allow-list, never *, whenever cookies or Authorization are involved.

SameSite=None without Secure

Symptom

A cookie meant to be sent cross-site silently never shows up on the request, in an otherwise correctly configured integration.

Cause

Modern browsers reject a SameSite=None cookie that isn't also marked Secure - it's simply dropped.

Fix

Always pair SameSite=None with Secure, which in practice means the cookie is HTTPS-only.

Treating a permissive CORS policy as access control

Symptom

An internal API ships with Access-Control-Allow-Origin: * "so the frontend just works," and months later someone calls it directly from a script and reads data across tenants with no browser involved at all.

Cause

CORS only ever blocks a browser acting on a page's behalf; it does nothing to a direct call from curl, a script, or another service.

Fix

Authenticate and authorize every request server-side regardless of what the CORS policy says - CORS decides who a browser will let read a response, not who is allowed to ask.

Tricky questions

If a request works in Postman, will it work from my browser frontend?
Most people say

Yes - it's the same HTTP request going to the same URL with the same headers.

Actually

No guarantee. Postman never runs the browser's same-origin policy or CORS preflight, so a request that looks identical on the wire can be blocked in a real browser while succeeding in Postman every time.

Does a strict CORS policy stop an attacker from reading my API's data?
Most people say

Yes, that's the whole point of CORS.

Actually

A strict policy stops a victim's browser, running attacker-controlled JavaScript on some other page, from reading your response using the victim's own cookies. It does nothing to stop the attacker from calling your API directly with their own tool and their own credentials - which is exactly why server-side authorization is still required.

Scenario question

You're handed a bug report - "the mobile app team says our signup API returns cross-origin errors, but curl -i works fine" - along with a request to "just add Access-Control-Allow-Origin: *" to fix it. The endpoint sets a session cookie.

Model answer

I would refuse the wildcard specifically because this response sets a cookie - a credentialed response - and browsers won't combine a wildcard origin with Access-Control-Allow-Credentials: true, so the wildcard wouldn't even fix the reported error. curl "working" is a red herring, since curl never enforces the check the browser is complaining about. I would add the app's real origin(s) to an explicit allow-list, return that exact origin, set Access-Control-Allow-Credentials: true, and make sure OPTIONS preflights are handled explicitly. The trade-off is a small allow-list to maintain going forward, versus the appearance of a fix that would have silently kept failing for every real browser user.

Interview corner

CommonWhat is CORS and who actually enforces it?
“I'd describe CORS as an HTTP-header mechanism where the server states which origins a browser should let read its responses. The key part is that the browser enforces it, not the server - a non-browser client never runs that check at all.”
TrickA request succeeds in curl or Postman but fails in the browser with a CORS error. What's going on?
“I'd explain that curl and Postman never enforce same-origin or CORS rules, so they can't reproduce this class of failure. I'd check the response headers - Access-Control-Allow-Origin and, if credentials are involved, Access-Control-Allow-Credentials - against the exact origin the browser is sending, using curl -i with an Origin header to simulate what the browser sees.”
Experience checkWhere has a CORS misconfiguration or cookie attribute bitten you or a teammate?
“I've seen a wildcard Access-Control-Allow-Origin ship alongside a session cookie, which meant browsers rejected it outright while Postman testing showed no problem at all, and it cost a day of debugging the wrong layer before anyone opened devtools.”
DesignWould you rely on cookie-based sessions or CORS configuration to protect a multi-frontend product?
“Neither on its own - CORS decides which frontends a browser will let read responses, and cookie attributes like HttpOnly and SameSite reduce specific attack surfaces, but the actual protection is server-side authentication and authorization on every request, which I'd never skip just because CORS looks locked down.”

Memory hook

CORS is a sticky note the server writes for the browser - nobody else in the room has to read it.

If you remember only one thing:

CORS is browser policy, not access control - authenticate and authorize every request regardless.

Self-check

  1. Who enforces the Access-Control-Allow-Origin header - the client, the server, or the browser?

    The browser. The server only states a policy; the browser decides whether to let the page's JavaScript read the response.

  2. What must always accompany SameSite=None?

    Secure - browsers reject a SameSite=None cookie that isn't also marked Secure.

  3. Why can't Access-Control-Allow-Origin be a wildcard when a response uses cookies?

    Browsers disallow the combination of a wildcard origin and Access-Control-Allow-Credentials: true; a credentialed response must echo a specific origin.

Prove it

Write two small servers: one returning Access-Control-Allow-Origin: * on a cookie-setting response, and one returning the real requesting origin from an allow-list on the same response. Open a minimal HTML page served from a different local port that runs fetch() against each, and paste the two different console outcomes (blocked vs allowed) from devtools.

ce-03-04Card 4 of 6

Trace TLS and HTTP/1.1, HTTP/2 and HTTP/3

HardP3 · Role-dependentTalk about it15 min
tlshttp2http3alpnquic

Why you care

"Walk me through what happens when you hit an HTTPS URL" is a depth question that separates people who can name TLS/HTTP-2/HTTP-3 from people who can explain what each layer actually buys you and where it stops helping. On the job, this shows up when a "slow page" or "flaky API call" turns out to be a TLS handshake failure, a certificate mismatch, or a lossy connection that HTTP/2's multiplexing was supposed to fix but doesn't, because the bottleneck moved from the HTTP layer to the TCP layer underneath it. You don't usually write this code, but debugging it without understanding it means guessing.

Explain like I'm 5

Think of it as a phone call. The TLS handshake is the "hello, who is this, let's agree on a secret code" part before either of you says anything real - both sides check who they're talking to and agree on how to encrypt the rest of the call, and they also quietly agree on which language to speak next (HTTP/1.1 or HTTP/2). HTTP/1.1 is talking one sentence at a time on that line and waiting for a reply before saying the next thing - so browsers open up to six separate phone lines just to say several things "at once." HTTP/2 keeps one phone line but you and the other person can interleave several conversations on it, tagging each sentence with which topic it belongs to, so nobody waits for an unrelated topic to finish. HTTP/3 throws away the phone line (TCP) for independent walkie-talkie channels (QUIC/UDP), so a burst of static on channel 3 doesn't freeze channels 1 and 2 the way one crackly phone line would.

Core concept

The TLS handshake, before any HTTP happens: the client sends a ClientHello proposing TLS versions, cipher suites, and via the ALPN extension, the list of application protocols it can speak (h2, http/1.1). The server replies with its chosen cipher, its certificate chain (which the client verifies against a trusted CA and against the hostname it asked for), and picks one entry from the ALPN list - this single choice decides whether the connection speaks HTTP/2 or falls back to HTTP/1.1. Both sides then complete a key agreement, and everything from that point on is encrypted; the negotiated protocol doesn't change again for the life of that connection.

HTTP/1.1 is a text protocol where, without pipelining, one request must complete before the next is sent on the same connection - browsers work around this by opening several TCP connections (commonly up to six) per origin, each with its own handshake. HTTP/2 is a binary, multiplexed protocol: many requests and responses interleave as framed streams over a single TCP connection, identified by stream IDs, removing HTTP-level head-of-line blocking. HTTP/3 keeps the same HTTP semantics (methods, headers, status codes are unchanged) but replaces the transport: it runs over QUIC, itself built on UDP, which folds the TLS 1.3 handshake into the transport handshake and, crucially, gives each stream independent loss detection and retransmission.

The decision rule: application code almost never chooses the HTTP version directly - your load balancer, CDN or reverse proxy negotiates it via ALPN, and a plain Python http.client/requests call speaks HTTP/1.1 regardless of what the server offers, because the standard library never advertises h2 in its ALPN list. What you need to know for debugging is which layer a given symptom points to: a page slow to load many small assets often means an HTTP/1.1 connection limit; a connection that stalls entirely on a lossy network, even over HTTP/2, points at TCP-level packet loss, not your application.

Under the hood

Session resumption is why a returning client doesn't always pay for a full TLS handshake: the server issues a session ticket the client can present next time to skip straight to a shorter handshake (or, with TLS 1.3, even send encrypted application data in the very first flight - "0-RTT" - at the cost of weaker replay protection for that first flight). ALPN is a small extension carried inside the ClientHello itself - the protocol list is visible before encryption even starts, which is exactly what lets a middlebox or load balancer route by protocol before the handshake finishes. From plain Python code with http.client or requests, none of this is visible - the socket layer does it silently - but curl -v or openssl s_client -connect host:443 -alpn h2,http/1.1 show the actual handshake, including which protocol was chosen, which is often the fastest way to confirm whether a "slow API" is a network problem or an application one.

Minimal example HTTP/1.1 persistence - one connection, several requests

PythonOne TCP connection carries three HTTP/1.1 requests when the server allows itruns
import http.server, threading, http.client

class Handler(http.server.BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"          # enables persistent connections
    def log_message(self, *a): pass
    def do_GET(self):
        self.send_response(200); self.send_header("Content-Length", "2"); self.end_headers()
        self.wfile.write(b"ok")

server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()

conn = http.client.HTTPConnection("127.0.0.1", port)
ports_used = set()
for _ in range(3):
    conn.request("GET", "/ping")
    ports_used.add(conn.sock.getsockname())
    r = conn.getresponse(); r.read()
print("requests over one persistent connection, distinct local sockets used:", len(ports_used))
conn.close()

fresh_ports = set()
for _ in range(3):
    c = http.client.HTTPConnection("127.0.0.1", port)
    c.request("GET", "/ping")
    fresh_ports.add(c.sock.getsockname())
    r = c.getresponse(); r.read()
    c.close()
print("requests over a fresh connection each time, distinct local sockets used:", len(fresh_ports))
server.shutdown()
Output
requests over one persistent connection, distinct local sockets used: 1
requests over a fresh connection each time, distinct local sockets used: 3

In production Inspecting a real handshake's negotiated protocol

PythonRead the negotiated TLS version and ALPN protocol from a live handshake (needs real network access)
import socket
import ssl

ctx = ssl.create_default_context()
ctx.set_alpn_protocols(["h2", "http/1.1"])

with socket.create_connection(("example.com", 443), timeout=5) as sock:
    with ctx.wrap_socket(sock, server_hostname="example.com") as tls:
        print("TLS version negotiated:", tls.version())
        print("ALPN protocol chosen:", tls.selected_alpn_protocol())
        print("Peer certificate subject:", dict(x[0] for x in tls.getpeercert()["subject"]))

See it happen One handshake, then three transport stories

clientserverClientHello: TLS versions, ciphers, ALPN=[h2, http/1.1]ServerHello + certificate, ALPN=h2Finished (key agreement complete)HEADERS stream 1 (GET /a) + HEADERS stream 3 (GET /b)DATA stream 1, DATA stream 3 interleavedone TCP segment lost - both streams stallsame two requests, now over HTTP/3 (QUIC/UDP)
  1. The client proposes what it can speak, including which HTTP versions via ALPN, before anything is encrypted.
  2. The server picks a cipher, sends its certificate chain for hostname verification, and selects one ALPN entry - here, h2.
  3. Both sides now share a session key; everything from here is encrypted, and the connection is locked to HTTP/2 for its lifetime.
  4. With h2 negotiated, two requests go out on the same TCP connection, tagged with different stream IDs.
  5. Frames for both streams arrive interleaved on one connection; the client reassembles each by stream ID - this is multiplexing.
  6. A single dropped packet blocks delivery for every stream on this connection until TCP retransmits it; HTTP/2's multiplexing is at the HTTP layer, not the transport layer.
  7. Over HTTP/3, each stream gets independent loss recovery inside QUIC, so a lost packet for stream 1 no longer stalls stream 3.
Notice: ALPN decides the version once, at the handshake; HTTP/2 removes head-of-line blocking at the HTTP layer, but only HTTP/3's transport removes it at the packet-loss layer too.
Scenario A

In your world

The situation

Your nightly job calls an internal partner API over HTTPS. On the day the partner rotates their TLS certificate, the job starts failing with SSLCertVerificationError: hostname 'api.internal' doesn't match certificate, and someone under deadline pressure "fixes" the red Jenkins build by adding verify=False to the requests call.

What actually breaks

The fix works, in the sense that the build turns green - and stays green for three months, through a corporate proxy migration that starts silently intercepting TLS on that network path. Because verification is off, nothing in your job notices or complains; you have no way to prove that any of ninety days of settlement data reaching that endpoint arrived at the real partner unmodified, because the one check that would have caught an interception was the one thing disabled.

What you'd do on Monday

Fix the actual cause of the verification failure - update the trust store or correct the hostname/DNS mismatch - instead of disabling the check:

PythonVerify against a real CA bundle instead of disabling checks (needs network + a valid CA bundle)
import ssl
import urllib.request

ctx = ssl.create_default_context()   # uses the system trust store; never set ctx.check_hostname = False
req = urllib.request.Request("https://api.partner.internal/health")
with urllib.request.urlopen(req, context=ctx, timeout=5) as resp:
    print(resp.status)
Scenario B

Imagine you work at Fastly

200,000 new TLS connections/second at the edge

The numbers

Illustrative: an edge network terminating 200,000 new TLS connections/second globally, with a full TLS 1.3 handshake costing roughly one network round trip plus asymmetric-crypto CPU work - even at a modest ~1ms of crypto per handshake, that's about 200 CPU-seconds of handshake work alone, every second, spread across the fleet. Session resumption (skipping the full handshake for a returning client) is not a nice-to-have at this volume; it's the difference between that cost and a much smaller fraction of it.

What the Scenario-A answer does here

The Scenario A design - one connection per script run, verified properly - is completely fine for a nightly job's single connection. Doing the same thing (open a fresh TLS connection per request, no reuse) at edge scale would mean recomputing a full handshake for every single request instead of amortizing it, which is exactly the naive approach that breaks down once request volume moves from "hundreds a day" to "hundreds of thousands a second."

What flips at this scale

At this scale, connection reuse and protocol negotiation stop being an application concern at all - they move into the edge/proxy layer as infrastructure defaults: session resumption tickets, ALPN choosing HTTP/2 or HTTP/3 per client capability, and QUIC's independent per-stream loss recovery for the connections that support it, none of which your application code decides per request.

The trade-off you'd defend

I would defend terminating TLS and negotiating the HTTP version at a dedicated edge/proxy layer over handling it in application code, trading some debugging indirection (you often can't see the raw handshake from inside your app) for handshake cost amortized across enormous request volume and protocol upgrades that don't require an app change.

The version/ALPN/transport facts are documented behavior (MDN's Evolution of HTTP, retrieved 2026-09-27); the connection-volume and CPU-cost figures are illustrative, not Fastly-published numbers.

HTTP/1.1 vs HTTP/2 vs HTTP/3

AxisHTTP/1.1HTTP/2HTTP/3
TransportTCP, TLS optionalTCP + TLS, version chosen via ALPNQUIC over UDP, TLS 1.3 built into the transport handshake
Requests per connectionOne in flight at a time; browsers open ~6 connections/origin to parallelizeMultiplexed streams on one connectionMultiplexed streams on one QUIC connection
Head-of-line blockingYes, per connectionRemoved at the HTTP layer; still present at the TCP layer on packet lossRemoved at both layers - independent per-stream loss recovery
Handshake costFull TCP + TLS handshake per new connectionSame per-connection cost, amortized over many requests on one connectionOne combined QUIC+TLS handshake; 0-RTT possible on reconnect
Where it's decidedUniversal fallback, always supportedNegotiated via ALPN when both ends support itNeeds UDP allowed through firewalls/proxies; falls back to /2 or /1.1 otherwise

Gotchas

HTTP/2 multiplexing does not remove TCP loss-induced blocking across streams

Symptom

After "upgrading to HTTP/2," a client on a lossy network still sees every in-flight request on a connection stall together.

Cause

HTTP/2 multiplexes at the HTTP layer, but still rides on one TCP connection - TCP's strict in-order delivery means a single lost packet blocks every multiplexed stream until it's retransmitted.

Fix

HTTP/3 over QUIC gives each stream independent loss recovery, which fixes exactly this case; where HTTP/3 isn't available, the old HTTP/1.1-era workaround of a few parallel connections still applies.

A hostname mismatch after certificate rotation

Symptom

SSLCertVerificationError: hostname mismatch appears right after a certificate renewal that previously worked fine.

Cause

The new certificate's subject alternative names no longer include the hostname you're connecting to - often a missed rotation step or a DNS/hostname change on one side only.

Fix

Verify the certificate's SANs against the exact hostname you connect to; fix the certificate or the DNS entry, never the verification setting.

Disabling certificate verification "to fix the error"

Symptom

Setting verify=False (or curl -k) makes the error go away, and nobody circles back to it.

Cause

A verification failure is almost always surfacing a real infrastructure problem - an expired cert, a wrong hostname, or an intercepting proxy - not a library bug to be silenced.

Fix

Diagnose and fix the underlying cert/hostname/trust-store issue; alert on certificate expiry so this is caught before it becomes a production failure.

Tricky questions

Does HTTP/2 remove head-of-line blocking entirely?
Most people say

Yes - multiplexing is the whole point of HTTP/2, so no request has to wait for another anymore.

Actually

It removes HTTP-level head-of-line blocking - one slow response no longer blocks unrelated requests at the application layer. But because HTTP/2 still runs over a single TCP connection, a single lost packet blocks every multiplexed stream at the transport layer, because TCP guarantees one ordered byte stream. Only HTTP/3 over QUIC gives each stream its own independent loss recovery.

If ALPN picks h2 during the handshake, does my Python requests call automatically get HTTP/2 benefits?
Most people say

Yes - ALPN negotiated it, so the connection speaks HTTP/2 from then on.

Actually

ALPN only negotiates what the TLS layer is offered. The standard library's http.client (and requests on top of it) never advertises h2 in its ALPN list, so it always negotiates and speaks http/1.1, regardless of what the server supports. Getting real HTTP/2 needs an HTTP/2-capable client, or letting a CDN/proxy handle the upgrade.

Scenario question

You're handed a report that api.partner.com became slower after "upgrading to HTTP/2," but only for users on a flaky mobile network - broadband users actually got faster. What's likely happening, and what would you check?

Model answer

I would suspect TCP-level head-of-line blocking, not an HTTP/2 regression: multiplexing puts many requests on one TCP connection, so on a lossy mobile link, any single lost packet stalls every in-flight stream until it's retransmitted, which broadband connections rarely hit. I would check retransmit/loss statistics for those sessions, and consider whether the edge supports HTTP/3/QUIC, since that isolates loss per stream instead of per connection - for a lossy-network audience, that's the actual fix, not tuning HTTP/2 settings. The trade-off is that QUIC needs UDP to be allowed through the client's network path, and some corporate or carrier firewalls block it, so HTTP/2 or HTTP/1.1 still has to work as a fallback.

Interview corner

CommonWhat happens, step by step, when you hit an HTTPS URL?
“I'd walk through DNS resolution, a TCP handshake, then a TLS handshake where the client and server agree on a cipher, the server proves its identity with a certificate, and ALPN picks the HTTP version - then the actual HTTP request and response happen inside that encrypted connection.”
TrickDoes upgrading to HTTP/2 fix head-of-line blocking for a lossy connection?
“I'd say only at the HTTP layer. It removes the old problem of one slow response blocking unrelated ones on the same connection, but it still rides on a single TCP connection, so packet loss stalls every stream until TCP retransmits. HTTP/3 over QUIC is what actually isolates loss per stream.”
Experience checkHave you had to debug a TLS or certificate issue in a pipeline? What did you check first?
“Yes - I start with the exact error text, since a hostname mismatch, an expired cert, and an untrusted CA all look similar at a glance but need different fixes. I use curl -v or openssl s_client to see the actual certificate chain and negotiated protocol rather than guessing from the application-level stack trace.”
DesignWhy might a service move from HTTP/2 to HTTP/3, and what's the cost?
“I'd move for exactly the loss-isolation benefit on unreliable networks, plus faster reconnection via QUIC's combined handshake. The cost is that HTTP/3 needs UDP to reach the client, and some corporate networks or older proxies block or don't support it, so I'd keep HTTP/2 as a fallback rather than assuming universal support.”

Memory hook

ALPN is the one line in the handshake that decides which HTTP you're actually speaking for the rest of the connection.

If you remember only one thing:

HTTP/2 fixes app-level head-of-line blocking; only HTTP/3's transport fixes packet-loss blocking.

Self-check

  1. What does ALPN do, and at what point in the connection does it happen?

    It lets the client and server agree on which application protocol (h2, http/1.1) to speak, negotiated inside the TLS handshake, before any HTTP request is sent.

  2. Why does a single lost packet stall every stream on an HTTP/2 connection?

    HTTP/2 multiplexes at the HTTP layer but still runs over one TCP connection, and TCP enforces strict in-order delivery of the whole byte stream.

  3. Does a plain Python http.client/requests call ever actually speak HTTP/2?

    No - the standard library's HTTP client never advertises h2 in its ALPN list, so it always negotiates and speaks http/1.1, whatever the server supports.

Prove it

Run curl -v against any real HTTPS site you have access to, and separately openssl s_client -connect <host>:443 -alpn h2,http/1.1. Identify and paste the lines showing the negotiated TLS version, the chosen ALPN protocol, and the certificate's subject - then state in one sentence whether the connection ended up on HTTP/1.1 or HTTP/2 and how you can tell.

ce-03-05Card 5 of 6

Receive authenticated, replay-aware webhooks

MediumP2 · Very likelyKnow cold14 min
webhookshmacreplayidempotency

Why you care

A webhook receiver is an HTTP endpoint that accepts requests from the public internet and, if you're not careful, trusts whatever arrives. Interviewers probe this because it tests three things at once: can you verify a sender is who they claim to be, can you handle a request that arrives more than once without double-applying it, and do you know that "acknowledge" and "process" are different moments. In real integration work - payment notifications, order events, partner callbacks - getting any one of these wrong produces either a security hole or a duplicate-processing incident, both of which show up in production, not in review.

Explain like I'm 5

A webhook is a stranger knocking on your door with a parcel, claiming to be from a courier company you trust. You don't just take their word for it: you check a wax seal on the parcel that only the real courier's stamp could have made (the signature, computed with a shared secret), stamped with today's date so an old seal can't be reused on a different day (the timestamp, guarding against replay). You say "got it" quickly at the door before you've even opened the box, because the courier is standing there waiting and has other deliveries - then you actually unpack and deal with the contents afterward. And if the same parcel shows up twice because the courier's tracking said "delivery unconfirmed," you check the tracking number against what you've already logged before treating it as new.

Core concept

The mechanics below match the model documented for Stripe webhooks, one of the most widely deployed real implementations of this pattern: a Stripe-Signature header carries a timestamp and one or more scheme-prefixed signatures, like t=1492774577,v1=5257a869.... The receiver reconstructs the exact signed payload as "{timestamp}.{raw_body}", computes an HMAC-SHA256 over it using the shared secret, and compares the result to the received signature with a constant-time comparison (to avoid leaking timing information about how close a guess was). Two checks matter, and both are mandatory, not optional extras:

  • Signature verification proves the request was signed by someone who holds your shared secret - it must run against the exact raw bytes of the body. Any middleware that parses and re-serializes JSON before your verification code sees it will produce different bytes than were signed, and verification will fail for a reason that has nothing to do with the sender.
  • Timestamp / replay window rejects an otherwise-valid signature if it's too old - the timestamp is itself part of what's signed, so an attacker can't just replay an old, genuinely-signed request forever. A default tolerance around five minutes is typical. Skipping the check altogether - not the value you pass it - is the real replay-protection gap, since that leaves you accepting requests that could be minutes, hours or days old. Watch for a related bug in some implementations: gating the check with if tolerance: instead of comparing unconditionally treats a deliberate tolerance=0 ("no clock drift allowed") as falsy and skips the check by accident. The verify() below compares unconditionally, so passing it 0 makes it stricter - rejecting almost everything - not disabled.

Beyond verification, two delivery realities you must design for: senders expect a fast acknowledgment (a 2xx, before any slow processing) and will retry on a timeout or non-2xx response, sometimes for days, with no guarantee of ordering and no guarantee of exactly-once delivery. The decision rule follows directly: verify first, acknowledge fast, and deduplicate by the sender's unique event id before applying any side effect - treat the sender's own id as the natural key, the same way you would for any other retried write (Apply REST semantics without promising exactly once), and validate the body's contents only after you trust who sent it (Application threat boundaries: SQLi, SSRF and IDOR).

Minimal example Verify, and reject a replay

PythonVerify a webhook signature, Stripe's t=..,v1=.. schemeruns
import hashlib
import hmac
import time


SECRET = b"whsec_test_shared_secret"


def sign(payload: bytes, timestamp: int) -> str:
    signed_payload = f"{timestamp}.".encode() + payload
    digest = hmac.new(SECRET, signed_payload, hashlib.sha256).hexdigest()
    return f"t={timestamp},v1={digest}"


def verify(payload: bytes, header: str, tolerance_seconds: int = 300) -> bool:
    parts = dict(kv.split("=", 1) for kv in header.split(","))
    timestamp = int(parts["t"])
    if abs(time.time() - timestamp) > tolerance_seconds:
        return False
    expected = hmac.new(SECRET, f"{timestamp}.".encode() + payload, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, parts["v1"])


body = b'{"type":"payment_intent.succeeded","id":"evt_1"}'
now = int(time.time())

print("genuine signature:", verify(body, sign(body, now)))
print("body edited after signing:", verify(body + b"!", sign(body, now)))
print("valid signature, but replayed an hour later:", verify(body, sign(body, now - 3600)))
Output
genuine signature: True
body edited after signing: False
valid signature, but replayed an hour later: False

In production A receiver that verifies, acks fast, and dedupes by event id

PythonSame event delivered twice is applied once; a forged signature never gets inruns
import hashlib, hmac, http.server, http.client, json, threading, time

SECRET = b"whsec_test_shared_secret"
processed = set()          # event ids already applied
ledger = []                 # what we actually recorded, in processing order

def sign(payload, timestamp):
    digest = hmac.new(SECRET, f"{timestamp}.".encode() + payload, hashlib.sha256).hexdigest()
    return f"t={timestamp},v1={digest}"

def verify(payload, header, tolerance=300):
    parts = dict(kv.split("=", 1) for kv in header.split(","))
    ts = int(parts["t"])
    if abs(time.time() - ts) > tolerance:
        return False
    expected = hmac.new(SECRET, f"{ts}.".encode() + payload, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, parts["v1"])


class Handler(http.server.BaseHTTPRequestHandler):
    def log_message(self, *a): pass
    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        raw = self.rfile.read(length)                 # the exact signed bytes
        sig = self.headers.get("X-Signature", "")
        if not verify(raw, sig):
            self.send_response(400); self.end_headers(); return

        event = json.loads(raw)
        self.send_response(200); self.end_headers()    # ack immediately

        if event["id"] in processed:
            return                                       # duplicate delivery, already handled
        processed.add(event["id"])
        ledger.append(event["id"])


server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()
conn = http.client.HTTPConnection("127.0.0.1", port)

events = [
    {"id": "evt_1", "type": "order.paid"},
    {"id": "evt_2", "type": "order.shipped"},
    {"id": "evt_1", "type": "order.paid"},        # retried delivery of evt_1
]
for e in events:
    body = json.dumps(e).encode()
    header = sign(body, int(time.time()))
    conn.request("POST", "/webhooks/orders", body=body, headers={"X-Signature": header})
    r = conn.getresponse(); print(e["id"], "->", r.status); r.read()

conn.request("POST", "/webhooks/orders", body=b'{"id":"evt_3"}', headers={"X-Signature": "t=1,v1=deadbeef"})
r = conn.getresponse(); print("forged signature ->", r.status); r.read()

print("processed exactly once each:", ledger)
server.shutdown()
Output
evt_1 -> 200
evt_2 -> 200
evt_1 -> 200
forged signature -> 400
processed exactly once each: ['evt_1', 'evt_2']

See it happen Verify, ack, then dedupe before applying

senderreceiverworkerPOST + X-Signature: t=.., v1=..recompute HMAC over the same raw bytes200 OKenqueue event by idsame event, redeliveredid already in processed set - skip
  1. The sender signs the timestamp plus the raw body with the shared secret and sends it as an ordinary HTTP POST.
  2. The receiver rejects the request if the signature doesn't match or the timestamp is outside the tolerance window.
  3. The receiver acknowledges immediately, before doing any slow work, because the sender is waiting with a short deadline.
  4. The actual processing is handed off to a worker keyed by the event's unique id, not by arrival order.
  5. A network hiccup on the sender's side means the same event can be redelivered with nothing but the attempt count changed.
  6. Because processing is keyed by event id, the redelivered copy is deduped before it ever reaches the worker a second time.
Notice: the signature check happens before anything else runs, and the id check happens before anything is applied twice - acknowledging fast and deduplicating are two separate jobs, not one.
Scenario A

In your world

The situation

Your team's webhook receiver ingests settlement notifications from a payment partner straight into Postgres. It trusts the JSON body without checking any signature "because it's over HTTPS, that's secure enough," and the handler takes about 45 seconds to insert the row and recompute a summary report before it responds.

What actually breaks

During a slow deploy window, 40 real settlement events arrive. The partner's client-side timeout is 20 seconds - far shorter than your 45-second handler - so every one of those 40 events gets retried twice before your first response ever lands. With no signature check and no dedupe, all 120 deliveries get inserted as new rows. The audit table shows 120 settlements for 40 real events, and finance flags a doubled total before anyone notices the actual cause.

What you'd do on Monday

Verify the signature before parsing anything, acknowledge immediately, and let a database constraint - not application memory - make the duplicate a no-op:

PythonAck fast, verify first, dedupe with a DB constraint (needs psycopg + a real connection)
if not verify(raw_body, request.headers.get("X-Signature", "")):
    return Response(status=400)

respond_now(200)                      # ack before doing anything slow

cur.execute(
    "INSERT INTO webhook_events (event_id, payload) VALUES (%s, %s) "
    "ON CONFLICT (event_id) DO NOTHING RETURNING event_id",
    (event["id"], raw_body),
)
if cur.fetchone() is None:
    return                             # duplicate delivery, already recorded
enqueue_for_processing(event)
Scenario B

Imagine you work at Stripe

50,000 webhook events/day, 20x bursts at billing cycles

The numbers

Illustrative: an integration receiving 50,000 webhook events/day averages about 0.6 events/second, but subscription-renewal bursts at the start of a billing cycle can spike to roughly 20x that - about 12 events/second for a few minutes. If your handler takes 2 seconds to respond during that burst instead of well under 1 second, and the sender's delivery attempt times out, you can see the same batch of events redelivered two or three times within minutes, not as a rare event but as a routine feature of every billing cycle.

What the Scenario-A answer does here

The Scenario A fix - verify, ack fast, dedupe by event id - is exactly the model documented for real webhook senders, and it still holds. What breaks at this scale is treating duplicates as rare enough to skip the dedupe step: a documented "at least once, no ordering guarantee" delivery model means duplicates during a burst aren't an edge case, they're expected traffic.

What flips at this scale

At scale, the dedupe set itself has to move out of one process's memory - an in-memory set() like the minimal example forgets everything on restart and never sees what a second, horizontally-scaled replica has processed. The flip is a durable, shared idempotency store every replica checks against: a unique database constraint on event id, or a keyed cache entry with a TTL, not a per-process collection.

The trade-off you'd defend

I would defend a database-level unique constraint on event id over an in-memory dedupe set, trading one extra index for correctness across restarts and multiple replicas - an in-memory set forgets every duplicate the instant the process restarts, which is exactly when a burst is most likely to be retried.

The signature scheme, timestamp tolerance, retry behavior and no-ordering-guarantee are documented for Stripe webhooks (docs.stripe.com/webhooks, retrieved 2026-09-27); the event volumes and burst multipliers are illustrative, not Stripe-published traffic figures.

Gotchas

Signature verification fails only in production

Symptom

The exact same verification code works in local tests but always rejects real requests once deployed.

Cause

A framework or proxy middleware parsed and re-serialized the JSON body before your handler saw it, so the bytes being verified are not the exact bytes that were signed.

Fix

Capture and verify against the raw request body, before any JSON parsing or body-rewriting middleware touches it.

Duplicate rows despite "checking for duplicates"

Symptom

The same event gets applied more than once even though the code has a dedupe check.

Cause

The dedupe set lives in one process's memory; a second replica, or a restart, has no memory of what was already processed.

Fix

Dedupe against a store shared by every replica and durable across restarts - a unique constraint or a keyed cache with a TTL, not an in-memory collection.

Retries pile up or the sender stops delivering

Symptom

The same event arrives repeatedly, or the sender eventually marks the endpoint as failing.

Cause

The handler does the real, slow work before acknowledging, missing the sender's short response deadline.

Fix

Verify and persist the raw event fast, acknowledge with a 2xx, then do the slow work asynchronously afterward.

"It's HTTPS, that's secure enough"

Symptom

A receiver applies whatever arrives at its public endpoint without checking who actually sent it.

Cause

HTTPS proves you're talking to the holder of a certificate for that hostname over an encrypted channel - it says nothing about whether the specific sender inside the payload is who they claim to be.

Fix

Verify the application-level signature regardless of transport security; HTTPS and signature verification solve different problems.

Tricky questions

If the connection is HTTPS, do I still need to verify the webhook signature?
Most people say

No - HTTPS already guarantees the request is authentic.

Actually

HTTPS guarantees you're talking to whoever holds a certificate for that hostname, over an encrypted channel - anyone who knows your public endpoint URL can send it an HTTPS POST. The signature is what proves the specific sender holds your shared secret, which HTTPS alone never checks.

Does returning 200 quickly mean I've finished processing the event?
Most people say

Yes - a 200 means the operation is done.

Actually

Here, 200 means "received and verified," matching the accept-versus-complete distinction from HTTP wire contract: messages and observable outcomes. The actual processing can, and for slow work should, happen after the response is already sent.

Scenario question

You're handed a webhook receiver that verifies signatures correctly but has no dedupe logic, and product wants to know if it's safe to ship as-is given the sender documents "at least once, no ordering guarantee" delivery.

Model answer

I would say no, not without dedupe - "at least once" means every event that drives a side effect, like crediting an account, can and will arrive more than once, and a correct signature check only proves who sent it, not that it's new. I would add a durable idempotency check keyed by the sender's event id - a unique database constraint is enough at moderate volume - before applying any side effect, and make the side effect itself idempotent where possible, such as an upsert on the sender's natural key, as a second line of defense. The trade-off is one more table and index and a small amount of extra latency per event, which is cheap compared to double-crediting an account during the next retry burst.

Interview corner

CommonHow do you verify a webhook is really from the sender and not forged?
“I'd compute an HMAC over the exact raw request body plus a timestamp, using a secret shared only with the sender, and compare it to the signature they send in a header with a constant-time comparison. I'd reject anything where the signature doesn't match or the timestamp is outside a tolerance window, to also block replay.”
TrickThe connection is HTTPS - is that enough to trust the payload?
“No. HTTPS only proves I'm talking to whoever holds a certificate for that hostname, over an encrypted channel - anyone who finds my endpoint URL can send it an HTTPS request. I still need a signature check to prove the specific sender holds my shared secret.”
Experience checkHave you built or fixed a webhook receiver that double-processed events? What was the fix?
“Yes - a receiver took too long to respond, missed the sender's timeout, and got retried into duplicate database rows. I fixed it by acknowledging immediately after verifying the signature and doing the real insert with a unique constraint on the event id, so a retried delivery just hits ON CONFLICT DO NOTHING.”
DesignHow would you dedupe webhook events across multiple horizontally scaled receiver instances?
“I'd move the dedupe check out of process memory into something every instance shares - a unique constraint in the database or a keyed entry in a cache with a TTL - because an in-memory set on one replica has no idea what another replica, or a previous instance before a restart, already processed.”

Memory hook

A webhook is a stranger at the door claiming to be the courier - check the seal before you take the parcel, and note the parcel's ID before you unpack it twice.

If you remember only one thing:

Verify the signature over the exact raw bytes, ack fast, and dedupe by event id - skipping any one of the three is a real incident.

Self-check

  1. What two things does a webhook signature scheme like Stripe's protect against, together?

    Forged senders (the HMAC proves who holds the shared secret) and replay of an old, genuinely-signed request (the timestamp tolerance rejects anything too old).

  2. Why must signature verification run against the raw request body?

    Any middleware that parses and re-serializes the body before verification changes the exact bytes, so a correct signature will no longer match what's computed.

  3. Why does an in-memory dedupe set fail once you scale to multiple receiver replicas?

    Each replica (and each restart) has its own memory; a set in one process has no knowledge of events another replica or a previous run already processed.

Prove it

Extend the production example so processed event ids are stored in a SQLite table with a UNIQUE constraint instead of an in-memory set. Simulate a process restart by recreating the in-memory processed set between two batches, and show that the in-memory version now accepts a duplicate while the database constraint still rejects it. Paste both outcomes.

ce-03-06Card 6 of 6

Debug a request with curl and timing evidence

EasyP2 · Very likelyKnow cold12 min
curldebugginglatencytooling

Why you care

"The API is slow" or "the API is down" tickets arrive with no other detail more often than they should, and the fastest way to turn that into an actual diagnosis is a handful of curl flags most people never use past -X POST. Interviewers ask about this because it's a direct test of whether you can isolate a problem - is it DNS, the network, TLS, or the application itself - before writing a single line of application code to "investigate." In day-to-day ETL and integration work, this is the difference between spending twenty seconds with curl and spending an afternoon adding print statements to a script that was never the problem.

Explain like I'm 5

Think of a relay race with a stopwatch that only shows the finish time - all you know is "the race took 40 seconds," with no idea which runner was slow. curl -w gives you lap splits instead: how long DNS lookup took, how long the TCP handshake took, how long the TLS handshake took, how long until the first byte of the response arrived, and the total. Now instead of "the API is slow," you can say exactly which leg of the race - DNS, network, TLS, or the server itself thinking about your request - is the one costing you time.

Core concept

-i/--include prints the response headers along with the body. -I/--head sends a HEAD request and shows only headers, no body - useful for a quick check, but only as safe as the server's own HEAD handling, since the HTTP spec's promise that HEAD is side-effect-free is a contract the server has to honor, not something curl can guarantee for you (Apply REST semantics without promising exactly once). -v/--verbose shows the full exchange, including the TLS negotiation and the exact headers sent, prefixed with > for what curl sent and < for what came back - and separately, lines starting with * describing connection-level events (DNS, connect, TLS), which is how you tell a transport failure from an application-level one. -X/--request overrides the method; -H/--header adds (or, given a bare Name:, removes) a header.

-w/--write-out is the timing tool: it prints named variables after the transfer completes. The key chain, each cumulative from the start of the request, is time_namelookup (DNS resolution), time_connect (TCP handshake complete), time_appconnect (TLS handshake complete - present only for HTTPS), time_starttransfer (time to first response byte, i.e. TTFB), and time_total. Because each is cumulative, the actual duration of a given step is the difference between it and the previous one: time_appconnect - time_connect is TLS handshake time alone; time_starttransfer - time_appconnect is how long the server itself took to start responding, once the connection was fully ready.

The decision rule: use --connect-timeout to bound DNS+TCP+TLS setup specifically, and --max-time to bound the whole request, so a script calling curl (or your own code doing the equivalent) doesn't hang forever on a server that accepted the connection but never replies. Before sharing curl output in a ticket or a chat thread, strip Authorization/Cookie/API-key values from what you paste - -v/-H will happily echo back exactly what you sent, secrets included - and never leave -k/--insecure in anything permanent, since disabling certificate verification silently reopens the exact risk TLS exists to close (Trace TLS and HTTP/1.1, HTTP/2 and HTTP/3).

Minimal example Time the pieces yourself, the way curl -w does

PythonSplitting one stopwatch into DNS/connect/TLS/TTFB/total-style lapsruns
import http.client
import http.server
import socket
import threading
import time


class Handler(http.server.BaseHTTPRequestHandler):
    def log_message(self, *a): pass
    def do_GET(self):
        time.sleep(0.05)                 # stand-in for "server think time"
        self.send_response(200)
        self.send_header("Content-Length", "2")
        self.end_headers()
        self.wfile.write(b"ok")


server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()

t_start = time.perf_counter()
probe = socket.create_connection(("127.0.0.1", port), timeout=2)
t_connect = time.perf_counter()
probe.close()

conn = http.client.HTTPConnection("127.0.0.1", port, timeout=2)
conn.request("GET", "/ping")
r = conn.getresponse()                    # status line + headers have arrived
t_first_byte = time.perf_counter()
r.read()
t_total = time.perf_counter()
conn.close()
server.shutdown()

print("status:", r.status)
print("time_connect  <  time_starttransfer:", (t_connect - t_start) < (t_first_byte - t_start))
print("time_starttransfer  <=  time_total:", (t_first_byte - t_start) <= (t_total - t_start))
print("server think time (starttransfer - connect) at least 0.04s:", (t_first_byte - t_connect) >= 0.04)
Output
status: 200
time_connect  <  time_starttransfer: True
time_starttransfer  <=  time_total: True
server think time (starttransfer - connect) at least 0.04s: True

In production A bounded probe that redacts secrets before printing

PythonBound the wait, and never print a live credentialruns
import http.client
import http.server
import threading
import time


class Handler(http.server.BaseHTTPRequestHandler):
    def log_message(self, *a): pass

    def do_GET(self):
        if self.path == "/slow":
            time.sleep(0.15)
        if self.path == "/status":
            self.send_response(503)
            self.send_header("Retry-After", "30")
            self.end_headers()
            return
        self.send_response(200)
        self.send_header("Content-Length", "2")
        self.end_headers()
        self.wfile.write(b"ok")


def probe(port, path, headers=None, connect_timeout=0.5, max_time=0.2):
    redacted = {k: ("***" if k.lower() == "authorization" else v) for k, v in (headers or {}).items()}
    t0 = time.perf_counter()
    try:
        conn = http.client.HTTPConnection("127.0.0.1", port, timeout=connect_timeout)
        conn.request("GET", path, headers=headers or {})
        r = conn.getresponse()
        elapsed = time.perf_counter() - t0
        if elapsed > max_time:
            conn.close()
            return {"error": "max_time exceeded", "sent_headers": redacted}
        r.read()
        return {"status": r.status, "retry_after": r.getheader("Retry-After"), "sent_headers": redacted}
    except (TimeoutError, OSError) as e:
        return {"error": f"{type(e).__name__}: {e}", "sent_headers": redacted}


server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()

print(probe(port, "/ping", headers={"Authorization": "Bearer secret-token"}))
print(probe(port, "/status"))
print(probe(port, "/slow", max_time=0.05))
server.shutdown()
Output
{'status': 200, 'retry_after': None, 'sent_headers': {'Authorization': '***'}}
{'status': 503, 'retry_after': '30', 'sent_headers': {}}
{'error': 'max_time exceeded', 'sent_headers': {}}
Scenario A

In your world

The situation

A partner API your nightly job calls "hangs" intermittently. Someone's fix, under time pressure, is to keep raising the requests timeout - 30s, then 120s, then 300s - each time it happens again.

What actually breaks

During a backlog catch-up run of 600 calls, a handful of them each take the full 300s before eventually failing, and the whole run blows past its window by hours - blamed on "the network being flaky." Nobody has actually confirmed whether it's DNS, connection setup, TLS, or the partner's application code that's slow, so every fix has been a guess.

What you'd do on Monday

Get a timing breakdown on one of the slow calls before touching the timeout again:

Shell
curl -sS -o /dev/null -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total} status=%{http_code}\n' \
  --connect-timeout 3 --max-time 10 \
  -H "Authorization: Bearer ***REDACTED***" \
  https://api.partner.internal/orders/88214
Scenario B

Imagine you work at Swiggy

600 backlog calls, one slow leg to isolate

The numbers

Illustrative: a backlog catch-up run of 600 calls, where roughly 2% (about 12 calls) hit the slow path described above. At an average of 250ms of extra delay attributable to just one leg - say, the TLS handshake, discovered via time_appconnect - time_connect - that's about 3 seconds of avoidable extra wait across the run's 12 affected calls; the real cost isn't those 3 seconds, it's the hours lost to raising timeouts blindly instead of measuring which 12 calls, and which leg, were actually slow.

What the Scenario-A answer does here

The Scenario A fix (measure the exact leg with -w before changing anything) still applies to a single ad hoc check, but running one curl probe manually for every incident doesn't scale to hundreds of jobs - the naive version is "someone remembers to run curl when it breaks," which depends on a person noticing and being available.

What flips at this scale

At this scale, the same timing breakdown needs to be automatic and structured: your job's own HTTP client should log connect time, TLS time and TTFB per call (or sample a percentage of calls) into the same observability stack used for everything else, so "which leg is slow" is a query against existing data, not a manual curl session after the fact.

The trade-off you'd defend

I would defend adding lightweight per-call timing instrumentation to the client over relying on ad hoc curl debugging after each incident, trading a small amount of added code and log volume for being able to answer "which leg, how often" from history instead of only from whichever incident someone happened to catch live.

Illustrative call counts and delay figures chosen to make the arithmetic concrete; not Swiggy-published data. The curl flag behavior itself follows the documented curl manual (curl.se/docs/manpage.html, retrieved 2026-09-27).

Gotchas

Pasted curl output leaks a live credential

Symptom

A Slack thread or ticket contains a working bearer token or session cookie, visible to anyone who can read the thread.

Cause

-v and -H echo back exactly what was sent, including Authorization and Cookie values, with no redaction of their own.

Fix

Redact secrets before sharing output - reproduce with a placeholder value, or manually strip the real one from anything pasted.

-k becomes permanent

Symptom

A debugging session adds curl -k to get past a certificate error, and it quietly stays in a saved script or runbook afterward.

Cause

-k/--insecure disables certificate verification entirely, hiding the same MITM risk covered in Trace TLS and HTTP/1.1, HTTP/2 and HTTP/3.

Fix

Fix the underlying certificate, hostname or trust-store issue instead of keeping -k in anything that isn't a one-off local check.

A probe with no timeout hangs indefinitely

Symptom

A script calling out to a partner endpoint occasionally hangs forever, with no error and no result.

Cause

Neither --connect-timeout nor --max-time was set, so curl (and by extension the calling script) waits as long as the server allows the connection to sit open.

Fix

Always set both a connect timeout and a total timeout in anything run as part of automation.

Tricky questions

Is -I (HEAD) always a safe way to check any endpoint without side effects?
Most people say

Yes - the HTTP spec says HEAD has no side effects, so it's always safe to use for a quick check.

Actually

The spec says a HEAD response should mirror a GET's headers with no body, but that's a promise the server implementation has to keep - a misbehaving or legacy endpoint can still run non-idempotent code on HEAD. Treat it as usually safe, and verify on an unfamiliar internal API before scripting it.

Does curl -v output mean the request definitely reached the server?
Most people say

Yes - if curl shows verbose output, the request went through.

Actually

-v shows connection attempts and TLS negotiation as well as the completed exchange, and it distinguishes them - a * Recv failure or a stalled * SSL_connect line means the request never reached the application at all. Read the * connection-level lines, not just the >/< request/response lines, to tell a transport failure from an application-level one.

Scenario question

You're handed a ticket - "the partner API call is slow, please investigate" - with no other detail.

Model answer

I would run curl with -i and -w against the same URL to get the status plus a DNS/connect/TLS/TTFB/total timing breakdown, redacting any credentials before I share the output. A high time_namelookup points at DNS; a large gap before time_connect points at network/routing; a large time_appconnect - time_connect points at the TLS handshake; and a large time_starttransfer - time_appconnect means the server itself is slow to start responding, which is an application problem, not a network one. I would bound my own probe with --connect-timeout/--max-time so it can't hang, and repeat it two or three times before concluding anything, since one slow sample could be a fluke. The trade-off is that a handful of manual probes only sample the problem - if it's intermittent, I would also pull logs or traces for the affected window instead of relying purely on ad hoc checks.

Interview corner

CommonHow do you use curl to tell whether a slow API call is a network or an application problem?
“I'd use curl -w with the timing variables - namelookup, connect, appconnect, starttransfer, total - and look at the gap between consecutive ones. A big gap before appconnect is network or TLS; a big gap between appconnect and starttransfer is the server itself being slow.”
TrickDoes curl -I ever have side effects on the server?
“It shouldn't, per the HTTP spec's promise for HEAD, but that promise is only as good as the server's implementation. I treat -I as usually safe, and I'd confirm on an unfamiliar internal API before relying on it in a script that runs unattended.”
CommonWhat's the difference between curl -i and curl -I?
“-i includes the response headers along with the body of a normal request. -I sends a HEAD request instead and shows only headers, with no body at all.”
Experience checkWalk me through a time you diagnosed a 'the API is down' ticket that turned out to be something else.
“I've had a 'down' report turn out to be a single slow endpoint with a normal TTFB everywhere else - a quick curl -w against a few different paths showed one specific route was the outlier, which redirected the investigation to that handler instead of the network or infrastructure.”

Memory hook

curl -w splits one stopwatch into laps - DNS, connect, TLS, first byte, total - read the laps, not just the finish time.

If you remember only one thing:

Time each leg separately before you blame "the network."

Self-check

  1. What does curl -I send, and what's the risk of using it on an unfamiliar endpoint?

    It sends a HEAD request; the risk is that some servers still run side effects on HEAD despite the spec saying they shouldn't.

  2. Which write-out variable pair isolates TLS handshake time specifically?

    time_appconnect minus time_connect.

  3. Why should you never leave -k/--insecure in a saved script or runbook?

    It disables certificate verification entirely, silently reopening the man-in-the-middle risk that TLS exists to close.

Prove it

Pick any HTTPS endpoint you have access to (or your own local server), run curl -i -v and separately curl -w with the five timing variables, and paste both outputs with any credentials or tokens redacted. Then introduce an artificial 100ms delay in a small local endpoint and show which single write-out variable moves as a result.