Shelf · AWS · Chapter 3

Chapter 3: Compute & the decision framework

Every compute choice you make on AWS maps to one of five services, and knowing which fits which workload is the difference between a job that costs $5 a month and one that requires a pager rota. By the end of this chapter you can explain the Lambda-versus-Fargate decision, operate EC2 Auto Scaling Groups, package Lambda functions, run ECS tasks on Fargate, schedule retryable Batch jobs, and know when Kubernetes adds cost without adding value.

Difficulty
Priority
aws-03-01Card 1 of 6

AWS compute choice

MediumP2 · Very likelyTalk about it14 min
computelambdafargateec2batcheksdecision-framework

Why you care

When a recruiter for a product company says "walk me through your compute choices on AWS", your answer is the first filter. The question is not about reciting service names; it is about showing you can map workload properties — duration, traffic shape, state, operational budget — to the right abstraction. You are currently deploying Python ETL scripts on Jenkins that write to S3 and Postgres. That workload sits exactly in the Lambda-versus-Batch decision space, and every interviewer at a product company will probe it. Interviewers ask "Lambda or Fargate?" because the answer tells them whether you understand operational cost, not just AWS pricing.

Explain like I'm 5

Imagine you need staff to answer phone calls. Lambda is a freelancer you ring when needed — they pick up, do the job, hang up, and you pay per call. Fargate is a staffing agency running a small team of contractors: always available during their shift, you decide what they do but not what workstations they use. EC2 is hiring full-time employees: you own everything, from the desk to the payslip. Batch is a factory shift: you queue work orders, the factory opens when there are enough of them, and workers go home when the queue drains. Kubernetes is buying and running your own factory building — only worth it if you already know factory management.

Core concept

Five services, five operating models.

AWS Lambda executes event-driven functions. Duration limit: 900 seconds (15 minutes) per invocation — a hard ceiling, check the current quota page for any account-specific detail. Memory: 128 MB to 10,240 MB. No persistent local disk beyond /tmp (512 MB to 10,240 MB). You pay per GB-second. Use Lambda when: tasks are short, triggered by events (S3, SQS, API Gateway, EventBridge), and you want zero server management. Do not use Lambda when: duration exceeds 15 minutes, the workload needs a GPU or a specific kernel build, or a cold start would break your latency contract.

AWS Fargate runs containers via ECS or EKS without you provisioning EC2 instances. Tasks can run indefinitely. You pay per vCPU-second while a task is running. Use Fargate when: the workload is containerised, duration can exceed 15 minutes, or you need a runtime Lambda does not support. Do not use Fargate when: you need EC2-level hardware (GPU, local NVMe, bare metal) or want maximum compute density.

Amazon EC2 is a full virtual machine. You manage patching, scaling, capacity planning and OS. Use EC2 when: you need specific hardware families, run jobs longer than a Fargate task would tolerate, need kernel customisation, or want to control cost tightly with Reserved Instances or Spot. Do not use EC2 when: you can get the same result from Lambda or Fargate; the operational overhead is real.

AWS Batch is a managed job scheduler. You define a compute environment, a job queue, and job definitions; Batch provisions compute as jobs arrive and scales to zero when the queue drains. Use Batch when: workloads are parallelisable, retryable, and you want job dependencies and retry logic handled without writing your own orchestration.

Amazon EKS runs Kubernetes on AWS. Use EKS when: you already operate a Kubernetes platform and need its ecosystem. Do not use EKS when: the team has no Kubernetes expertise — ECS gives you 90 % of the benefit with 10 % of the operational overhead.

Decision rule: event-driven under 15 minutes → Lambda. Containerised, longer-running, or needs a specific runtime → Fargate. Specific hardware, long-lived steady-state, maximum cost control → EC2. Parallelisable batch ETL with retries → Batch. Already on Kubernetes → EKS, otherwise skip it.

Minimal example Four workloads, one decision function

Pythonruns
def pick_compute(duration_s, stateful, traffic, packaging):
    if duration_s > 900:
        return "Batch" if traffic == "batch" else "EC2/Fargate"
    if stateful:
        return "EC2/Fargate+EFS"
    if packaging == "container":
        return "Fargate"
    if traffic == "batch":
        return "Lambda/Batch"
    return "Lambda"

print(pick_compute(10,   False, "bursty", "zip"))       # S3 trigger
print(pick_compute(1800, False, "batch",  "zip"))       # 30-min ETL
print(pick_compute(300,  False, "batch",  "container")) # containerised job
print(pick_compute(60,   True,  "steady", "zip"))       # stateful worker
Output
Lambda
Batch
Fargate
EC2/Fargate+EFS

In production Audit five workloads against the decision framework

Pythonruns
from dataclasses import dataclass


@dataclass
class Workload:
    name: str
    duration_s: int
    stateful: bool
    traffic: str   # bursty | batch | steady
    packaging: str # zip | container


def pick(w: Workload) -> tuple[str, str]:
    if w.duration_s > 900:
        svc = "Batch" if w.traffic == "batch" else "EC2/Fargate"
        return svc, "duration > 15 min"
    if w.stateful:
        return "EC2/Fargate+EFS", "stateful"
    if w.packaging == "container":
        return "Fargate", "container image"
    if w.traffic == "batch":
        return "Lambda/Batch", "batch trigger"
    return "Lambda", "event-driven"


jobs = [
    Workload("S3-triggered transform",   30,   False, "bursty",  "zip"),
    Workload("Nightly settlements ETL",  3600, False, "batch",   "zip"),
    Workload("Report container job",     600,  False, "batch",   "container"),
    Workload("Session cache warmer",     120,  True,  "steady",  "zip"),
    Workload("Payment API handler",      5,    False, "bursty",  "zip"),
]

for j in jobs:
    svc, reason = pick(j)
    print(f"{j.name:<30}  {svc:<20}  ({reason})")
Output
S3-triggered transform          Lambda                (event-driven)
Nightly settlements ETL         Batch                 (duration > 15 min)
Report container job            Fargate               (container image)
Session cache warmer            EC2/Fargate+EFS       (stateful)
Payment API handler             Lambda                (event-driven)

See it happen The compute decision tree

yes + batchyes + othernoyesnoyesnoworkloadduration> 15 min?AWS BatchEC2 / Fargatestateful?EC2 / Fargate+EFScontainer?FargateLambda
  1. Every workload enters at the same first question: does it run longer than 15 minutes?
  2. Long-running batch jobs belong in AWS Batch; other long-running work goes on EC2 or Fargate.
  3. Short work hits the second question: does the job need persistent local disk or sticky state?
  4. Stateful jobs need EC2 or Fargate with EFS. Stateless jobs face one more question.
  5. Container image? Use Fargate. No container? Lambda is the default for short event-driven work.
Notice: the tree asks three yes/no questions in order: duration, state, packaging. Lambda is the right answer only when all three are no.
Scenario A

In your world

The situation

Your Jenkins job runs a Python ETL script that reads from S3, transforms, and writes to Postgres. It takes 4 minutes and is triggered by a cron at 2 AM. Someone on your team suggests "just put it in Lambda".

What actually breaks

Lambda has a hard limit of 900 seconds (15 minutes) per invocation (documented). At 4 minutes and 400,000 rows today, you are well inside the limit, but the S3 file triples to 1.2 million rows in six months at the same growth rate; the job now needs roughly 12 minutes and breaches the ceiling mid-load. Lambda terminates the environment with no exception raised in your code, so the job's own logs show no error — but Postgres ends up with only the ~800,000 rows that were committed before the cutoff, and finance does not notice the missing 400,000 rows until a downstream reconciliation report fails to balance days later. Also, Lambda runs inside a VPC only if you configure one; a naive deployment cannot reach your private Postgres instance.

What you'd do on Monday

The right home is AWS Batch (job definition + queue + compute environment on Fargate) or Fargate directly. Both support jobs over 15 minutes, both can live in your VPC, and Batch adds retry-on-failure for free. Lambda is correct only for the short event-driven pieces — like triggering the Batch job when the S3 file lands.

PythonTrigger a Batch job from an S3 event (needs boto3 + AWS credentials)
import boto3

def handler(event, context):
    bucket = event["Records"][0]["s3"]["bucket"]["name"]
    key    = event["Records"][0]["s3"]["object"]["key"]
    boto3.client("batch").submit_job(
        jobName="settlements-etl",
        jobQueue="etl-queue",
        jobDefinition="settlements-etl-job",
        containerOverrides={"environment": [
            {"name": "S3_BUCKET", "value": bucket},
            {"name": "S3_KEY",    "value": key},
        ]},
    )
Scenario B

Imagine you work at CRED

50k credit-card API calls per day

The numbers

Illustrative: 50k calls/day ≈ 0.6 calls/s average; burst to ~500 calls/s during peak rewards cycle. Lambda concurrency at burst: 500 × 0.05 s average duration = 25 concurrent executions. Lambda cost at 50k calls with 256 MB and 50 ms average: ~50k × 256/1024 × 0.05 s = 625 GB-seconds ≈ $0.01/day. Three t3.micro On-Demand instances 24/7 ≈ $30/month. Lambda wins until sustained throughput exceeds ~30k calls/minute, where EC2 flat rate becomes competitive.

What the Scenario-A answer does here

Lambda looks right and is cheap at this scale. The issue is not cost — it is the 15-minute ceiling. CRED's monthly statement job consolidates 12 months of transaction history per user. If a single user has 18,000 transactions and each takes 3 ms to process, the job takes 54 seconds: fine for Lambda. But the batch that processes 1 million users in parallel is still fine as individual Lambda functions, until a p99 user has 200k transactions and the function hits 600 seconds — still inside the limit, but only just.

What flips at this scale

At product scale you split: the per-user transform is a Lambda function (under 1 minute), and the fan-out orchestration is either a Step Functions state machine or an AWS Batch array job — not one giant Lambda. The flip is that you design for composability before you hit the edge.

The trade-off you'd defend

I would defend Lambda for the per-user transform and Batch for the bulk fan-out, trading a slightly more complex deployment (two compute targets) for the ability to scale each independently and stay well within service limits.

Illustrative numbers chosen for the arithmetic; not CRED-published figures.

Lambda vs Fargate vs EC2 vs Batch

AxisLambdaFargateEC2Batch
Max duration15 min (900 s)UnlimitedUnlimitedUnlimited
Cold startYes (100 ms–several s)Yes (30–60 s typical; longer for large image pulls, see ECS and Fargate in operation)No (always running)Yes (scale-out)
Operations overheadNoneLow (no patching)High (AMI, patches, ASG)Low (scheduler managed)
Pricing modelPer GB-secondPer vCPU-secondPer instance-hourPer vCPU-second (Fargate) or instance-hour (EC2)
Best forEvent-driven, short tasksContainerised, variable loadSteady-state, custom HWParallel ETL, retryable jobs

Gotchas

Lambda silently terminates at the 15-minute wall

Symptom

A long-running function produces no error, no exception, and no logs after the 14:59 mark. The job appears to succeed.

Cause

Lambda hard-terminates the execution environment at the configured timeout (max 900 s). No exception is raised inside your code; the environment just stops.

Fix

Set a timeout in your Lambda configuration that is lower than your measured p99 duration, so you get a visible Task timed out error. For jobs that can exceed 15 minutes, move them to Batch or Fargate.

Lambda inside a VPC without a NAT gateway cannot reach S3

Symptom

A Lambda function configured with a VPC can reach Postgres on the private subnet but fails to connect to S3 (Connection refused or timeout on port 443).

Cause

Traffic from a VPC-attached Lambda exits through the VPC's ENI; without a NAT Gateway or an S3 Gateway Endpoint, it has no route to the public S3 API.

Fix

Add an S3 Gateway VPC Endpoint (free) to your VPC's route tables. That routes S3 traffic through AWS's backbone, not the internet.

Choosing Lambda for a workload that holds a database connection open

Symptom

Postgres max_connections exhaustion during load testing, even though no single function takes long.

Cause

Each Lambda execution environment holds one database connection in module-level scope; at 500 concurrent executions you have 500 open connections.

Fix

Use RDS Proxy or PgBouncer in transaction mode between Lambda and Postgres. This is covered in Parameter groups and connection behavior.

Tricky questions

Lambda is always cheaper than EC2
Most people say

Lambda's per-request pricing is always cheaper.

Actually

At sustained high throughput (millions of requests per day), EC2 Reserved Instances or Fargate Savings Plans can cost significantly less. The break-even depends on your request rate, duration, and memory. Lambda's advantage is at low or bursty traffic.

Fargate and ECS are the same thing
Most people say

Fargate is AWS's container service.

Actually

Amazon ECS is the container orchestrator; Fargate is the serverless compute backend for ECS (and EKS). You can run ECS tasks on EC2 launch type (you manage instances) or Fargate launch type (AWS manages them). ECS on Fargate is the combination.

Scenario question

You're handed a Python ETL pipeline that reads a 2 GB CSV from S3, joins it with a Postgres table, and writes results back to S3. The script takes 8 minutes and runs once a night. The current setup is an EC2 t3.medium that stays running all day. Your task: propose a cheaper and more resilient replacement.

Model answer

I would move this to AWS Batch with a Fargate compute environment. The job runs once a night, so paying for a t3.medium ($30+/month) all day is pure waste. A Batch job definition points at a Docker image containing the Python script; the compute environment spins a Fargate task when the job starts and terminates it when it finishes — you pay for 8–10 minutes of vCPU/GB-second instead of 24 hours. I would add maxAttempts: 3 to the job definition so transient S3 or Postgres failures trigger automatic retries. The trade-off is a longer start-up time (30–60 s Fargate cold start vs immediate EC2 response), but a nightly job can absorb that. I would also add a CloudWatch alarm on job failure to replace the current "check the server in the morning" pattern.

Interview corner

CommonLambda or Fargate: how do you choose?
“I'd look at duration first: anything over 15 minutes can't use Lambda, so I go to Fargate or Batch. If it fits in 15 minutes and is event-driven, Lambda wins on cost and ops overhead. If it's containerised and I want zero patching, Fargate. The gotcha I watch for is Postgres connection pooling — Lambda at scale needs a proxy in front of the database.”
TrickCan Lambda be used for long-running batch jobs?
“Only up to 15 minutes. For anything longer, I'd use AWS Batch with a Fargate or EC2 compute environment. Lambda can still be useful to trigger the Batch job when an S3 file lands, but the heavy lifting belongs in Batch.”
Experience checkHow did your current ETL scripts run, and what would you change?
“My current scripts run on a Jenkins EC2 instance. I'd break them into two parts: a short Lambda triggered by the S3 event that validates the file and submits a Batch job, and the Batch job itself for the multi-minute transform. That removes the always-on EC2 cost and adds automatic retries.”
DesignYour team is debating Lambda vs ECS for a new microservice. What do you say?
“I'd ask three questions: duration (over 15 min rules out Lambda), traffic shape (spiky favours Lambda, steady favours ECS Fargate on cost), and packaging (if they already have a Docker image and CI pipeline, Fargate is lower friction). For a new service with bursty HTTP traffic and sub-15-min handlers, Lambda wins. For an always-on service with a container image, ECS Fargate.”

Memory hook

Lambda is the freelancer, Fargate is the contractor team, EC2 is the full-time hire, Batch is the factory shift.

If you remember only one thing:

Ask duration first. Over 15 minutes means no Lambda.

Self-check

  1. What is Lambda's maximum invocation duration by default?

    900 seconds (15 minutes), a hard ceiling. Check the quota page for the current figure before relying on it in production.

  2. What is the key operational difference between ECS on EC2 and ECS on Fargate?

    With EC2 launch type you provision and patch the EC2 instances. With Fargate, AWS manages the underlying compute.

  3. When would you choose AWS Batch over Lambda for an ETL job?

    When the job exceeds 15 minutes, needs parallel workers across many records, requires retry-on-failure at the job level, or the job duration varies widely and a per-invocation Lambda would be wasteful.

Prove it

Write the pick_compute function with all five compute options (including EKS). Add a sixth case: a stateless 45-minute containerised job triggered nightly. Show the recommendation and confirm it is not Lambda. Then extend the function to flag any EC2 recommendation as "review Spot savings" and rerun the audit table.

aws-03-02Card 2 of 6

EC2 baseline and Auto Scaling

MediumP3 · Role-dependentTalk about it12 min
ec2amiebsauto-scalinggravitonspotsession-manager

Why you care

EC2 is still the baseline. Every AWS service either runs on EC2 under the hood or needs a VPC that you designed around EC2 principles. More practically, Graviton compatibility and Spot interruption handling are the two most common questions about EC2 in product-company interviews, because both directly affect running costs. If you can explain why a Graviton3 c7g instance runs a Python workload faster per dollar than a comparable x86 instance, you demonstrate that you think about unit economics, not just feature lists.

Explain like I'm 5

Renting an EC2 instance is renting a room in a hotel. The AMI is the room's initial state when you check in — furniture, amenities. EBS is a hard drive you can unplug from one room and carry to another. An Auto Scaling Group is the hotel's reservation system: it books more rooms when the lobby fills up and checks people out when it empties. Session Manager is a spare key the hotel keeps for the maintenance team — no need to leave the front door open (no port 22 needed).

Core concept

Instance families and when they matter:

  • t3/t4g: burstable CPU. Fine for dev, light workloads, or jobs that spike briefly. Not for sustained CPU.
  • c (compute-optimised, e.g. c7g): best CPU-per-dollar for CPU-bound work. c7g uses Graviton3.
  • m (general purpose, e.g. m7i): balanced CPU and memory. The default for unknown workloads.
  • r (memory-optimised, e.g. r7g): 8+ GB RAM per vCPU. Right for in-memory joins, Spark executors.
  • p/g (GPU): model training and inference. GPU instances have a separate discussion.

Graviton: AWS Graviton (Arm64) instances deliver 10–40 % better price-performance than comparable x86 for most compute-bound and memory-bound workloads (AWS-published benchmark guidance; check aws.amazon.com/ec2/graviton for current numbers). Python 3.11+ and most common Python packages (numpy, pandas, psycopg2) publish Arm64 wheels. The only blockers are x86-only compiled dependencies or very old OS images.

AMIs: An AMI is a snapshot of a root volume plus launch metadata. Use Amazon Linux 2023 or an Ubuntu LTS AMI from the official publisher; never use community AMIs from unknown sources. Pin AMI IDs in Terraform or CloudFormation; an AMI rebuilt by the vendor can change what software is present.

EBS: The default gp3 volume type delivers 3,000 IOPS and 125 MB/s throughput at no extra cost; you can independently provision up to 16,000 IOPS and 1,000 MB/s. Use gp3 unless you have a measured IOPS requirement that justifies io2 (database-grade durability at higher cost). EBS is AZ-scoped — you can only attach a volume to an instance in the same AZ.

Session Manager (Systems Manager): SSM Session Manager gives shell access to an EC2 instance without opening port 22. The instance needs the AmazonSSMManagedInstanceCore IAM policy and a route to the SSM endpoints (VPC endpoint or NAT Gateway). The Session Manager workflow: IAM permission → aws ssm start-session --target i-xxxxx → interactive shell. No key pair, no inbound SG rule needed.

Auto Scaling Groups and Spot: An ASG has three capacity values: desired (current target), minimum and maximum. It integrates with an Elastic Load Balancer Target Group; health checks that fail lead to instance replacement. For cost savings, a mixed instance policy lets you run a base of On-Demand (for availability) and a surge of Spot (cheap, interruptible). Spot instances can be reclaimed with a 2-minute warning. Any workload on Spot must handle SIGTERM gracefully: checkpoint state, drain connections, acknowledge the interruption hook.

Decision rule for Spot: use Spot for stateless workers, batch jobs and CI runners. Keep On-Demand for your minimum healthy fleet, database replicas and anything that cannot tolerate a 2-minute eviction window.

Minimal example VPC subnet planning with ipaddress

Pythonruns
import ipaddress

vpc = ipaddress.IPv4Network("10.0.0.0/16")
subnets = list(vpc.subnets(new_prefix=24))
print(f"total /24 subnets in /16: {len(subnets)}")
for s in subnets[:4]:
    print(f"  {s}  usable={s.num_addresses - 5}")
Output
total /24 subnets in /16: 256
  10.0.0.0/24  usable=251
  10.0.1.0/24  usable=251
  10.0.2.0/24  usable=251
  10.0.3.0/24  usable=251

In production Mixed On-Demand and Spot fleet planner

Pythonruns
from dataclasses import dataclass


@dataclass
class FleetConfig:
    desired: int
    on_demand_base: int
    on_demand_pct: int   # of instances above base


def fleet_split(cfg: FleetConfig) -> dict:
    above_base = max(0, cfg.desired - cfg.on_demand_base)
    on_demand_surge = round(above_base * cfg.on_demand_pct / 100)
    spot = above_base - on_demand_surge
    total_od = cfg.on_demand_base + on_demand_surge
    return {"on_demand": total_od, "spot": spot, "desired": cfg.desired}


configs = [
    FleetConfig(desired=10,  on_demand_base=2, on_demand_pct=0),   # max Spot savings
    FleetConfig(desired=10,  on_demand_base=2, on_demand_pct=50),  # balanced
    FleetConfig(desired=10,  on_demand_base=10, on_demand_pct=0),  # no Spot (SLA)
]

for c in configs:
    f = fleet_split(c)
    print(
        f"desired={f['desired']}  "
        f"on_demand={f['on_demand']}  "
        f"spot={f['spot']}  "
        f"spot_pct={round(f['spot'] / f['desired'] * 100)}%"
    )
Output
desired=10  on_demand=2  spot=8  spot_pct=80%
desired=10  on_demand=6  spot=4  spot_pct=40%
desired=10  on_demand=10  spot=0  spot_pct=0%
Scenario A

In your world

The situation

Your team runs a nightly batch on a single On-Demand c5.xlarge. Someone notices the instance is idle 22 hours a day and proposes Spot to cut costs by ~70 %. Your job is to assess whether the workload is safe on Spot.

What actually breaks

A Spot instance can receive a 2-minute interruption notice while the batch is running. If the script has no checkpoint logic and no SIGTERM handler, the job is halfway through the settlements file when the instance is terminated. The next night's run starts from row 0, potentially double-loading the already-committed rows.

What you'd do on Monday

Three changes make it Spot-safe. First, add a Spot interruption handler that catches the EC2 metadata service notice and writes the last committed batch ID to a known S3 path. Second, add startup logic that checks that S3 path and resumes from the checkpoint. Third, run a base of two On-Demand instances in the ASG with Spot for the batch surge workers, so the instance pool is never empty.

PythonSIGTERM handler for Spot interruption (needs signal + checkpoint logic)
import signal
import sys

last_committed_id: int = 0

def _spot_shutdown(signum, frame):
    # write checkpoint before process exits
    import boto3
    boto3.client("s3").put_object(
        Bucket="my-etl-checkpoints",
        Key="settlements/last_id.txt",
        Body=str(last_committed_id).encode(),
    )
    sys.exit(0)

signal.signal(signal.SIGTERM, _spot_shutdown)
Scenario B

Imagine you work at Nykaa

200 EC2 c5.xlarge at peak sale event

The numbers

Illustrative: Nykaa runs a flash sale that spikes web traffic 20× overnight. Normal fleet: 20 c5.xlarge instances. Peak fleet: 200 instances. Spot price for c5.xlarge in ap-south-1 is roughly 60–70 % less than On-Demand (check current Spot pricing; prices fluctuate). Running 180 Spot + 20 On-Demand at peak vs 200 On-Demand saves ~$200/hour at that scale. In a 6-hour sale window: $1,200 saved vs. $0 extra risk if Spot is capacity-constrained.

What the Scenario-A answer does here

A flat Spot fleet works during normal times but a large sale causes Spot pool depletion across multiple instance families simultaneously. If the ASG is configured with a single instance type, all 200 Spot requests compete in the same pool; a supply crunch can leave the fleet at 20.

What flips at this scale

At this scale you define a mixed instance policy with 5–6 instance types (c5.xlarge, c5a.xlarge, c6i.xlarge, c6a.xlarge, c7g.xlarge for Graviton) across the same size band. EC2 diversifies across pools and the effective interruption rate drops sharply. You also add a scheduled scale-up 30 minutes before the sale window to pre-warm On-Demand capacity.

The trade-off you'd defend

I would defend a 6-type mixed fleet with 15 % On-Demand base over a pure Spot or pure On-Demand fleet, trading configuration complexity for 50–60 % cost savings with interruption risk well under 1 %.

Illustrative figures for the arithmetic; Spot discount range is directionally accurate but check the current Spot price history in your region.

Gotchas

Spot interruption mid-job with no handler

Symptom

Batch job produces partial results. Postgres has rows from the first half of the file, and the row count does not match the source.

Cause

The instance was reclaimed with a SIGTERM; no checkpoint was written; the job restarted from row 0 on a new instance.

Fix

Write a SIGTERM handler that flushes the current batch and records the last committed key to a durable store (S3 or Postgres) before exiting.

EBS volume stuck in wrong AZ

Symptom

aws ec2 attach-volume fails with "The specified attachment point is not available" despite both volume and instance existing.

Cause

EBS volumes are AZ-scoped. A volume created in ap-south-1a cannot attach to an instance in ap-south-1b.

Fix

Snapshot the volume and restore it in the correct AZ, or redesign so persistent state lives in a multi-AZ service like S3 or RDS.

Session Manager unavailable: instance cannot reach SSM endpoints

Symptom

aws ssm start-session --target i-xxx returns "An error occurred (TargetNotConnected)".

Cause

The instance's IAM role is missing AmazonSSMManagedInstanceCore, or there is no route to the SSM VPC endpoints or NAT Gateway.

Fix

Attach the managed policy to the instance role, then either create VPC interface endpoints for ssm, ssmmessages and ec2messages, or confirm the instance can reach the internet via NAT.

Tricky questions

Is Graviton (Arm64) compatible with Python 3.12?
Most people say

You need to recompile all Python packages for Arm64.

Actually

Python 3.11+ and almost all major packages (numpy, pandas, scipy, boto3, psycopg2-binary) publish official Arm64 wheels. The only risk is packages that ship only x86 compiled extensions. Always test with pip install --dry-run on an Arm64 instance before committing.

An ASG set to desired=0 saves money
Most people say

Yes, scaling desired to 0 removes all running costs.

Actually

The instances are terminated and you save compute costs, but the EBS volumes attached to those instances still accrue storage charges until deleted. Also, the launch template, ASG configuration and associated ELB target groups have no cost, but an ALB in front has a base hourly charge.

Scenario question

You're handed an EC2 Auto Scaling Group with 20 t3.xlarge On-Demand instances running a stateless data processing job. The instances are at ~15 % CPU most of the time and spike to 80 % for 2 hours nightly. What changes do you propose, and what are the risks?

Model answer

I would replace t3.xlarge On-Demand with a mixed fleet: a base of 3 On-Demand Graviton instances (c7g.xlarge or m7g.xlarge) for the steady minimum, plus Spot capacity from 3–4 instance types to handle the nightly spike. The Graviton instances give 20–40 % better price-performance for this CPU-bound work. The scheduled scale-out runs 15 minutes before the nightly spike begins. The risks: Spot capacity may be constrained at the exact moment the spike starts; mitigate by pre-warming and using multiple instance families. Graviton may reveal a compiled dependency that only ships x86 wheels; I would run the full test suite on a c7g instance in staging first.

Interview corner

CommonWhat is the difference between a launch template and an AMI?
“I'd say an AMI is the disk image — the root volume plus OS — and a launch template is the configuration that says which AMI to use, which instance type, which security groups and key pair. You can update the launch template without rebuilding the AMI.”
TrickShould you always use Graviton?
“I'd say usually yes for Linux CPU-bound and memory-bound workloads, but I'd first check that all compiled dependencies have Arm64 wheels. I'd also note that Windows on Graviton is not supported, and some GPU instance families are x86-only.”
Experience checkHow have you handled the Spot interruption risk in a batch job?
“I'd add a SIGTERM handler that checkpoints the last committed batch ID to S3, and startup logic that reads that checkpoint and resumes. I combine that with a mixed instance policy across 4–5 instance types so pool depletion is less likely.”
DesignDescribe how you'd auto-scale an ECS cluster on EC2 (not Fargate).
“I'd use two ASGs: one On-Demand base and one Spot surge, both feeding the same ECS cluster. ECS Cluster Auto Scaling uses a Capacity Provider to track how many tasks need to be placed and asks the ASG to scale up or down. I set the minimum healthy percentage and maximum surge on the Capacity Provider to control how fast it scales and how many tasks coexist during a replacement.”

Memory hook

The AMI is the room's starter kit, EBS is the suitcase, the ASG is the booking system, and Session Manager is the spare key.

If you remember only one thing:

Spot saves 60–70 %; add a SIGTERM checkpoint or risk half-loaded data.

Self-check

  1. What are the three capacity values on an Auto Scaling Group?

    Desired (current target), minimum (never scale below this) and maximum (never scale above this).

  2. Why does EBS zone-lock matter for Auto Scaling?

    If the ASG launches a replacement instance in a different AZ, the original EBS volume cannot follow. Design so persistent state lives in S3 or RDS, not a single EBS volume.

  3. What IAM policy must an instance role have for Session Manager to work?

    AmazonSSMManagedInstanceCore (or its individual permission subset), plus a route to the SSM VPC endpoints or NAT.

Prove it

Launch a /16 VPC in your mind (or on paper): design three public subnets and three private subnets across two AZs. Choose CIDR ranges so that no subnets overlap and each has at least 50 usable addresses. Show the CIDR plan as output from the ipaddress module. Then add a fourth column to the planner output that shows whether the subnet is public or private.

aws-03-03Card 3 of 6

Lambda execution and packaging

HardP3 · Role-dependentTalk about it13 min
lambdacold-startconcurrencylayerspackagingzipcontainerexecution-environment

Why you care

Cold start latency and concurrency exhaustion are the two most common production surprises with Lambda. If you cannot explain both in one minute, you have not operated Lambda at any meaningful traffic. "Your Lambda slowed down on Monday morning" is a cold-start question; "your Lambda is throttling downstream Postgres" is a concurrency question. Both are likely interview prompts for any company that runs Python microservices on AWS. The card also closes the packaging decision — zip vs container image — which affects cold start latency and build complexity.

Explain like I'm 5

A Lambda execution environment is a sleeping employee who needs to boot up a laptop before answering the first call. Booting the laptop is the cold start — it includes loading the runtime, importing your packages, and running your module-level code. Once awake, the employee handles calls one at a time, very fast, because the laptop stays on between calls. If 500 people call simultaneously, Lambda hires 500 employees and boots 500 laptops at once — that spike is the concurrency surge, and if each employee needs Postgres, you have just opened 500 database connections at the same moment.

Core concept

Execution environment lifecycle:

When Lambda receives an invocation for which no warm environment exists, it runs the Init phase: starts the Firecracker microVM, loads the runtime (Python 3.12), and runs your module-level code (imports, global variables, connection pool initialisation). This is the cold start. It then runs the Invoke phase: calls your handler function. Subsequent invocations that land on the same environment skip Init and go straight to Invoke — the warm path. The environment is frozen between invocations; module-level objects persist across warm calls, which is both a performance feature and a bug surface (stale state, exhausted iterators).

Concurrency formula:

Text
concurrent executions = requests_per_second × average_duration_s

The default account limit is 1,000 concurrent executions across all functions in a region (documented; soft limit, can be increased). Lambda always reserves 100 of those for unreserved functions. If you configure reserved concurrency on a function, it carves that number out exclusively — no other function can use it, and the function cannot exceed it. Provisioned concurrency pre-initialises environments so they are warm and ready to serve; it costs extra but eliminates cold starts.

Downstream backpressure: every concurrent execution that needs Postgres opens one connection. At 200 concurrent executions you saturate a typical max_connections=200 PostgreSQL instance. The fix is RDS Proxy or a connection pool in transaction mode (Parameter groups and connection behavior).

Concurrency scaling rate: per AWS's Lambda scaling behavior documentation, each function in each region can add up to 1,000 new execution environments per 10 seconds (equivalently, 10,000 requests per second every 10 seconds); this is a function-level rate, and Lambda does not accrue unused capacity from quiet periods. A sudden burst from 0 to 5,000 concurrent requests therefore takes about 50 seconds of ramp-up (5,000 ÷ 1,000 per 10 s = 5 intervals), during which excess requests are throttled with a 429.

Packaging:

  • Zip (deployment package): maximum 50 MB zipped (or 250 MB unzipped including layers); up to 5 layers maximum (documented). Fast to upload, fast to cold start on small packages. Best for pure Python with lightweight dependencies.
  • Container image: up to 10 GB uncompressed (documented). Required when dependencies exceed the zip limit, you need custom runtimes, or you want Docker-native CI/CD. Container images have a longer cold start (image pull adds seconds) unless provisioned concurrency is used.

Layers: a Lambda layer is a zip of shared code or binaries that your function can reference without bundling them in its own package. Use layers for shared utilities, compiled libraries (numpy, psycopg2) and per-team SDKs. The 5-layer limit and the 250 MB unzipped combined limit apply to function + layers together (documented).

Decision rule: use zip packaging until you hit the 250 MB limit, then move to container images. Use layers for shared dependencies across multiple functions. Use provisioned concurrency only when cold start latency is measured and proven to be a problem in production.

Under the hood

Lambda execution environments run inside Firecracker microVMs — a KVM-based virtual machine manager designed for sub-second startup. Firecracker boots a minimal Linux kernel; the Python runtime sits on top of that. The Init phase is the sum of Firecracker boot time, runtime startup, and your module-level import time. Python's importlib is the dominant contributor for large packages like pandas or scipy because it must parse, compile and cache each module. You can reduce this by lazy-importing (import inside the handler) or by using importlib.util.spec_from_file_location to defer loading until needed. The execution environment's /tmp directory persists between warm invocations, so a file written there in invocation N is still present in invocation N+1 on the same environment — useful for caching small lookup tables, dangerous if you assume a clean slate.

Minimal example Concurrency formula in Python

Pythonruns
def lambda_concurrency(rps, avg_ms):
    return round(rps * avg_ms / 1000, 1)

print(lambda_concurrency(100, 50))    # fast handler
print(lambda_concurrency(100, 500))   # 500 ms handler
print(lambda_concurrency(1000, 200))  # 200 ms at high traffic
print(lambda_concurrency(50, 15000))  # 15-second job
Output
5.0
50.0
200.0
750.0

In production Throttle budget checker

Pythonruns
def throttle_budget(rps, avg_ms, reserved=0, account_limit=1000):
    """
    Returns concurrent executions and whether the function will throttle.
    reserved=0 means the function uses the unreserved pool.
    """
    concurrent = round(rps * avg_ms / 1000, 1)
    cap = reserved if reserved > 0 else (account_limit - 100)  # 100 always held back
    throttle = concurrent > cap
    headroom = round(max(0.0, cap - concurrent), 1)
    return concurrent, cap, throttle, headroom


scenarios = [
    (100,  500,  0,   "ETL trigger, no reservation"),
    (500,  2000, 200, "heavy ETL, 200 reserved"),
    (5000, 200,  1000,"high-volume API, 1000 reserved"),
]

for rps, avg_ms, res, label in scenarios:
    c, cap, thr, hd = throttle_budget(rps, avg_ms, res)
    status = "THROTTLE" if thr else "ok"
    print(f"{label}")
    print(f"  concurrent={c}  cap={cap}  headroom={hd}  status={status}")
Output
ETL trigger, no reservation
  concurrent=50.0  cap=900  headroom=850.0  status=ok
heavy ETL, 200 reserved
  concurrent=1000.0  cap=200  headroom=0.0  status=THROTTLE
high-volume API, 1000 reserved
  concurrent=1000.0  cap=1000  headroom=0.0  status=ok

See it happen Cold start vs warm invocation

Request 1Init phase(cold start)Invoke phaseRequest 2Invoke phase(warm reuse)
  1. Request 1 arrives. No warm environment exists; Lambda must start the Firecracker VM and load the Python runtime.
  2. Init finishes. Lambda calls your handler. Total latency = Init time + Invoke time.
  3. The environment stays frozen between requests. Request 2 arrives while the first environment is still alive.
  4. Lambda reuses the existing environment. No Init phase. Latency = Invoke time only — the warm path.
  5. If 500 requests arrive at once, Lambda boots 500 environments simultaneously. Each opens connections, reads configs, imports packages — that is the cold-start storm.
Notice: the Init phase is a one-time cost per environment, but 500 concurrent cold starts mean 500 Init phases at the same moment, which is when your database connection limit breaks.
Scenario A

In your world

The situation

Your team puts the nightly ETL job in Lambda. It processes 1,000 S3 keys in parallel by triggering 1,000 invocations from an SQS queue with batch size 1 and maximum concurrency uncapped. Each invocation opens a Postgres connection at module level using psycopg2.connect().

What actually breaks

All 1,000 invocations cold-start within a few seconds of the SQS queue draining. Each opens one Postgres connection. Postgres max_connections is 200. The first 200 invocations proceed; the remaining 800 fail with connection refused or time out, and the job silently drops those records — no exception reaches the Lambda log because psycopg2.connect() was called at import time and the environment init failure is not surfaced as a handler error.

What you'd do on Monday

Three changes. First, set ReservedConcurrency on the Lambda function to a value well below Postgres max_connections (e.g. 50). Second, move the psycopg2.connect() call inside the handler (lazy connection) so a failed connection raises inside your handler where you can catch and log it. Third, for production, use RDS Proxy which pools connections between Lambda and Postgres.

PythonLazy Postgres connection inside handler (needs psycopg2 + RDS credentials)
import psycopg2

_conn = None

def get_conn():
    global _conn
    if _conn is None or _conn.closed:
        _conn = psycopg2.connect(host="rds-proxy.cluster.rds.amazonaws.com", ...)
    return _conn

def handler(event, context):
    conn = get_conn()   # only opens a connection when handler runs
    ...
Scenario B

Imagine you work at Uber

8000 trip-event invocations per second

The numbers

Illustrative: 8,000 Lambda invocations/second, each taking 200 ms → concurrent = 8000 × 0.2 = 1,600 executions. The default account limit is 1,000; Uber at this scale would hold a quota of several thousand (soft limit, per-request increase). At 1,600 concurrent executions each holding one Postgres connection, you have 1,600 open connections — 8× a standard max_connections=200 instance.

What the Scenario-A answer does here

The Scenario A fix (lazy connection + reserved concurrency cap) is still correct but now the reserved concurrency cap limits throughput. Capping at 200 means 200 / 0.2 = 1,000 invocations/second throughput, not 8,000. The job queues up in SQS.

What flips at this scale

At this scale you add RDS Proxy in front of Postgres. RDS Proxy multiplexes hundreds of Lambda connections into a small pool of real DB connections and handles reconnection after Lambda environment recycling. The Lambda concurrency cap is no longer set for connection safety; it is set for downstream quota management. Cold start time also matters at scale: a 300 ms cold start on a function that handles 10 ms of real work adds 30× latency for the first invocation after a scale-out event.

The trade-off you'd defend

I would defend RDS Proxy plus provisioned concurrency for the hot path (the trip-event Lambda) over setting a low reserved concurrency cap, trading provisioned concurrency cost for consistent sub-50 ms latency and full throughput.

Illustrative scenario; not Uber-published architecture.

Gotchas

Module-level code runs on every cold start, not once per account

Symptom

The team adds an S3 get_object call at the top of the Lambda module to load a config file. It works fine in dev but at scale every cold start triggers an S3 API call, and the config is re-fetched on every new environment.

Cause

Module-level code in Python runs once per execution environment creation (cold start), not once per account or per deployment.

Fix

Load the config once into a global variable. This is correct: the variable persists across warm invocations on the same environment. If the config must be refreshed, add a TTL check inside the handler.

Concurrency scaling floods a downstream database

Symptom

Postgres error remaining connection slots are reserved for non-replication superuser connections followed by job failure, coinciding with a traffic spike.

Cause

Lambda scales to 1,000 concurrent executions faster than Postgres can handle the new connections. Each environment opens one connection at module level.

Fix

Set ReservedConcurrency to stay below max_connections, use lazy connection with retry, or deploy RDS Proxy.

Container image cold start is significantly longer than zip

Symptom

A containerised Lambda function's p99 cold start is 3–8 seconds vs ~200 ms for a zip function.

Cause

The container image must be pulled from ECR and uncompressed before the runtime can start. A 2 GB image adds several seconds even with caching.

Fix

Minimise the container image (slim base, multi-stage build, delete build tools). Use provisioned concurrency to eliminate cold starts on the hot path.

Tricky questions

Provisioned concurrency eliminates all cold starts
Most people say

Yes, provisioned concurrency keeps environments warm so there are never cold starts.

Actually

Provisioned concurrency eliminates cold starts up to the provisioned count. Requests that exceed provisioned concurrency spill into on-demand environments, which can still cold-start. If reserved concurrency is not set, the function can use unreserved pool for the spill.

A Lambda function at 15 minutes duration can handle any ETL job
Most people say

As long as it finishes in 15 minutes, Lambda is fine.

Actually

A 15-minute function that opens a Postgres transaction holds the lock for 15 minutes and ties up a connection for the full duration. Long-held locks block other writers. Lambda is not designed for long-duration transactional work; use Batch or Fargate for those patterns.

Scenario question

You're handed a Lambda function that processes SQS messages from an ETL pipeline. It works correctly at 10 messages/second but fails intermittently at 500 messages/second with Postgres connection errors. Walk me through your investigation and fix.

Model answer

I would start with the concurrency formula: 500 messages/second × average duration = concurrent executions. If average duration is 200 ms, that is 100 concurrent executions. If average duration is 2 seconds, that is 1,000 — hitting the account limit. I would check CloudWatch ConcurrentExecutions and Throttles metrics first. If Postgres is failing, I would check DatabaseConnections on the RDS instance — if it is at max_connections, the fix is either a lower ReservedConcurrency setting or RDS Proxy. I would prefer RDS Proxy because it keeps Lambda throughput high while protecting the database. I would also move psycopg2.connect() inside a helper function with a module-level cache so connection failures raise inside the handler where I can log them with the message ID.

Interview corner

CommonWhat is a Lambda cold start and how do you reduce it?
“I'd describe it as the Init phase: Lambda creates the Firecracker VM, loads the Python runtime, and runs module-level code. I reduce it by keeping the deployment package small, lazy-importing heavy packages, and using provisioned concurrency on latency-sensitive functions.”
TrickDoes module-level code in a Lambda function run once per account?
“I'd say no: it runs once per execution environment, meaning once per cold start. Each environment is an isolated Firecracker VM. Module-level code does persist across warm invocations on the same environment, which is useful for connection pooling but dangerous if you rely on a clean slate.”
Experience checkHow has Lambda concurrency caused problems in your experience?
“I've seen Lambda functions that open a Postgres connection at module level scale to hundreds of concurrent environments during a spike, exhausting max_connections. The fix was reserved concurrency to cap the pool and eventually RDS Proxy to multiplex connections.”
DesignWhen would you use provisioned concurrency?
“When I have measured cold start latency that violates a latency SLA, and the traffic is high enough that environments stay warm anyway. The cost is extra; I would only pay it for functions on the critical path where p99 latency matters, like a payment authorisation handler.”

Memory hook

Lambda boots a new laptop for every cold start; the laptop stays on between warm calls, and every concurrent execution is a separate laptop.

If you remember only one thing:

Concurrent executions = req_per_sec × avg_duration_s. Watch that number against your account limit and your Postgres max_connections.

Self-check

  1. What is the default account-level Lambda concurrency limit per region?

    1,000 concurrent executions (documented; soft limit, can be increased via Service Quotas).

  2. What is the maximum number of Lambda layers on a single function?

    5 layers (documented hard limit).

  3. What is the combined unzipped size limit for a Lambda deployment package including layers?

    250 MB (documented). Use a container image (up to 10 GB) when you need more.

Prove it

Write a script that calculates Lambda concurrency for a function receiving traffic from an SQS queue. Given: queue depth 10,000 messages, Lambda processes each in 1.5 seconds with batch size 1, reserved concurrency 100. How long does the queue drain? Show the arithmetic and the drain time calculation. Then show what happens if reserved concurrency is raised to 500.

aws-03-04Card 4 of 6

ECS and Fargate in operation

HardP3 · Role-dependentTalk about it13 min
ecsfargatetasksservicestask-roleexecution-rolerolling-deployhealth-check

Why you care

ECS and Fargate are the dominant container platform for Python microservices on AWS. If you know Docker and can write a Dockerfile, ECS/Fargate is the fastest path from a container image to a production service with auto-scaling, health checks and rolling deployments. Product companies run ECS for the majority of their backend services because it requires no Kubernetes expertise. Interviewers ask "task role vs execution role" specifically because it separates people who have actually deployed to ECS from people who have only read the docs.

Explain like I'm 5

Imagine a shipping company. The task definition is the shipping label: it says what the container carries, how much space it needs, and which dock it loads at. The task is a single shipment in transit. The service is the logistics manager that keeps exactly five shipments moving at all times and replaces any that get lost. Fargate is the fleet of lorries: you hand them the label, they handle the driving. The task execution role is the credentials the lorry driver uses to collect the container from the warehouse (ECR). The task role is the credentials the shipment itself uses when it calls for help along the route (S3, Postgres, other AWS services).

Core concept

Key ECS concepts:

  • Task definition: a JSON or Terraform blueprint describing one or more containers, the image URL, CPU/memory, environment variables, log configuration and IAM roles.
  • Task: a running instance of a task definition. One task = one Fargate sandbox (one or more containers sharing a network namespace).
  • Service: keeps a desired count of tasks running. Handles replacement of unhealthy tasks, integrates with an ALB for HTTP routing, and manages rolling deployments.
  • Cluster: a logical grouping of services and tasks. With Fargate, the cluster has no EC2 instances to manage.

The role split — memorise this:

RoleWho uses itWhat it is for
Task execution roleECS agent (the platform)Pull image from ECR, write to CloudWatch Logs, decrypt secrets
Task roleYour container codeCall S3, Postgres, SQS, DynamoDB — whatever your app needs

Confusing them produces AccessDenied on your app's S3 calls even though the execution role is correct.

Rolling deployments: when you deploy a new task definition version, ECS replaces old tasks with new ones according to two parameters:

  • minimumHealthyPercent: the floor — never let the running task count fall below this fraction of desired.
  • maximumPercent: the ceiling — never let total running tasks (old + new) exceed this fraction of desired.

A 100/200 configuration (minimum 100 %, maximum 200 %) adds new tasks first, then removes old ones; the service briefly has 2× the desired count. A 50/100 configuration removes half the old tasks before starting new ones — risky if health checks are slow.

Health checks: ECS checks the ALB target group health or (for tasks without an ALB) the container health check command. A task that fails health checks is deregistered and replaced. Set a grace period (e.g. 60 seconds) so that tasks aren't killed before they finish initialising.

Immutable images: never use :latest in production. Tag images with the Git commit SHA or a semantic version. Rolling back then means redeploying the previous tagged image, not guessing which :latest was running.

Decision rule: use ECS Fargate for containerised services where you want managed compute, rolling deployments and ALB integration. Add EC2 launch type only if you need GPU or Spot with tight cost control. Use ECS Express Mode when you need App Runner's simplicity on ECS — see App Runner eligibility and Batch workloads for the sourced App Runner status.

Under the hood

Each Fargate task gets its own Elastic Network Interface (ENI) injected directly into your VPC — the awsvpc network mode. The task's IP is a real VPC IP from one of your subnets. Two consequences follow. First, it consumes a VPC IP per task; a /24 subnet has 251 usable addresses after AWS reservations — 300 tasks need at least two subnets. Second, security group rules apply per task, not per instance, giving precise per-service network controls. The ENI is created at task start and destroyed at task stop; that is part of the Fargate cold start time (30–60 seconds for a new task).

Minimal example Rolling deploy surge cap

Pythonruns
def ecs_surge_cap(desired, max_pct):
    return desired + desired * max_pct // 100

print(ecs_surge_cap(10, 0))    # no surge: one-for-one swap
print(ecs_surge_cap(10, 50))   # 50% surge
print(ecs_surge_cap(10, 100))  # 100% surge: double briefly
Output
10
15
20

In production ECS deploy parameter validator

Pythonruns
from dataclasses import dataclass


@dataclass
class DeployConfig:
    desired: int
    min_healthy_pct: int  # minimumHealthyPercent
    max_pct: int          # maximumPercent
    health_grace_s: int   # healthCheckGracePeriodSeconds


def validate(c: DeployConfig) -> list[str]:
    issues = []
    floor = c.desired * c.min_healthy_pct // 100
    cap = c.desired * c.max_pct // 100
    if floor < 1:
        issues.append("min_healthy_pct rounds to 0 tasks — service will have zero running tasks")
    if cap <= c.desired and c.min_healthy_pct == 100:
        issues.append("max_pct <= 100 with min_healthy=100: no room to add new tasks, deploy will stall")
    if c.health_grace_s < 30:
        issues.append(f"grace period {c.health_grace_s}s is short — fast-failing apps need at least 30s")
    if not issues:
        issues.append("ok")
    return issues


configs = [
    DeployConfig(desired=4, min_healthy_pct=100, max_pct=200, health_grace_s=60),
    DeployConfig(desired=4, min_healthy_pct=100, max_pct=100, health_grace_s=60),
    DeployConfig(desired=4, min_healthy_pct=100, max_pct=200, health_grace_s=10),
]

for cfg in configs:
    for msg in validate(cfg):
        print(f"desired={cfg.desired} min={cfg.min_healthy_pct}% max={cfg.max_pct}%: {msg}")
Output
desired=4 min=100% max=200%: ok
desired=4 min=100% max=100%: max_pct <= 100 with min_healthy=100: no room to add new tasks, deploy will stall
desired=4 min=100% max=200%: grace period 10s is short — fast-failing apps need at least 30s

See it happen Rolling deployment task lifecycle

deploy triggerECS serviceold tasknew taskALBnew task definition v2start task (v2)register + health checkHEALTHYderegister old taskconnection drainSIGTERM → SIGKILL
  1. A new task definition version is registered. The service begins a rolling deployment.
  2. ECS launches a new Fargate task. It takes 30–60 s for the ENI and container to come up.
  3. The new task registers with the ALB target group. Health checks begin.
  4. Once the task passes health checks, ECS marks it healthy.
  5. ECS asks the ALB to stop sending new requests to the old task.
  6. The ALB drains existing connections from the old task (default 300 s).
  7. After draining, ECS sends SIGTERM then (after 30 s) SIGKILL. The old task stops.
Notice: the old task continues to receive traffic until drain completes. A task that does not handle SIGTERM gracefully may drop in-flight requests when the process is killed.
Scenario A

In your world

The situation

You deploy a new version of your ETL container to ECS. The task definition changes the S3 bucket name via an environment variable. Post-deploy, CloudWatch logs show the service is healthy but data is going to the old bucket.

What actually breaks

The ECS service uses minimum_healthy_percent=100 and maximum_percent=200. The rolling deploy correctly starts new tasks and drains old ones. But the new task has the updated environment variable, and the old task — which is still running while drain completes — uses the old bucket name. During the 300-second default drain window, writes to the old bucket continue.

What you'd do on Monday

Two changes. First, reduce the drain timeout (deregistration_delay.timeout_seconds) to 30–60 seconds for your stateless ETL jobs so old tasks drain faster. Second, make the code idempotent on the bucket: always read os.environ["S3_BUCKET"] at invocation time, not at module level, so a restart would immediately pick up the new value. For future deploys that change destination, add a blue/green step where both old and new tasks write to a staging prefix first, validated before promoting.

PythonConfirm task is using the updated environment variable (needs boto3)
import boto3

ecs = boto3.client("ecs")
tasks = ecs.list_tasks(cluster="etl-cluster", serviceName="settlements-writer")["taskArns"]
desc = ecs.describe_tasks(cluster="etl-cluster", tasks=tasks)["tasks"]
for t in desc:
    env = {e["name"]: e["value"] for c in t["containers"] for e in c.get("environment", [])}
    print(t["taskArn"][-12:], env.get("S3_BUCKET", "NOT SET"))
Scenario B

Imagine you work at Shopify

400 ECS tasks per service during flash sale

The numbers

Illustrative: 400 tasks × 1 vCPU × 2 GB RAM = 400 vCPU and 800 GB RAM in flight during the sale. A rolling deploy with max_pct=200 briefly runs 800 tasks for the duration of the deploy. At 40 s per task startup and health check, and replacing 400 tasks in batches, a full rolling deploy takes 10–15 minutes. 800 tasks × 2 GB = 1,600 GB RAM momentarily, requiring subnet IP space to cover 800 simultaneous task ENIs.

What the Scenario-A answer does here

The Scenario A fix (shorten drain, idempotent env reads) works at hundreds of tasks, but the rolling deploy window itself is a risk at peak traffic: 15 minutes of mixed versions handling orders simultaneously, with the old version potentially using a deprecated API or schema.

What flips at this scale

At this scale, the rolling deploy becomes a blue/green deployment: a second ECS service running the new version receives a small percentage of traffic via a weighted target group, validated against error metrics, before the old service is switched off entirely. The deploy window drops from 15 minutes to 30 seconds of switchover time.

The trade-off you'd defend

I would defend blue/green over rolling for any service handling financial transactions at scale, trading higher temporary resource cost (two full services running simultaneously) for a two-minute validation window and instant rollback by reweighting the ALB.

Illustrative numbers for the arithmetic; not Shopify-published architecture.

Gotchas

Task execution role vs task role confusion

Symptom

AccessDenied: s3:GetObject in the container, even though the IAM console shows the ECS task has an S3 policy attached.

Cause

The S3 policy was attached to the task execution role (used by the ECS agent for ECR and CloudWatch), not the task role (used by code inside the container).

Fix

Attach the S3 policy to the IAM role in the task definition's taskRoleArn field. Confirm by running aws sts get-caller-identity inside the container — the role name should match the task role.

Using :latest in the task definition

Symptom

After a redeploy, old tasks are replaced with new ones, but rolling back to the previous version means guessing which image was running.

Cause

:latest is a mutable pointer; the previous image is lost unless it was also tagged.

Fix

Tag every image with the Git commit SHA (sha256 or short hash from CI). The task definition refers to my-service:abc1234, and rollback is a new deployment of the previous task definition revision.

Tasks fail health checks during startup and are killed in a loop

Symptom

ECS service shows tasks cycling: starting, failing, stopping, starting again. The service never reaches desired count.

Cause

Health check grace period is too short; the application takes longer to initialise than the grace period allows, so ECS deregisters it before it is ready.

Fix

Set healthCheckGracePeriodSeconds to at least the measured p95 startup time. Add a lightweight /health endpoint that returns 200 as soon as the app is up.

Tricky questions

Does the task role need to be the same as the task execution role?
Most people say

Yes, they are the same role configured in the task definition.

Actually

They are two separate IAM roles with different purposes. The task execution role is used by the ECS/Fargate agent. The task role is assumed by your application code via the EC2 instance metadata service endpoint available inside the container (169.254.170.2). They must be different: giving your app code the execution role's ECR/CloudWatch permissions is a security violation.

Does Fargate awsvpc mode require a load balancer?
Most people say

Yes, each Fargate task must be behind an ALB.

Actually

An ALB is required only if you want HTTP routing and health-check-based task registration. A standalone Fargate task (a one-off batch job) can run without any load balancer. Service Connect or Cloud Map handles service discovery between tasks when you want internal routing.

Scenario question

You're handed an ECS service running 20 Fargate tasks. A deployment has just rolled out and 30 % of tasks are failing health checks. The ECS console shows the service is in DRAINING state for many old tasks. What do you do?

Model answer

I would immediately stop the deployment by registering the previous task definition revision as the current deployment. In ECS this means updating the service with the old task definition revision number, forcing ECS to roll back. While that runs, I would look at the logs of a failing new task to understand why health checks fail: is it a missing environment variable, a bad secret reference, a missing IAM permission, or an application startup crash? I would not wait for the rollback to complete before investigating — the CloudWatch log group for the service will have the failure reason. Once the old version is stable, I would fix the root cause in a feature branch, test in staging, and redeploy with a longer health check grace period and a canary target group carrying 5 % of traffic before full rollout.

Interview corner

CommonWhat is the difference between a task execution role and a task role in ECS?
“I'd say the execution role is used by the ECS agent itself to pull the image from ECR and write logs to CloudWatch. The task role is the IAM role your application code assumes — the one you use to call S3, SQS or Postgres. They're separate in the task definition: executionRoleArn and taskRoleArn.”
TrickWhat happens if minimumHealthyPercent=100 and maximumPercent=100 in an ECS rolling deploy?
“I'd say the deploy stalls. ECS can't start new tasks without going over the max, and can't remove old tasks without going under the min. You need max_percent > 100 to make room for new tasks to start before old ones are drained.”
Experience checkHave you seen an ECS deployment fail? What was the cause?
“I've seen tasks cycle in a crash loop because the health check grace period was shorter than the application startup time. Every time a new task started, ECS killed it before it was ready. The fix was increasing the grace period to match the measured p95 startup time.”
DesignHow would you minimise downtime during an ECS deployment?
“I'd use blue/green with a weighted ALB target group. The new service starts with 0 % traffic, gets validated for 5 minutes, then traffic shifts to 100 % new and 0 % old. Rollback is instant: reweight the ALB. For a simpler service, 100/200 rolling with connection draining and a long grace period gets you to near-zero downtime.”

Memory hook

The task execution role is the lorry driver's dock pass; the task role is the shipment's customs declaration. They are not the same.

If you remember only one thing:

Two separate IAM roles: execution role for the ECS agent, task role for your app code.

Self-check

  1. What is awsvpc network mode and what does it cost per Fargate task?

    Each Fargate task gets its own ENI, consuming one VPC IP address. Plan subnet CIDR blocks to cover your maximum task count, not your average.

  2. What is the minimum ECS deploy configuration that avoids service downtime?

    minimumHealthyPercent=100 and maximumPercent>=200. This ensures at least the desired count is always healthy, and there is room to start new tasks before removing old ones.

  3. How do you roll back an ECS deployment in under two minutes?

    Update the service to the previous task definition revision. ECS will immediately start tasks with the old definition and drain the new ones.

Prove it

Write the validate() function from the production block but extend it with a fourth check: warn if desired × (100 - min_healthy_pct) / 100 rounds to more than 2 tasks (meaning more than 2 tasks could be in transition at once on a 10-task service). Run it against five representative configs and paste the output.

aws-03-05Card 5 of 6

App Runner eligibility and Batch workloads

MediumP3 · Role-dependentTalk about it12 min
batchapp-runnerjob-queuecompute-environmentretryecs-express-mode

Why you care

AWS Batch is the right compute for embarrassingly parallel ETL — exactly the kind of work you do at TCS. If you are running a Python script that processes 10 GB of data overnight, Batch handles the queue, the retry logic, the scaling and the Spot cost optimisation for you. Understanding Batch queues, job definitions and retry strategies is a concrete skill that directly maps to your current workload. The App Runner section is a mandatory awareness item: it is closed to new customers, yet plenty of older tutorials and blog posts still walk through creating one, so knowing the restriction before it bites you is essential.

Explain like I'm 5

AWS Batch is a postal sorting office. You send parcels (job definitions) to a loading bay (job queue). The sorting office has a fleet of vans (compute environment) that it dispatches as needed, using cheap overnight vans (Spot) where possible, with expensive express vans (On-Demand) as a fallback. If a parcel is damaged in transit, the office re-sends it automatically (retries). You never deal with the vans directly. App Runner, meanwhile, was a self-service post office counter for web applications — but the counter is now closed to new customers, and existing customers are told to migrate to the new express service (ECS Express Mode).

Core concept

AWS Batch core concepts:

  • Job definition: a blueprint for a job: the container image, vCPU and memory requirements, environment variables, mount points, command override and retry configuration.
  • Job queue: an ordered queue that holds submitted jobs. A queue is connected to one or more compute environments by priority.
  • Compute environment: a pool of EC2 or Fargate compute that Batch manages. Batch scales it from zero to your maxvCpus as the queue fills and drains it back to zero when empty.
  • Array jobs: a single submit call that fans out to N identical jobs, each receiving its index as AWS_BATCH_JOB_ARRAY_INDEX. Ideal for processing N files or N shards of a dataset in parallel.
  • Job dependencies: a job can declare dependsOn another job. Batch holds it until the parent succeeds.

Retry strategy: you set maxAttempts in the job definition (check the current quota page for the maximum; at time of writing the maximum maxAttempts is 10 for Fargate, check the documentation). Batch retries on container exit code != 0. Add evaluateOnExit rules to differentiate: retry on SPOT_INTERRUPTION, fail on OUT_OF_MEMORY.

Spot with Batch: Fargate Spot compute environments can achieve ~70 % cost savings vs On-Demand (price varies by region and pool; check the current Spot price history). For ETL jobs that can checkpoint and restart, always use Fargate Spot with maxAttempts >= 3 and evaluateOnExit retrying on spot interruption.

App Runner restriction: per AWS's own App Runner availability change documentation, App Runner is closed to new customers. Existing customers can continue to use the service and create new resources, but AWS has stated no new features will be added. The documented migration path is Amazon ECS Express Mode, which provisions a full ECS service on Fargate — including an Application Load Balancer, auto-scaling and networking — from a single aws ecs create-express-gateway-service call taking a container image and two IAM roles (ecsTaskExecutionRole and ecsInfrastructureRoleForExpressServices). If you cannot create an App Runner service in your account, you are a new customer; use ECS (via Express Mode or directly) instead. Check account eligibility before planning any App Runner deployment.

Decision rule for ETL: if the job is under 15 minutes and simple → Lambda. If over 15 minutes, needs parallel workers, or requires Spot cost savings with automatic retry → Batch. For web services that need App Runner's simplicity → ECS Express Mode.

Minimal example Batch retry simulator with exponential backoff

Pythonruns
def simulate_batch_job(job_id, outcomes):
    for attempt, outcome in enumerate(outcomes, 1):
        wait = 2 ** (attempt - 1)
        print(f"  {job_id} attempt {attempt}: {outcome}  (backoff: {wait}s)")
        if outcome == "ok":
            return True
    return False

print("transient error (fails twice, then succeeds):")
simulate_batch_job("etl-001", ["fail", "fail", "ok"])
print("permanent error (exhausts all attempts):")
simulate_batch_job("etl-002", ["fail", "fail", "fail"])
Output
transient error (fails twice, then succeeds):
  etl-001 attempt 1: fail  (backoff: 1s)
  etl-001 attempt 2: fail  (backoff: 2s)
  etl-001 attempt 3: ok  (backoff: 4s)
permanent error (exhausts all attempts):
  etl-002 attempt 1: fail  (backoff: 1s)
  etl-002 attempt 2: fail  (backoff: 2s)
  etl-002 attempt 3: fail  (backoff: 4s)

In production Batch array job fan-out planner

Pythonruns
import math
from dataclasses import dataclass


@dataclass
class ArrayJobPlan:
    total_records: int
    records_per_job: int
    max_vcpus: int
    vcpu_per_job: int


def plan_array(p: ArrayJobPlan):
    array_size = math.ceil(p.total_records / p.records_per_job)
    max_parallel = p.max_vcpus // p.vcpu_per_job
    concurrent = min(array_size, max_parallel)
    batches = math.ceil(array_size / max(1, concurrent))
    print(f"total records:   {p.total_records:>10,}")
    print(f"records per job: {p.records_per_job:>10,}")
    print(f"array size:      {array_size:>10,}  (jobs to submit)")
    print(f"max parallel:    {concurrent:>10,}  (limited by max_vcpus={p.max_vcpus})")
    print(f"serial batches:  {batches:>10,}  (how many rounds to drain the queue)")


plan_array(ArrayJobPlan(
    total_records=5_000_000,
    records_per_job=10_000,
    max_vcpus=2000,
    vcpu_per_job=4,
))
Output
total records:    5,000,000
records per job:     10,000
array size:             500  (jobs to submit)
max parallel:           500  (limited by max_vcpus=2000)
serial batches:           1  (how many rounds to drain the queue)
Scenario A

In your world

The situation

Your nightly settlements ETL takes 45 minutes on a single EC2 instance. The team wants to parallelise across 50 shards. Someone suggests Lambda.

What actually breaks

Lambda has a 15-minute timeout. A 45-minute job cannot run in Lambda. Even if you split the job into 50 shards that each take under 15 minutes, Lambda concurrency at 50 simultaneous functions risks saturating the Postgres max_connections. And there is no built-in retry-on-failure at the job level; a Lambda that fails mid-shard silently drops that shard unless you build SQS dead-letter logic.

What you'd do on Monday

Use AWS Batch. Define a job definition with 1 vCPU / 2 GB RAM per shard. Submit an array job with arraySize=50. Each job receives its shard index from AWS_BATCH_JOB_ARRAY_INDEX. Set maxAttempts=3 with evaluateOnExit to retry on Spot interruption. Batch scales the compute environment to 50 jobs running in parallel (up to maxvCpus) and retries any failed shard automatically. Total wall clock time drops from 45 minutes (serial) to under 5 minutes (parallel 50-way).

PythonSubmit an AWS Batch array job (needs boto3 + AWS credentials)
import boto3

batch = boto3.client("batch")
response = batch.submit_job(
    jobName="settlements-etl",
    jobQueue="etl-queue",
    jobDefinition="settlements-etl-job",
    arrayProperties={"size": 50},       # fan out to 50 parallel jobs
    retryStrategy={
        "attempts": 3,
        "evaluateOnExit": [
            {"onStatusReason": "Host EC2*terminated", "action": "RETRY"},
            {"onReason": "CannotPullContainerError*", "action": "RETRY"},
        ],
    },
)
print("submitted:", response["jobId"])
Scenario B

Imagine you work at Lyft

8M ride records per night

The numbers

Illustrative: 8M records ÷ 10k per shard = 800 shards. At 4 vCPU per shard and a Batch compute environment capped at maxvCpus=3200, all 800 shards run in parallel (3200 / 4 = 800). Each shard processes 10k records in ~3 minutes, which is 18 ms/record (180,000 ms ÷ 10,000 records). Total wall time in parallel: ~3 minutes. Serial on one EC2 at that same 18 ms/record: 8M × 18 ms ≈ 144,000,000 ms ≈ 144,000 s ≈ 40 hours — consistent with the 800-way parallel speedup, since 40 hours ÷ 800 ≈ 3 minutes. Spot compute at 70 % discount: 800 shards × 4 vCPU × (3 min / 60) × Fargate vCPU price ≈ check current pricing; illustrative magnitude is a few dollars.

What the Scenario-A answer does here

800 simultaneous Batch jobs each connecting to Postgres overwhelms max_connections. Even with RDS Proxy, the write rate from 800 parallel writers hitting the same table causes lock contention and write amplification.

What flips at this scale

The flip is to write each shard's output to S3 (a Parquet file per shard) rather than directly to Postgres. A final aggregation job — a 801st array child that runs after all 800 succeed via dependsOn — merges the Parquet files and bulk-loads into Postgres with COPY. This separates compute parallelism from database write serialisation.

The trade-off you'd defend

I would defend write-to-S3-then-COPY over direct parallel Postgres writes, trading an extra aggregation step and S3 storage cost for zero lock contention and a single, controllable load transaction.

Illustrative numbers for the arithmetic; not Lyft-published architecture.

Gotchas

New customers cannot create AWS App Runner services

Symptom

You follow an App Runner tutorial, run aws apprunner create-service, and receive a ServiceQuotaExceededException or AccessDeniedException error even with correct IAM permissions.

Cause

AWS App Runner is closed to new customers. New accounts cannot create App Runner services. No amount of IAM changes will resolve this.

Fix

Use Amazon ECS with Express Mode (aws ecs create-express-gateway-service, per AWS's official App Runner-to-ECS migration guide) as the direct replacement. It provides the same one-API-call simplicity with a full ECS stack under the hood.

Batch job succeeds but produces wrong output silently

Symptom

The Batch job exits 0, the CloudWatch log shows no error, but only 40 of 50 shards have output in S3.

Cause

A shard that processes zero records still exits 0. If the input sharding logic does not account for file count or records count mod shard count, the last few shards receive empty inputs and produce empty outputs — which is not an error.

Fix

Add a shard-level assertion: if records_processed == 0 and is_not_empty_shard, exit 1. This surfaces as a Batch job failure and triggers a retry.

Batch compute environment stays INVALID after creation

Symptom

The Batch compute environment enters INVALID state immediately after creation.

Cause

Most commonly a missing service-linked role (AWSServiceRoleForBatch) or the VPC/subnets not matching the ECS service endpoints required by Batch.

Fix

Ensure the service-linked role exists (aws iam create-service-linked-role --aws-service-name batch.amazonaws.com). Check that the subnets have routes to ECS and ECR endpoints (VPC endpoints or NAT Gateway).

Tricky questions

Can existing App Runner customers still create new services?
Most people say

No, App Runner is shut down for everyone.

Actually

Existing customers can continue to use App Runner and create new resources within their existing accounts. Only new customers (accounts that have never used App Runner) are blocked. AWS will continue security and availability investment but add no new features.

Lambda is always faster than Batch for short ETL jobs
Most people say

Lambda starts instantly while Batch has cold start overhead.

Actually

Lambda is faster to invoke, but Batch on Fargate with a pre-loaded container image has a similar 30–60 second cold start for the first job in a queue. For a nightly job, the Batch cold start is irrelevant. Lambda's advantage is immediate invocation for sub-15-minute event-driven work; Batch's advantage is managed parallelism, Spot, retry and job dependency logic.

Scenario question

You're handed a Python ETL script that processes one S3 file per night, takes 3 hours, and runs on a single EC2 instance. Your task: reduce it to under 30 minutes without changing the core transform logic.

Model answer

I would shard the input by splitting the S3 file into 10 shards (by line range or record count), each processed by one Batch array job child. Each child reads its shard from S3, transforms it, and writes a Parquet output to S3. A final dependency job merges and loads the Parquet files. With 10 parallel workers, a 3-hour job becomes 20 minutes of compute plus a few minutes of merge — well under 30 minutes. I would use a Fargate Spot compute environment with maxAttempts=3 and a SPOT_INTERRUPTION retry rule to keep costs low. The trade-off is that the merge step adds a few minutes and requires storing intermediate Parquet files in S3, adding a small storage cost. I would add a job-level assertion that the record count in the merged output matches the source file.

Interview corner

CommonWhat is an AWS Batch array job?
“I'd describe it as a single job submission that fans out to N copies of the same job definition, each receiving its index as an environment variable. I use it to parallelise a large file across N shards, each processing one slice of the data.”
TrickIs App Runner a good choice for a new containerised web service in 2025?
“I'd say no. App Runner is closed to new customers — you can't create a new service on a new account. The recommended replacement is ECS Express Mode, which gives the same one-command simplicity but runs on ECS/Fargate with full ECS feature access.”
Experience checkHow do you make a Batch job restartable after a Spot interruption?
“I'd write the last committed record ID or S3 key to a checkpoint in S3 or DynamoDB at the end of each batch. On startup, the job reads the checkpoint and skips already-processed records. I also add an evaluateOnExit rule in the Batch retry strategy to retry on the Spot interruption status reason.”
DesignHow would you process 5 million records overnight in under 10 minutes with AWS Batch?
“I'd split into 500 shards of 10,000 records each, submit a Batch array job of size 500, cap the compute environment at 2,000 vCPUs with 4 per job, and all 500 run in parallel. Each shard writes to S3. A final job uses dependsOn to wait for all 500 and then runs COPY to load Postgres. The wall time is the time for one shard plus the merge.”

Memory hook

Batch is the sorting office: submit a parcel, the office handles the vans, retries and Spot.

If you remember only one thing:

App Runner is closed to new customers. Use ECS Express Mode instead.

Self-check

  1. What environment variable does each child in a Batch array job receive to know its index?

    AWS_BATCH_JOB_ARRAY_INDEX.

  2. What ECS feature is the recommended migration path from App Runner?

    Amazon ECS Express Mode (aws ecs create-express-gateway-service).

  3. What does evaluateOnExit in a Batch retry strategy do?

    It allows you to conditionally retry or fail based on the exit code, exit reason or status reason. For example: retry if the status reason indicates a Spot interruption, but fail immediately on an out-of-memory exit reason.

Prove it

Write the array job planner from the production block and extend it with a time estimate: given that each shard takes shard_duration_min minutes, compute total wall time as max(1, serial_batches) × shard_duration_min. Show plans for three configurations: (5M records, 10k/shard, 2000 vCPU, 4 vCPU/job, 3 min/shard), (5M records, 1k/shard, 500 vCPU, 1 vCPU/job, 2 min/shard), and (500k records, 50k/shard, 100 vCPU, 2 vCPU/job, 1 min/shard).

aws-03-06Card 6 of 6

EKS-lite and orchestration boundaries

MediumP4 · DifferentiatorKnow it exists12 min
ekskubernetesorchestrationmanaged-control-planeawareness

Why you care

You will encounter EKS in job descriptions and architecture discussions. You need to know enough to say "we don't need Kubernetes for this" and defend it. Product companies that run EKS have entire platform engineering teams maintaining it. If your five-engineer backend team is considering adding EKS because the job description said Kubernetes, you need to explain exactly what operational tax that adds — in concrete terms, not abstract "complexity". Interviewers at platform-heavy companies ask EKS questions to test operational judgement, not just familiarity with kubectl.

Explain like I'm 5

Running ECS is like hiring a restaurant chain to cook for you: you say what's on the menu and how many tables to set, they handle the kitchen. Running EKS is like buying a restaurant building and installing a commercial kitchen: the building company (AWS) maintains the structure, but you hire the kitchen staff, set up the ventilation, calibrate every oven, and train the cooks. For most teams cooking three meals a day, the restaurant chain is the right call. Buy the building only if you need a custom commercial kitchen that no restaurant chain offers.

Core concept

What EKS is: Amazon EKS runs a managed Kubernetes control plane. AWS manages the API server, etcd, controller manager and scheduler. You pay a per-cluster hourly charge (check the current EKS pricing page; illustrative: $0.10/hour per cluster = ~$72/month). You still manage the worker nodes — either self-managed EC2 nodes, EKS managed node groups, or Fargate profiles.

Your responsibility on EKS:

  • Nodes: patching the OS and Kubernetes kubelet. Managed node groups automate the rolling update of node OS.
  • Add-ons: VPC CNI, CoreDNS, kube-proxy. Each has a version that must be kept compatible with the Kubernetes version.
  • Cluster version upgrades: Kubernetes releases a new minor version roughly every 4 months; EKS supports a version for about 14 months. You must upgrade clusters on a regular schedule.
  • Workload concerns: Pod resource requests and limits (CPU/memory), Pod Disruption Budgets, Horizontal Pod Autoscaler, node affinity, taints and tolerations, network policies.

What EKS does not give you for free that ECS does: ECS handles service discovery, load balancer integration and rolling updates natively. On EKS you add separate controllers: AWS Load Balancer Controller (for ALB integration), ExternalDNS (for Route 53), Cluster Autoscaler or Karpenter (for node scaling). Each add-on has its own versioning and upgrade cycle.

When EKS makes sense:

  • You already operate Kubernetes and have engineers with deep Kubernetes knowledge.
  • You need Kubernetes-specific tooling: custom operators, admission webhooks, custom schedulers, or a multi-cloud deployment requirement.
  • You are running more than ~30 microservices and the ECS mental model is becoming hard to scale operationally.

When EKS does not make sense:

  • Your team has no Kubernetes operators. Learning Kubernetes to deploy five Python services adds 6–12 months of ramp-up.
  • Your workload is straightforward containerised HTTP services. ECS Fargate gives you the same result with no cluster management.
  • You are a startup with a product deadline. Kubernetes operational overhead delays shipping.

Decision rule: if nobody on the team has operated Kubernetes in production, use ECS. If the team grows to the point where ECS's lack of network policies, custom schedulers or multi-cluster federation becomes a real blocker — not a theoretical one — then add EKS.

Minimal example Kubernetes resource request estimator

Pythonruns
def k8s_node_estimate(replicas, cpu_m, mem_mib, node_cpu_m=2000, node_mem_mib=4096):
    pods_per_node = min(node_cpu_m // cpu_m, node_mem_mib // mem_mib)
    pods_per_node = max(1, pods_per_node)
    return -(-replicas // pods_per_node)   # ceiling division

for replicas, cpu, mem in [(10, 250, 512), (10, 1000, 2048), (50, 250, 512)]:
    nodes = k8s_node_estimate(replicas, cpu, mem)
    print(f"replicas={replicas} cpu={cpu}m mem={mem}Mi -> min_nodes={nodes}")
Output
replicas=10 cpu=250m mem=512Mi -> min_nodes=2
replicas=10 cpu=1000m mem=2048Mi -> min_nodes=5
replicas=50 cpu=250m mem=512Mi -> min_nodes=7

In production Cluster capacity planner for a Python microservice fleet

Pythonruns
from dataclasses import dataclass


@dataclass
class Service:
    name: str
    replicas: int
    cpu_m: int     # millicores
    mem_mib: int   # MiB


def fleet_capacity(services, node_cpu_m=4000, node_mem_mib=8192):
    total_cpu = sum(s.replicas * s.cpu_m for s in services)
    total_mem = sum(s.replicas * s.mem_mib for s in services)
    nodes_by_cpu = -(-total_cpu // node_cpu_m)
    nodes_by_mem = -(-total_mem // node_mem_mib)
    nodes = max(nodes_by_cpu, nodes_by_mem)
    print(f"fleet: {len(services)} services, {sum(s.replicas for s in services)} replicas")
    print(f"  total CPU: {total_cpu}m  ({nodes_by_cpu} nodes by CPU)")
    print(f"  total MEM: {total_mem}Mi  ({nodes_by_mem} nodes by MEM)")
    print(f"  minimum nodes needed: {nodes}")
    print(f"  add 30% buffer: {round(nodes * 1.3)}")


fleet_capacity([
    Service("settlements-api",   3, 500, 1024),
    Service("transform-worker",  5, 250,  512),
    Service("report-generator",  2, 1000, 2048),
    Service("audit-logger",      2, 100,  256),
])
Output
fleet: 4 services, 12 replicas
  total CPU: 4950m  (2 nodes by CPU)
  total MEM: 10240Mi  (2 nodes by MEM)
  minimum nodes needed: 2
  add 30% buffer: 3
Scenario A

In your world

The situation

Your team lead says "the new product company job descriptions all mention Kubernetes, so let's add EKS to our stack to build experience." There are currently five services, all containerised, all running on ECS Fargate.

What actually breaks

Adding EKS to a five-person team with no Kubernetes operators means someone will spend 30–50 % of their time on cluster maintenance: version upgrades, node group patching, add-on compatibility, PodDisruptionBudget tuning, Karpenter or Cluster Autoscaler debugging. That time comes out of product work. The first cluster version end-of-life cycle arrives in 14 months and requires a controlled upgrade — a task that is non-trivial for a team that is still learning the basics.

What you'd do on Monday

Build Kubernetes knowledge on the side, not on the critical path. For the team's five services, keep ECS Fargate. For personal skill-building, run a local cluster with minikube or kind, deploy the same services, and practice the operational tasks (upgrades, autoscaling, network policies) in a low-stakes environment. When the team grows to 20+ engineers with two dedicated platform engineers, evaluate EKS. The job descriptions mention Kubernetes because product companies already have platform teams; they are not expecting a new hire to build the cluster from scratch.

Scenario B

Imagine you work at Duolingo

50 microservices on EKS

The numbers

Illustrative: 50 microservices × 3 replicas = 150 pods at baseline. EKS cluster fee (check current EKS pricing for your region). Per-node: a team managing 20 c5.xlarge nodes spends ~30 person-hours per year on Kubernetes version upgrades plus incident response (illustrative based on common industry benchmarks; check your own team's experience). A dedicated platform engineer costs ~$100k–$150k/year; the EKS operational tax should be assessed against this. With ECS Fargate for the same 50 services, there are no nodes to upgrade and no cluster version upgrade cycle.

What the Scenario-A answer does here

Copying a Duolingo-scale architecture onto a 5-engineer team works as inspiration but not as a blueprint. Duolingo has a platform engineering team. The tool that fits their scale causes net harm at your scale.

What flips at this scale

The flip is not "use Kubernetes" but "build platform experience without paying the operational tax on your critical path". Use managed services (ECS, Fargate, RDS) for production and Kubernetes locally for learning. When your team reaches the scale where ECS's abstractions genuinely limit you — not theoretically but measurably — revisit EKS with a dedicated platform team.

The trade-off you'd defend

I would defend ECS Fargate for a team of under 15 engineers with no dedicated platform operator, accepting that you won't have Kubernetes-native features like admission webhooks or custom operators in exchange for shipping product without a cluster upgrade rota.

Illustrative; not Duolingo-published figures. Cost of Kubernetes operations is widely discussed in the industry; check your own team's measured overhead.

Gotchas

Adding EKS to learn Kubernetes breaks your production SLA

Symptom

A Kubernetes version goes end-of-support; nobody on the team has done a cluster upgrade before; the upgrade breaks a VPC CNI version and services lose connectivity for two hours.

Cause

Kubernetes version upgrades require careful ordering: control plane first, then node groups, then add-ons, then workloads. An unplanned upgrade on a production cluster is a high-risk change.

Fix

Never introduce EKS for learning purposes on a production cluster. Run a dedicated staging cluster for Kubernetes training. Use a runbook for every version upgrade, test on staging first.

EKS Pod resource requests set too low

Symptom

Node CPU is at 90 % even though all Pods report "CPU throttled". Horizontal Pod Autoscaler does not trigger.

Cause

HPA scales on CPU utilisation relative to requests. If requests are set to 100m but a pod routinely uses 500m, utilisation is 500 %, and HPA scales aggressively. Nodes run out of capacity.

Fix

Profile actual CPU and memory usage in staging with kubectl top pods. Set requests at ~80 % of typical usage and limits at ~150 %. Run VPA (Vertical Pod Autoscaler) in recommendation mode to suggest values.

Using :latest tag with a Kubernetes deployment

Symptom

After a failed deployment, kubectl rollout undo restores the previous deployment spec but the Pod still runs the broken image because :latest pulled a new image on the previous version's nodes.

Cause

:latest is mutable; rolling back the spec does not guarantee the previous image if the registry has been overwritten.

Fix

Use immutable tags (commit SHA or semantic version). Kubernetes rollout undo then reliably restores the previous image.

Tricky questions

Does AWS manage the Kubernetes control plane on EKS?
Most people say

No, you manage everything on EKS.

Actually

AWS manages the API server, etcd, controller manager and scheduler — the control plane. You manage worker nodes (unless using Fargate profiles), add-ons and workloads. You still pay for and manage cluster version upgrades.

Is EKS cheaper than ECS because you have more control?
Most people say

EKS gives more control so it must be more efficient and cheaper.

Actually

EKS has a per-cluster fee plus node costs. ECS with Fargate has no cluster fee; you pay only for task vCPU-seconds. For small teams and workloads, ECS Fargate is typically cheaper and operationally lighter. EKS only becomes cost-competitive when you pack many workloads onto shared EC2 nodes with careful bin-packing, which itself requires platform engineering effort.

Scenario question

You're handed a team of 8 engineers who want to adopt Kubernetes to match the job description of their target companies. You have five production services on ECS Fargate, a daily deploy cadence and a two-person on-call rotation. What do you recommend?

Model answer

I would not migrate production to EKS yet. The operational tax of a Kubernetes cluster — version upgrades, add-on management, node group patching — requires at least one dedicated platform engineer who is already expert. On a team of eight with a two-person on-call, that tax falls on engineers who are also shipping product. Instead, I would set up a learning cluster in a non-production account using eksctl or Terraform, and have each engineer go through the EKS Getting Started guide, deploy the five services there, simulate a version upgrade and practice kubectl operations. Meanwhile, the five production services stay on ECS Fargate. When the team grows to 20+ and the organisation hires a platform engineer, migrate production to EKS with someone at the wheel who has already done it. The trade-off is slower Kubernetes interview readiness in exchange for zero production incidents from a half-understood cluster upgrade.

Interview corner

CommonWhat is the difference between ECS and EKS?
“I'd say ECS is AWS's own container orchestrator with deep native AWS integration; EKS is managed Kubernetes. ECS is simpler to operate, and AWS handles more of the plumbing. EKS gives you the full Kubernetes ecosystem but you own the cluster upgrade cycle and add-on management. For most Python backend services, ECS Fargate is the better starting point.”
TrickIs Kubernetes always better than ECS at scale?
“Not necessarily. ECS on Fargate scales to thousands of tasks with no cluster management. Kubernetes adds features like custom operators and admission webhooks, but also adds platform engineering overhead. I'd use EKS when the team has Kubernetes expertise and the workload genuinely needs Kubernetes-native capabilities, not as a default.”
Experience checkWhat does AWS manage on EKS vs what you manage?
“I'd say AWS manages the control plane: API server, etcd, scheduler, controller manager. I manage worker nodes unless I use Fargate profiles, plus add-ons like VPC CNI and CoreDNS, and the cluster version upgrade cadence. That last one is the most operational burden: new Kubernetes versions are released regularly and you must upgrade every 14 months or face end-of-support.”
DesignA 50-microservice platform team wants to migrate from ECS to EKS. How do you evaluate the decision?
“I'd look for a concrete blocker, not a preference: do they need admission webhooks, custom schedulers, or multi-cluster federation that ECS genuinely can't do? I'd also check whether they already have platform engineers who have run Kubernetes in production, because the migration cost isn't the cutover, it's the ongoing version-upgrade and add-on maintenance cycle. If the blocker is real and the staffing is there, I'd migrate service by service behind the existing ALB, not all at once. If the only driver is 'Kubernetes looks better on a resume', I'd say no.”

Memory hook

EKS is buying the restaurant building; ECS is using the restaurant chain. Buy the building only if you need a custom kitchen the chain cannot build.

If you remember only one thing:

Do not add EKS until the team has a dedicated platform engineer. ECS Fargate covers 90 % of use cases with 10 % of the operational overhead.

Self-check

  1. What does AWS manage on EKS and what do you manage?

    AWS manages the control plane. You manage worker nodes (unless using Fargate profiles), add-ons and cluster version upgrades.

  2. Why is the per-cluster EKS fee relevant for small teams?

    For a team running 3–5 services, the per-cluster hourly fee plus the platform engineering cost of Kubernetes operations can exceed the equivalent ECS Fargate cost. Check current pricing.

  3. What is the Kubernetes version support window on EKS?

    Approximately 14 months per minor version (check the current EKS documentation). You must upgrade within that window or face running on an end-of-support version.

Prove it

Calculate the minimum EKS node group size for the four-service fleet from the production block (settlements-api 3 replicas, transform-worker 5 replicas, report-generator 2 replicas, audit-logger 2 replicas) using a node type of your choice. Include a 30 % buffer for Kubernetes system pods and the Karpenter/Cluster Autoscaler overhead. Show both the compute-bound and memory-bound calculations and choose the larger. Compare the result to running the same fleet on ECS Fargate (which needs no node calculation).