AWS compute choice
Why you care
When a recruiter for a product company says "walk me through your compute choices on AWS", your answer is the first filter. The question is not about reciting service names; it is about showing you can map workload properties — duration, traffic shape, state, operational budget — to the right abstraction. You are currently deploying Python ETL scripts on Jenkins that write to S3 and Postgres. That workload sits exactly in the Lambda-versus-Batch decision space, and every interviewer at a product company will probe it. Interviewers ask "Lambda or Fargate?" because the answer tells them whether you understand operational cost, not just AWS pricing.
Explain like I'm 5
Imagine you need staff to answer phone calls. Lambda is a freelancer you ring when needed — they pick up, do the job, hang up, and you pay per call. Fargate is a staffing agency running a small team of contractors: always available during their shift, you decide what they do but not what workstations they use. EC2 is hiring full-time employees: you own everything, from the desk to the payslip. Batch is a factory shift: you queue work orders, the factory opens when there are enough of them, and workers go home when the queue drains. Kubernetes is buying and running your own factory building — only worth it if you already know factory management.
Core concept
Five services, five operating models.
AWS Lambda executes event-driven functions. Duration limit: 900 seconds (15 minutes) per invocation — a hard ceiling, check the current quota page for any account-specific detail. Memory: 128 MB to 10,240 MB. No persistent local disk beyond /tmp (512 MB to 10,240 MB). You pay per GB-second. Use Lambda when: tasks are short, triggered by events (S3, SQS, API Gateway, EventBridge), and you want zero server management. Do not use Lambda when: duration exceeds 15 minutes, the workload needs a GPU or a specific kernel build, or a cold start would break your latency contract.
AWS Fargate runs containers via ECS or EKS without you provisioning EC2 instances. Tasks can run indefinitely. You pay per vCPU-second while a task is running. Use Fargate when: the workload is containerised, duration can exceed 15 minutes, or you need a runtime Lambda does not support. Do not use Fargate when: you need EC2-level hardware (GPU, local NVMe, bare metal) or want maximum compute density.
Amazon EC2 is a full virtual machine. You manage patching, scaling, capacity planning and OS. Use EC2 when: you need specific hardware families, run jobs longer than a Fargate task would tolerate, need kernel customisation, or want to control cost tightly with Reserved Instances or Spot. Do not use EC2 when: you can get the same result from Lambda or Fargate; the operational overhead is real.
AWS Batch is a managed job scheduler. You define a compute environment, a job queue, and job definitions; Batch provisions compute as jobs arrive and scales to zero when the queue drains. Use Batch when: workloads are parallelisable, retryable, and you want job dependencies and retry logic handled without writing your own orchestration.
Amazon EKS runs Kubernetes on AWS. Use EKS when: you already operate a Kubernetes platform and need its ecosystem. Do not use EKS when: the team has no Kubernetes expertise — ECS gives you 90 % of the benefit with 10 % of the operational overhead.
Decision rule: event-driven under 15 minutes → Lambda. Containerised, longer-running, or needs a specific runtime → Fargate. Specific hardware, long-lived steady-state, maximum cost control → EC2. Parallelisable batch ETL with retries → Batch. Already on Kubernetes → EKS, otherwise skip it.
Minimal example Four workloads, one decision function
def pick_compute(duration_s, stateful, traffic, packaging):
if duration_s > 900:
return "Batch" if traffic == "batch" else "EC2/Fargate"
if stateful:
return "EC2/Fargate+EFS"
if packaging == "container":
return "Fargate"
if traffic == "batch":
return "Lambda/Batch"
return "Lambda"
print(pick_compute(10, False, "bursty", "zip")) # S3 trigger
print(pick_compute(1800, False, "batch", "zip")) # 30-min ETL
print(pick_compute(300, False, "batch", "container")) # containerised job
print(pick_compute(60, True, "steady", "zip")) # stateful workerLambda
Batch
Fargate
EC2/Fargate+EFSIn production Audit five workloads against the decision framework
from dataclasses import dataclass
@dataclass
class Workload:
name: str
duration_s: int
stateful: bool
traffic: str # bursty | batch | steady
packaging: str # zip | container
def pick(w: Workload) -> tuple[str, str]:
if w.duration_s > 900:
svc = "Batch" if w.traffic == "batch" else "EC2/Fargate"
return svc, "duration > 15 min"
if w.stateful:
return "EC2/Fargate+EFS", "stateful"
if w.packaging == "container":
return "Fargate", "container image"
if w.traffic == "batch":
return "Lambda/Batch", "batch trigger"
return "Lambda", "event-driven"
jobs = [
Workload("S3-triggered transform", 30, False, "bursty", "zip"),
Workload("Nightly settlements ETL", 3600, False, "batch", "zip"),
Workload("Report container job", 600, False, "batch", "container"),
Workload("Session cache warmer", 120, True, "steady", "zip"),
Workload("Payment API handler", 5, False, "bursty", "zip"),
]
for j in jobs:
svc, reason = pick(j)
print(f"{j.name:<30} {svc:<20} ({reason})")S3-triggered transform Lambda (event-driven)
Nightly settlements ETL Batch (duration > 15 min)
Report container job Fargate (container image)
Session cache warmer EC2/Fargate+EFS (stateful)
Payment API handler Lambda (event-driven)See it happen The compute decision tree
- Every workload enters at the same first question: does it run longer than 15 minutes?
- Long-running batch jobs belong in AWS Batch; other long-running work goes on EC2 or Fargate.
- Short work hits the second question: does the job need persistent local disk or sticky state?
- Stateful jobs need EC2 or Fargate with EFS. Stateless jobs face one more question.
- Container image? Use Fargate. No container? Lambda is the default for short event-driven work.
In your world
The situation
Your Jenkins job runs a Python ETL script that reads from S3, transforms, and writes to Postgres. It takes 4 minutes and is triggered by a cron at 2 AM. Someone on your team suggests "just put it in Lambda".
What actually breaks
Lambda has a hard limit of 900 seconds (15 minutes) per invocation (documented). At 4 minutes and 400,000 rows today, you are well inside the limit, but the S3 file triples to 1.2 million rows in six months at the same growth rate; the job now needs roughly 12 minutes and breaches the ceiling mid-load. Lambda terminates the environment with no exception raised in your code, so the job's own logs show no error — but Postgres ends up with only the ~800,000 rows that were committed before the cutoff, and finance does not notice the missing 400,000 rows until a downstream reconciliation report fails to balance days later. Also, Lambda runs inside a VPC only if you configure one; a naive deployment cannot reach your private Postgres instance.
What you'd do on Monday
The right home is AWS Batch (job definition + queue + compute environment on Fargate) or Fargate directly. Both support jobs over 15 minutes, both can live in your VPC, and Batch adds retry-on-failure for free. Lambda is correct only for the short event-driven pieces — like triggering the Batch job when the S3 file lands.
import boto3
def handler(event, context):
bucket = event["Records"][0]["s3"]["bucket"]["name"]
key = event["Records"][0]["s3"]["object"]["key"]
boto3.client("batch").submit_job(
jobName="settlements-etl",
jobQueue="etl-queue",
jobDefinition="settlements-etl-job",
containerOverrides={"environment": [
{"name": "S3_BUCKET", "value": bucket},
{"name": "S3_KEY", "value": key},
]},
)Imagine you work at CRED
The numbers
Illustrative: 50k calls/day ≈ 0.6 calls/s average; burst to ~500 calls/s during peak rewards cycle. Lambda concurrency at burst: 500 × 0.05 s average duration = 25 concurrent executions. Lambda cost at 50k calls with 256 MB and 50 ms average: ~50k × 256/1024 × 0.05 s = 625 GB-seconds ≈ $0.01/day. Three t3.micro On-Demand instances 24/7 ≈ $30/month. Lambda wins until sustained throughput exceeds ~30k calls/minute, where EC2 flat rate becomes competitive.
What the Scenario-A answer does here
Lambda looks right and is cheap at this scale. The issue is not cost — it is the 15-minute ceiling. CRED's monthly statement job consolidates 12 months of transaction history per user. If a single user has 18,000 transactions and each takes 3 ms to process, the job takes 54 seconds: fine for Lambda. But the batch that processes 1 million users in parallel is still fine as individual Lambda functions, until a p99 user has 200k transactions and the function hits 600 seconds — still inside the limit, but only just.
What flips at this scale
At product scale you split: the per-user transform is a Lambda function (under 1 minute), and the fan-out orchestration is either a Step Functions state machine or an AWS Batch array job — not one giant Lambda. The flip is that you design for composability before you hit the edge.
The trade-off you'd defend
I would defend Lambda for the per-user transform and Batch for the bulk fan-out, trading a slightly more complex deployment (two compute targets) for the ability to scale each independently and stay well within service limits.
Illustrative numbers chosen for the arithmetic; not CRED-published figures.
Lambda vs Fargate vs EC2 vs Batch
| Axis | Lambda | Fargate | EC2 | Batch |
|---|---|---|---|---|
| Max duration | 15 min (900 s) | Unlimited | Unlimited | Unlimited |
| Cold start | Yes (100 ms–several s) | Yes (30–60 s typical; longer for large image pulls, see ECS and Fargate in operation) | No (always running) | Yes (scale-out) |
| Operations overhead | None | Low (no patching) | High (AMI, patches, ASG) | Low (scheduler managed) |
| Pricing model | Per GB-second | Per vCPU-second | Per instance-hour | Per vCPU-second (Fargate) or instance-hour (EC2) |
| Best for | Event-driven, short tasks | Containerised, variable load | Steady-state, custom HW | Parallel ETL, retryable jobs |
Gotchas
Lambda silently terminates at the 15-minute wall
A long-running function produces no error, no exception, and no logs after the 14:59 mark. The job appears to succeed.
Lambda hard-terminates the execution environment at the configured timeout (max 900 s). No exception is raised inside your code; the environment just stops.
Set a timeout in your Lambda configuration that is lower than your measured p99 duration, so you get a visible Task timed out error. For jobs that can exceed 15 minutes, move them to Batch or Fargate.
Lambda inside a VPC without a NAT gateway cannot reach S3
A Lambda function configured with a VPC can reach Postgres on the private subnet but fails to connect to S3 (Connection refused or timeout on port 443).
Traffic from a VPC-attached Lambda exits through the VPC's ENI; without a NAT Gateway or an S3 Gateway Endpoint, it has no route to the public S3 API.
Add an S3 Gateway VPC Endpoint (free) to your VPC's route tables. That routes S3 traffic through AWS's backbone, not the internet.
Choosing Lambda for a workload that holds a database connection open
Postgres max_connections exhaustion during load testing, even though no single function takes long.
Each Lambda execution environment holds one database connection in module-level scope; at 500 concurrent executions you have 500 open connections.
Use RDS Proxy or PgBouncer in transaction mode between Lambda and Postgres. This is covered in Parameter groups and connection behavior.
Tricky questions
Lambda's per-request pricing is always cheaper.
At sustained high throughput (millions of requests per day), EC2 Reserved Instances or Fargate Savings Plans can cost significantly less. The break-even depends on your request rate, duration, and memory. Lambda's advantage is at low or bursty traffic.
Fargate is AWS's container service.
Amazon ECS is the container orchestrator; Fargate is the serverless compute backend for ECS (and EKS). You can run ECS tasks on EC2 launch type (you manage instances) or Fargate launch type (AWS manages them). ECS on Fargate is the combination.
Scenario question
You're handed a Python ETL pipeline that reads a 2 GB CSV from S3, joins it with a Postgres table, and writes results back to S3. The script takes 8 minutes and runs once a night. The current setup is an EC2 t3.medium that stays running all day. Your task: propose a cheaper and more resilient replacement.
Model answer
I would move this to AWS Batch with a Fargate compute environment. The job runs once a night, so paying for a t3.medium ($30+/month) all day is pure waste. A Batch job definition points at a Docker image containing the Python script; the compute environment spins a Fargate task when the job starts and terminates it when it finishes — you pay for 8–10 minutes of vCPU/GB-second instead of 24 hours. I would add maxAttempts: 3 to the job definition so transient S3 or Postgres failures trigger automatic retries. The trade-off is a longer start-up time (30–60 s Fargate cold start vs immediate EC2 response), but a nightly job can absorb that. I would also add a CloudWatch alarm on job failure to replace the current "check the server in the morning" pattern.
Interview corner
CommonLambda or Fargate: how do you choose?
TrickCan Lambda be used for long-running batch jobs?
Experience checkHow did your current ETL scripts run, and what would you change?
DesignYour team is debating Lambda vs ECS for a new microservice. What do you say?
Memory hook
Lambda is the freelancer, Fargate is the contractor team, EC2 is the full-time hire, Batch is the factory shift.
Ask duration first. Over 15 minutes means no Lambda.
Self-check
What is Lambda's maximum invocation duration by default?
900 seconds (15 minutes), a hard ceiling. Check the quota page for the current figure before relying on it in production.
What is the key operational difference between ECS on EC2 and ECS on Fargate?
With EC2 launch type you provision and patch the EC2 instances. With Fargate, AWS manages the underlying compute.
When would you choose AWS Batch over Lambda for an ETL job?
When the job exceeds 15 minutes, needs parallel workers across many records, requires retry-on-failure at the job level, or the job duration varies widely and a per-invocation Lambda would be wasteful.
Write the pick_compute function with all five compute options (including EKS). Add a sixth case: a stateless 45-minute containerised job triggered nightly. Show the recommendation and confirm it is not Lambda. Then extend the function to flag any EC2 recommendation as "review Spot savings" and rerun the audit table.