Shelf · Core Eng · Chapter 2

Chapter 2: Linux & shell survival

Every ETL job you have run on a shared Linux box already depends on permissions, process signals, pipe semantics and a scheduler you have never had to configure yourself. This chapter turns that borrowed knowledge into a method: inspect before you change, read exit status all the way through a pipeline, diagnose from symptom to cause with the right tool, and write scripts that fail loudly and clean up after themselves.

Difficulty
Priority
ce-02-01Card 1 of 6

Navigate files and permission boundaries

EasyP2 · Very likelyKnow cold12 min
filesystempermissionsownershipumask
Needs: nothing. Start here.

Why you care

A wrong permission bit is the most common reason a job that works on your laptop fails silently, or leaks, on a shared build box: the pipeline user can read the repo but not the .pgpass a teammate copied back from a backup, or a fresh clone inherits a umask that leaves credentials group-readable. You do not get a helpful exception for this. You get Permission denied on someone else's line 40, or worse, no error at all until a security scan flags it three weeks later. Interviewers ask "what does 644 mean" and "what does umask do" not to test trivia, but because a candidate who can read ls -l output and reason about who can touch a file has clearly operated a real multi-user box, not just their own laptop. Everything else in this chapter assumes you can answer "who owns this, and who can write to it" before you touch it.

Explain like I'm 5

A path is an address in an apartment building, and every door has three keyrings: one for the resident (owner), one for their flat's family (group), and one for everyone else in the building (other). Read means you can see what's inside, write means you can rearrange the furniture, execute means you can use the door at all — walk through it, or for a directory, walk through it to reach the rooms behind it. When the building is constructed, the landlord's default lock policy is stamped onto every new door the moment it is installed. That policy is the umask. Changing the landlord's policy tomorrow does not reach into the door that was fitted yesterday.

Core concept

  • A path is either absolute (starts at /) or relative to your current directory (pwd); . and .. are the current and parent directory, and a symlink is a signpost to another path, while a hard link is a second name for the same underlying file.
  • Ownership is a user and a group, shown by ls -l and changed with chown user:group path (only root, or the owner handing off to a group they belong to, can do this).
  • Permissions are three triplets, owner/group/other, each rwx. ls -l shows them as text (-rw-r--r--); stat and chmod use the octal form, where each digit is r(4)+w(2)+x(1): 750 is owner all, group read+execute, other nothing.
  • umask is a bitmask subtracted from the kernel's own defaults at creation time: new files start at 666, new directories at 777, and the actual mode is default & ~umask. A umask of 022 turns a new file into 644 and a new directory into 755.
  • On a directory, execute means something different from a file: it lets you traverse into it and reach entries by name. Read on a directory only lets you list names; without execute you cannot stat or open anything inside, even ones you can "see".

Decision rule: use stat when a script needs to compare an exact octal mode; use ls -l for a fast human read across many files. Change ownership only when the process that must access the file genuinely runs as a different user; otherwise change only the mode. Never chmod -R 777 to make an error go away — that is a diagnosis substitute, it grants write to every user on the box, and it is exactly the kind of change a security review calls out by name. Inspect first, always: ls -l or stat before any chmod or chown.

Under the hood

umask is not consulted by chmod, and chmod does not touch the process umask. The kernel applies umask exactly once, inside the open(2) and mkdir(2) syscalls that create a new inode, computing mode & ~umask at that instant. Once the inode exists, its mode lives on disk and is changed only by an explicit chmod(2) call. That is why umask 077 run in your shell right now has zero effect on a file created an hour ago: there is no syscall left to consult it. It only shapes files your shell (or a script inheriting its umask) creates from this point forward.

Minimal example Umask arithmetic and the resulting rwx string

PythonSame formula the kernel applies at file/directory creationruns
import stat

def new_file_mode(umask_val):
    return 0o666 & ~umask_val

def new_dir_mode(umask_val):
    return 0o777 & ~umask_val

for umask_val in (0o022, 0o002, 0o077):
    fmode = new_file_mode(umask_val)
    dmode = new_dir_mode(umask_val)
    print(f"umask {oct(umask_val)}: file {oct(fmode)} {stat.filemode(stat.S_IFREG | fmode)}  "
          f"dir {oct(dmode)} {stat.filemode(stat.S_IFDIR | dmode)}")
Output
umask 0o22: file 0o644 -rw-r--r--  dir 0o755 drwxr-xr-x
umask 0o2: file 0o664 -rw-rw-r--  dir 0o775 drwxrwxr-x
umask 0o77: file 0o600 -rw-------  dir 0o700 drwx------

In production Flag secret-named files readable outside their owner

On the real build box this manifest would come from stat.S_IMODE(os.stat(p).st_mode) per file; this sandbox has no real Unix permission bits, so the audit logic runs against a provisioning manifest instead, which is exactly the shape of a pre-deploy permission gate.

PythonA pre-deploy check that would have caught the leak in Scenario Aruns
import stat

provisioned = {
    ".pgpass": 0o600,
    "config/app.yaml": 0o644,
    "orders_dump.csv": 0o664,
    "settlement_secrets.env": 0o644,
}

SECRET_HINTS = ("secret", ".pgpass", "credential", "password")
LEAK_BITS = stat.S_IRGRP | stat.S_IROTH | stat.S_IWGRP | stat.S_IWOTH

violations = [
    (name, oct(mode))
    for name, mode in provisioned.items()
    if any(hint in name.lower() for hint in SECRET_HINTS) and mode & LEAK_BITS
]

print("checked:", sorted(provisioned))
print("violations:", violations)
Output
checked: ['.pgpass', 'config/app.yaml', 'orders_dump.csv', 'settlement_secrets.env']
violations: [('settlement_secrets.env', '0o644')]
Scenario A

In your world

The situation

Your ETL job's Jenkins agent runs as svc-etl on a shared box. A teammate re-clones the repo for a hotfix and restores .pgpass and settlement_secrets.env from an old backup tarball, which lands at the shell's default umask instead of the mode they were created with.

What actually breaks

Jenkins stays green — nothing in the job ever checked file modes. Three weeks later a security scan run by another team flags settlement_secrets.env as group-readable and asks who else on the box could have read it. You cannot answer, because nothing recorded it.

What you'd do on Monday

Add a startup check that refuses to run instead of trusting the filesystem, and correct the mode on the spot:

ShellFail fast on a wrong secrets mode (needs a real Linux filesystem with POSIX permission bits)
for f in .pgpass settlement_secrets.env; do
  mode=$(stat -c '%a' "$f")
  if [ "$mode" != "600" ]; then
    echo "refusing to run: $f is $mode, expected 600" >&2
    chmod 600 "$f"
    exit 1
  fi
done
Scenario B

Imagine you work at DigitalOcean

50,000 build-agent droplets provisioned monthly

The numbers

Illustrative: a platform provisioning 50,000 fresh build-agent droplets a month from a base image. If a bad image variant ships with umask 002 instead of 022 for even 5% of that month's boots while two image versions are active during a rollout, that is 2,500 machines a month with the same secrets directory left group-writable — invisible until a weekly fleet scanner catches it, by which time roughly a week's worth, ~580 machines, have already run with the wider mode.

What the Scenario-A answer does here

The Scenario A fix (a startup check that refuses to run) still catches this per machine, but "per machine" is the problem: relying on 50,000 individual boot scripts to each notice and self-correct means you find out from the scanner's weekly report, not before the machine ever took traffic.

What flips at this scale

At this scale the umask and every secrets-file mode belong in the base image and are asserted once at boot, so a bad image variant fails cloud-init loudly on machine one instead of silently drifting on machine 2,500. The fleet scanner becomes a second, independent layer that proves the control worked, not the primary control itself.

The trade-off you'd defend

I would defend baking the mode into the image and asserting it at boot over trusting per-machine chmod, accepting one more boot-time check that can itself fail, in exchange for every one of 50,000 machines starting identical and a bad rollout failing on machine one instead of machine 2,500.

Illustrative figures chosen to make the arithmetic concrete; not DigitalOcean-published numbers.

Gotchas

An unquoted path splits into more than one argument

Symptom

chmod 600 $secret_path either errors with "No such file or directory" for two garbled paths, or silently changes the wrong file, when the path contains a space.

Cause

Unquoted variable expansion in the shell undergoes word splitting on whitespace before the command ever sees it.

Fix

Quote every path expansion — chmod 600 "$secret_path" — and use arrays (for f in "${files[@]}") when you genuinely have more than one path.

Umask changes do not reach files that already exist

Symptom

umask 077 in your shell, then ls -l secrets.env still shows -rw-r--r--.

Cause

The kernel consults umask only inside the open/mkdir syscall that creates a new inode; it never revisits an existing one.

Fix

chmod the existing file explicitly. Use umask to fix files created from now on, and recreate a file if you need it to pick up a new umask.

A "read-only" check misses the write bit

Symptom

A secrets audit that only checks "is it world-readable" passes a file that is 664 — not readable by other, but writable by the whole group.

Cause

Leak checks that test a single bit instead of the full triplet miss the other two ways a boundary can be crossed.

Fix

Check group and other read and write together, as the production example does with LEAK_BITS.

Tricky questions

If I chown a file away to another user, do I keep my read access?
Most people say

Yes, since I created the file originally.

Actually

No. Ownership determines which triplet applies to you. Once a file is chowned to another user, your access comes only from the group or other triplet, unless you happen to be a member of the new owning group.

Does execute on a directory mean the same thing as execute on a file?
Most people say

Yes, it means you can run it.

Actually

On a file, execute means "can be run as a program." On a directory, it means "can be traversed" — you can cd into it and open files inside by name. A directory with read but no execute lets you list names with ls, but you cannot stat or open anything you see there.

Scenario question

You're handed a nightly script that today fails with PermissionError: [Errno 13] Permission denied on /data/incoming/vendor.csv, which worked yesterday. ls -l shows -rw-r----- 1 vendor-svc data-team vendor.csv, and your job runs as etl-svc, which is not in data-team. What do you check, and what do you actually do?

Model answer

I would run id etl-svc to see its groups, and ls -ld on the parent directories too, because a denied traversal on a directory looks identical to a denied read on the file. If etl-svc genuinely needs read access, I would not chmod o+r the vendor's file — that grants read to every user on the box, including ones with no reason to see vendor data. I would ask an admin to add etl-svc to data-team, a one-line, auditable change, or route the handoff through a location with a group made for exactly this. The trade-off is speed versus blast radius: widening the file's mode fixes the job in ten seconds and quietly widens access forever; fixing group membership takes a request, but the access boundary stays intentional.

Interview corner

CommonWhat does chmod 750 mean?
“I'd read it right to left in groups of three: owner rwx is 7, group r-x is 5, other is 0. So the owner can do anything, the group can read and traverse or execute, and everyone else has no access at all.”
TrickYou just ran umask 077. Does that fix the permissions on files you created earlier?
“I'd say no — umask only affects files created after that point, because it's applied inside the open or mkdir syscall at creation time. Anything that already exists needs an explicit chmod.”
Experience checkTell me about a permissions bug you've actually hit.
“I've seen a fresh clone bring a .pgpass file back from a backup at 644 instead of 600, and it sat like that until a scanner flagged it weeks later. Now our setup script asserts the mode of every secrets file at startup and refuses to run if it's wrong, instead of trusting whatever the filesystem happens to have.”
DesignHow would you stop 50,000 machines in a fleet from drifting on default file permissions?
“I'd bake the umask and any special file modes into the base image, assert them at boot so a bad image fails loudly instead of drifting silently, and keep a fleet-wide scanner as an independent second layer rather than the only control.”

Memory hook

Every new door in the building is stamped with the landlord's lock policy the moment it's built. Changing the policy tomorrow doesn't reach the door fitted yesterday.

If you remember only one thing:

Inspect the actual owner and mode before you touch either — never chmod blind.

Self-check

  1. What does umask 022 do to a newly created file, and why doesn't it touch files created earlier?

    It computes 0o666 & ~0o022 = 0o644; it only applies at creation time, inside open/mkdir, so it can never reach a file that already exists.

  2. Why can a directory be readable but still block access to files inside it?

    Read on a directory only lets you list names. Traversal — opening or stat-ing something inside — needs the execute bit too.

  3. Name the exact syscall-level moment umask is applied.

    Inside open(2) or mkdir(2), as mode & ~umask, when the inode is created.

Prove it

Extend the production example's manifest to five files, including at least one secrets-style name with a leaking mode and one with a correct mode. Run the audit, then change the leaking file's mode to 600 in the manifest and rerun. Paste both printed violation lists and confirm the second one is exactly one shorter.

ce-02-02Card 2 of 6

Explain processes, jobs and signals

MediumP2 · Very likelyKnow cold12 min
processessignalsexit-statusjob-control
Needs: nothing. Start here.

Why you care

"Green build, dead job" happens more often from a misunderstood signal than from a bug in your code: a deploy script sends SIGTERM, your worker doesn't handle it, the orchestrator gets impatient and sends SIGKILL, and the in-flight row your worker was writing is now half-committed with no log line to explain it. You will not get a clean stack trace for this — you get an exit code, and only one of exit 137 or exit 143 tells you whether the process was killed outright or asked nicely first. Interviewers ask "what's the difference between SIGTERM and SIGKILL" because the answer predicts whether you design for graceful shutdown or just hope nothing is ever mid-write when the process dies. This card is the vocabulary the rest of the book's shutdown, rollout and incident cards assume you already have.

Explain like I'm 5

A running program is a worker doing a task. A signal is a shout across the room. SIGTERM is "please wrap up and go home" — the worker can hear it, finish the sentence they're writing, and leave tidily. SIGKILL is the building's fire alarm cutting the power: the worker stops mid-sentence, no matter what they were holding. jobs and ps are you glancing around the room to see who's still working and on what; wait is standing at the door until everyone currently inside has actually left.

Core concept

  • A process is a running program with its own memory, its own exit status, and a parent that (usually) collects that status. ps lists processes; top shows them live, ranked by resource use.
  • Job control is your shell's bookkeeping for processes it started: jobs lists them, & backgrounds one, fg/bg move it, and wait blocks until a background job finishes.
  • A signal is an asynchronous notification sent to a process by number or name. kill -TERM pid and kill -9 pid send SIGTERM (15) and SIGKILL (9). SIGHUP (1) traditionally meant "your controlling terminal went away"; SIGINT (2) is what Ctrl-C sends.
  • SIGTERM is a request; SIGKILL is not. A process can catch SIGTERM, run cleanup, and exit on its own terms. SIGKILL (and SIGSTOP) cannot be caught, blocked or ignored by any process — the kernel ends it immediately, with no chance to run a handler.
  • Exit status is a number 0–255. 0 is success. 126 means the shell found the command but could not execute it (commonly a missing execute bit); 127 means the shell could not find the command at all. A process killed by signal n is reported by its parent as exit status 128 + n — 137 for SIGKILL, 143 for SIGTERM.

Decision rule: always send SIGTERM first and give the process a real grace period to finish or checkpoint in-flight work; reach for SIGKILL only when that grace period has passed, and treat anything it interrupted as needing reconciliation, because nothing ran to tell you what was lost.

Under the hood

The 128+n encoding is not magic; it is how the shell reports what wait()/waitpid() returned to it. The kernel packs a child's termination status into one word: if the low byte says the process exited normally, the shell reports that value directly (0–255); if the low byte says it died from a signal, the shell reports 128 + signal_number. The one sharp edge: a process that legitimately calls exit(137) on its own is indistinguishable, purely from the number, from one killed by SIGKILL. You disambiguate with logs, not the exit code alone.

Minimal example Exit codes for a clean exit, a missing command and each signal

Python128 + signal number, plus 126/127 for shell lookup failuresruns
SIGNALS = {"SIGHUP": 1, "SIGINT": 2, "SIGKILL": 9, "SIGTERM": 15}

def exit_status(kind, signum=None):
    if kind == "ok":
        return 0
    if kind == "not_executable":
        return 126
    if kind == "not_found":
        return 127
    if kind == "signal":
        return 128 + signum
    raise ValueError(kind)

for name, num in SIGNALS.items():
    print(name, "->", exit_status("signal", num))
print("clean exit:", exit_status("ok"))
print("found but not executable:", exit_status("not_executable"))
print("command not found:", exit_status("not_found"))
Output
SIGHUP -> 129
SIGINT -> 130
SIGKILL -> 137
SIGTERM -> 143
clean exit: 0
found but not executable: 126
command not found: 127

In production A worker that can finish its current order under SIGTERM but not under SIGKILL

PythonSame escalation an orchestrator would apply to a stuck workerruns
def run_job(items, signal_after=None, signal_kind=None):
    done, in_progress = [], None
    stopping = killed = False
    for i, item in enumerate(items):
        in_progress = item
        if signal_after == i and signal_kind == "SIGKILL":
            killed = True
            break                       # the process dies mid-item, nothing recorded
        done.append(item)                # item fully processed and written
        in_progress = None
        if signal_after == i and signal_kind == "SIGTERM":
            stopping = True               # asked nicely: finish, then stop
        if stopping:
            break
    return done, in_progress, killed


done, lost, killed = run_job(["order-1", "order-2", "order-3"], signal_after=1, signal_kind="SIGTERM")
print("SIGTERM after order-2 starts:", done, "in-flight lost:", lost, "killed:", killed)

done, lost, killed = run_job(["order-1", "order-2", "order-3"], signal_after=1, signal_kind="SIGKILL")
print("SIGKILL after order-2 starts:", done, "in-flight lost:", lost, "killed:", killed)
Output
SIGTERM after order-2 starts: ['order-1', 'order-2'] in-flight lost: None killed: False
SIGKILL after order-2 starts: ['order-1'] in-flight lost: order-2 killed: True

See it happen Escalation from SIGTERM to SIGKILL

orchestratorworkerSIGTERMstill processing order-9grace period (10s) expiredSIGKILLexit 137 (128+9)log lost item + alert
  1. Ask nicely: the handler sets stopping=True and lets the current item finish.
  2. Seconds pass; the in-flight item is not done yet, so nothing has exited.
  3. The orchestrator's own timeout fires because the worker never confirmed it stopped.
  4. No more asking: the kernel ends the process immediately, mid-item.
  5. There was no chance to flush order-9's partial write; that record now needs reconciliation.
  6. SIGKILL never lets the process report what it lost, so the orchestrator, not the worker, has to notice.
Notice: everything before the grace-period timeout is a request the worker could ignore; everything after it is not, and that boundary is exactly where a clean shutdown turns into a reconciliation job.
Scenario A

In your world

The situation

Your nightly load script forks a worker per file and the Jenkins agent, on a build timeout, sends the whole process group SIGTERM and waits five seconds before the box recycles.

What actually breaks

One file is slower than the rest. At second five the box is destroyed regardless — effectively a SIGKILL from outside your script's control — mid-INSERT. Postgres rolls the open transaction back, but your checkpoint file, written just before, now claims more rows are done than actually committed.

What you'd do on Monday

Treat the checkpoint write and the commit as one unit, and make the grace period long enough to matter:

PythonCheckpoint only after commit succeeds, never beforeruns
def process_file(path, commit, write_checkpoint):
    rows_loaded = commit(path)           # returns rows actually committed, or raises
    write_checkpoint(path, rows_loaded)  # only reached if commit succeeded
    return rows_loaded


def fake_commit(path):
    return 4_213


def fake_checkpoint(path, rows):
    print(f"checkpoint: {path} -> {rows} rows")


print(process_file("orders_2026_09_27.csv", fake_commit, fake_checkpoint))
Output
checkpoint: orders_2026_09_27.csv -> 4213 rows
4213
Scenario B

Imagine you work at Netflix

2,000 encode workers recycled every deploy

The numbers

Illustrative: a fleet of 2,000 video-encode workers, each mid-way through a job lasting minutes, gets SIGTERM at deploy time with a 30-second grace period before the orchestrator escalates to SIGKILL. If 5% of workers are mid-encode on a job that needs more than 30 seconds to reach a safe checkpoint, that is 100 workers a deploy losing partial work and needing a full job requeue.

What the Scenario-A answer does here

The Scenario A fix (checkpoint only after commit) still holds per worker, but "give it a fixed 30-second grace period" does not scale across job types: a 3-second parse job and a 90-second encode segment cannot share one timeout without either killing the encode too early or making every deploy wait 90 seconds for the fast jobs too.

What flips at this scale

At this scale the grace period becomes per-job-type, not per-fleet, and the worker itself must expose "how close am I to a safe stopping point" so the orchestrator can decide whether to wait a little longer or requeue immediately instead of guessing. The design changes from "one timeout for everyone" to "the worker reports its own checkpoint distance, and the orchestrator's grace period is a function of that."

The trade-off you'd defend

I would defend a per-job-type grace period plus explicit progress reporting over one global timeout, accepting more coordination code between worker and orchestrator, in exchange for deploys that neither wait unnecessarily nor discard work that was seconds from a safe checkpoint.

Illustrative figures chosen for the arithmetic; not Netflix-published numbers.

Gotchas

kill sends a signal, it does not confirm the process handled it

Symptom

kill -TERM pid; echo done prints "done" immediately, while the target process is still very much running.

Cause

kill only queues the signal for delivery; it returns as soon as the kernel accepts the request, not when the target has acted on it.

Fix

Poll for the process to actually exit (kill -0 pid returning failure, or wait for a child), and only escalate to SIGKILL after a real timeout, not immediately after sending SIGTERM.

A process that ignores or blocks SIGTERM never stops on its own

Symptom

SIGTERM is sent every deploy, the process logs nothing, and it is still in ps output minutes later.

Cause

A process can install a handler that does nothing, or it can be in a state (blocked on an uninterruptible syscall) where the signal is queued but not yet delivered.

Fix

Confirm graceful shutdown is actually implemented and tested, and always have an escalation path to SIGKILL with a bounded timeout as the backstop.

Exit code 137 could be SIGKILL, or could just be the number 137

Symptom

A job "exits with code 137" and everyone assumes it was killed for memory.

Cause

128 + signal_number collides with a process that simply calls exit(137) for its own reasons; the number alone does not prove a signal was involved.

Fix

Check the actual cause — dmesg/OOM killer logs, orchestrator events, or your own logging around shutdown — instead of reading the number in isolation.

Tricky questions

Can a process ignore SIGKILL if it really needs to finish something first?
Most people say

Yes, if it installs a signal handler for it.

Actually

No. SIGKILL and SIGSTOP are the two signals that cannot be caught, blocked or ignored by any process, by design — they are the kernel's guaranteed way to end or pause something no matter what it is doing.

Does a background job started with & keep running if you close the terminal?
Most people say

Yes, & detaches it completely.

Actually

Not by default. Closing the terminal typically sends SIGHUP to jobs still attached to it, and many programs exit on that. nohup or a proper daemon/service is what actually detaches a job from terminal lifecycle.

Scenario question

You're handed a batch job that today exits with code 137 partway through a run, every time, at roughly the same point. There is no log line about a signal. How do you find out what actually happened, and what do you change?

Model answer

I would not assume SIGKILL from the number alone, since a plain exit(137) looks identical. I would check the orchestrator's own events first — most container schedulers log an OOMKilled reason separately from the exit code — and check memory usage right before the failure point. If it is genuinely memory, the fix is bounding memory use at that step, not just raising the limit. If the events show nothing, I would add explicit logging immediately before and after the point where it dies, to at least narrow which line the process was on when it stopped, since SIGKILL gives it no chance to log anything itself. The trade-off is that a hard memory bound might make the job fail faster and more visibly than it does today, which I would accept over a silent kill at 137 with no explanation.

Interview corner

CommonWhat's the difference between SIGTERM and SIGKILL?
“I'd say SIGTERM is a request the process can catch, clean up after, and exit on its own terms; SIGKILL is the kernel ending the process immediately, and it cannot be caught, blocked, or ignored by anything. I always send SIGTERM first and give a real grace period before escalating.”
TrickA job exits with code 137. What does that tell you?
“128 plus 9, so it looks like SIGKILL, but that's not proof — a process can exit(137) on its own. I'd check the orchestrator's own events, like an OOMKilled flag, before assuming a signal was involved.”
Experience checkWhere has signal handling actually bitten you?
“A deploy sent SIGTERM to a worker mid-write, the grace period was too short for that job, and it got SIGKILLed before it could checkpoint. The checkpoint file said more rows were done than were actually committed. Now I only write the checkpoint after the commit succeeds, never before.”
DesignHow would you make a fleet-wide shutdown safe across very different job durations?
“I wouldn't use one fixed grace period for every job type. I'd have each worker report how close it is to a safe stopping point, and let the orchestrator's timeout depend on that, so a fast job doesn't block a deploy and a slow one isn't killed mid-checkpoint.”

Memory hook

SIGTERM is a request the worker can hear and act on; SIGKILL is the fire alarm cutting the power.

If you remember only one thing:

Send SIGTERM, wait a real grace period, then SIGKILL — and treat anything SIGKILL interrupted as unreconciled.

Self-check

  1. What number does a shell report for a child killed by signal 15?

    143 (128 + 15). For signal 9 it is 137.

  2. Why can't a process ignore SIGKILL the way it can ignore SIGTERM?

    SIGKILL and SIGSTOP are deliberately uncatchable, unblockable and unignorable by any process; SIGTERM is a normal catchable signal.

  3. What do exit codes 126 and 127 mean, and how do they differ from a signal exit?

    126 means the shell found the command but could not execute it; 127 means it could not find the command at all. Both come from the shell's own lookup, not from 128 + signal.

Prove it

Extend the run_job simulation to five items with a grace-period counter: after SIGTERM, the worker gets exactly two more items' worth of time before the orchestrator escalates to SIGKILL. Run it for a job where the slow item is item 2 and again where it's item 4, and paste both outcomes, including which items were fully done, which was lost, and the reported exit code (0, 137 or 143).

ce-02-03Card 3 of 6

Build pipelines that preserve meaning

MediumP2 · Very likelyKnow cold13 min
pipesexit-statusjqtext-processing

Why you care

A pipeline that "worked" in testing and quietly reports success on an empty input in production is one of the fastest ways to lose a night of alerting: grep ERROR app.log | wc -l reports 0, the job exits 0, and nobody notices that grep itself found nothing because the log rotated early. The bug is not in your logic; it is in what "the pipeline's exit status" actually means, which most 3-year developers have never had to think about because it rarely bites in a short script. Interviewers ask about pipefail and about jq versus regex for the same reason: both separate people who have only ever chained commands that happened to work from people who know what a pipe silently discards. This card is where "does the shell tell me the truth" gets answered.

Explain like I'm 5

A pipe is a line of people passing a bucket down a chain, each doing one job to whatever's in it before passing it on: one person filters out anything that isn't red, the next rearranges what's left, the last one pours it into a jar. By default, only the last person's "did I do my job okay" gets reported back to whoever's watching — if the filterer at the front dropped the bucket, nobody downstream necessarily notices, and the person watching only hears from the one pouring into the jar. jq is a person in that chain who actually reads labels on what's inside the bucket, instead of a person who just guesses where the label is by counting commas.

Core concept

  • A pipe (|) connects one command's stdout to the next command's stdin. stderr is a separate stream and is not piped by default — it goes straight to the terminal (or wherever you redirect it with 2>), which is exactly why an error message can appear "out of order" relative to piped output.
  • Redirection order matters: cmd > out.log 2>&1 sends both streams to out.log, because 2>&1 is evaluated after > has already pointed stdout there; cmd 2>&1 > out.log does not, because at the point 2>&1 runs, stdout is still the terminal.
  • Pipeline exit status, by default, is the exit status of the last command only. set -o pipefail changes that to the right-most command that exited non-zero, or zero if all of them succeeded.
  • grep filters lines by pattern (exit status 1 means "ran fine, found nothing" — that is not a failure by default, and it is easy to mistake for one). awk operates on fields ($1, $2, …) and is a real (if minimal) programming language. sed rewrites text stream-by-stream. jq parses and queries actual JSON: .field, .[] | select(.status == "FAILED"), -r for raw (unquoted) string output.
  • Quoting in the shell controls whether a variable is split on whitespace and whether glob characters expand; unquoted $var in the middle of a pipeline is the single most common source of "it worked with one word and broke with two."

Decision rule: reach for grep/awk/sed on genuinely line-and-field-shaped text, and reach for jq the moment the data is JSON, even one field of it. A comma, a newline, or a nested object inside a string value will eventually break a comma- or whitespace-based parse; it will never break a proper JSON parser, because it never mistook structure for a delimiter in the first place.

Under the hood

Pipes are kernel objects with a bounded buffer (commonly 64 KB on Linux). If a downstream command exits or stops reading before upstream has written everything, the kernel does not silently swallow the excess: it delivers SIGPIPE to whatever tries to write into a pipe with no reader left. That is why some_command | head -n 5 can end some_command early rather than making it grind through the whole output for nothing — head reads five lines, closes the pipe, and the producer gets SIGPIPE on its next write. A process that has caught and ignored SIGPIPE instead sees EPIPE from write(), which is the same event surfacing as a return value instead of a signal.

Minimal example Pipeline exit status, with and without pipefail

PythonThe same upstream failure, reported two different waysruns
def pipeline_status(statuses, pipefail=False):
    if not pipefail:
        return statuses[-1]
    for s in reversed(statuses):
        if s != 0:
            return s
    return 0

upstream_fails = [1, 0, 0]     # e.g. grep found nothing (1), sed/awk ran fine after it
print("default:", pipeline_status(upstream_fails))
print("pipefail:", pipeline_status(upstream_fails, pipefail=True))

all_ok = [0, 0, 0]
print("default all ok:", pipeline_status(all_ok))
print("pipefail all ok:", pipeline_status(all_ok, pipefail=True))
Output
default: 0
pipefail: 1
default all ok: 0
pipefail all ok: 0

In production Why jq beats splitting text on commas

json.loads here stands in for jq -r; the point is identical either way, because jq is a real JSON parser, not a smarter regex.

PythonA comma inside one field breaks comma-splitting, not JSON parsingruns
import json

lines = [
    '{"order_id": "O1", "status": "FAILED", "reason": "timeout, retrying"}',
    '{"order_id": "O2", "status": "OK", "reason": ""}',
    '{"order_id": "O3", "status": "FAILED", "reason": "duplicate key"}',
]

def naive_reason(line):
    # stands in for: cut -d',' -f3 on a line that "looks like CSV"
    return line.split(",")[2].strip()

def jq_reason(line):
    # stands in for: jq -r '.reason'
    return json.loads(line)["reason"]

for line in lines:
    print("naive:", repr(naive_reason(line)), " jq:", repr(jq_reason(line)))

failed_ids = [json.loads(l)["order_id"] for l in lines if json.loads(l)["status"] == "FAILED"]
print("failed order ids:", failed_ids)
Output
naive: '"reason": "timeout'  jq: 'timeout, retrying'
naive: '"reason": ""}'  jq: ''
naive: '"reason": "duplicate key"}'  jq: 'duplicate key'
failed order ids: ['O1', 'O3']

See it happen One pipeline, then the naive version that hides a failure

access.loggrep ERRORawk print $4jq -r .code0 (misleading)
  1. Building the pipeline runs nothing yet; every stage is idle.
  2. grep scans access.log and matches three ERROR lines; its own exit status is 0 because it found something.
  3. awk reshapes each matching line down to just the fourth field, an embedded JSON blob.
  4. jq parses that blob properly, so a comma inside a message field never shifts a field boundary downstream.
  5. Bash reports the pipeline's exit status as jq's status alone: 0. Nobody asked what grep or awk actually did.
  6. Naive version: tonight grep matches zero ERROR lines (exit 1), but awk and jq still run on empty input and exit 0, so the pipeline still reports 0 and the alert never fires.
Notice: the pipeline's reported exit status is only ever the last stage's opinion, unless pipefail is on — everything upstream of that stage can fail in total silence.
Scenario A

In your world

The situation

Your alerting script runs grep ERROR app.log | awk '{print $4}' | wc -l, and pages when the count is above zero. The job logs no errors tonight and exits 0 whenever the count is zero.

What actually breaks

The log ships to a new mount point after a deploy, and app.log stops existing at the old path. grep fails to open the file, exits non-zero, but prints nothing to stdout; wc -l on empty input prints 0; the pipeline's exit status is wc -l's, which is 0. The script logs "no errors tonight" for a week while the app is actually failing loudly into a log nobody is reading anymore.

What you'd do on Monday

Turn on pipefail, and check the specific failure you care about instead of trusting a downstream count of zero to mean "all clear":

ShellPreserve the real failure through the pipeline (needs the actual log file and shell)
set -o pipefail
count=$(grep -c ERROR app.log | tail -1)
status=$?
if [ "$status" -ne 0 ] && [ "$status" -ne 1 ]; then
  echo "grep itself failed (exit $status), not just 'zero errors'" >&2
  exit 1
fi
Scenario B

Imagine you work at Cloudflare

50M edge log lines processed per minute

The numbers

Illustrative: an edge log pipeline handling 50M lines a minute across many nodes, feeding grep/awk stages before landing in a JSON sink. At even a 0.001% rate of lines with a stray comma inside a quoted field (a URL parameter, a user-agent string), that is 500 malformed extractions a minute if the parsing stage is comma-based rather than JSON-aware — enough to corrupt a meaningful slice of a real-time dashboard, not just one alert.

What the Scenario-A answer does here

The Scenario A fix (pipefail plus checking the real exit status) still matters at this volume, but it only protects the "did the pipeline run" question. It says nothing about the comma-in-a-field problem, because every stage can report exit 0 while still producing systematically wrong field values.

What flips at this scale

At this scale you stop treating the log format as "text that happens to be shaped like columns" and standardize on structured (JSON or protobuf) log lines at the source, so every downstream stage parses with a real parser instead of guessing a delimiter. pipefail protects you from a stage crashing; a structured format protects you from a stage lying to you while it reports success.

The trade-off you'd defend

I would defend standardizing the log format at the source over adding ever more defensive parsing downstream, accepting the one-time cost of changing what the application emits, in exchange for every downstream consumer getting correct fields for free instead of re-solving the same delimiter problem.

Illustrative figures chosen for the arithmetic; not Cloudflare-published numbers.

Gotchas

A successful last command hides an upstream failure

Symptom

A pipeline "succeeds" (exit 0) even though the first stage in it errored out or found nothing.

Cause

Without set -o pipefail, the pipeline's exit status is the last command's alone; everything before it is invisible to whoever checks $?.

Fix

set -o pipefail at the top of scripts that chain commands, and still check the specific stage's status directly when its failure mode (like grep's "exit 1 means no matches") isn't itself an error you care about.

grep's exit 1 gets treated as a crash

Symptom

A script wrapped in set -e dies the moment grep finds nothing, even though "found nothing" is a perfectly normal outcome.

Cause

grep returns 1 for "no lines matched," which is indistinguishable, as a bare number, from a real failure.

Fix

Handle it explicitly — grep ... || true when zero matches is fine, or check the count instead of the exit code, and reserve real error handling for genuine grep failures (exit 2).

Redirecting stderr after stdout has already moved

Symptom

cmd 2>&1 > out.log still prints errors to the terminal instead of the file.

Cause

Redirections apply left to right; at the point 2>&1 runs, stdout is still the terminal, so stderr gets pointed at wherever stdout was then, not wherever it ends up.

Fix

Order matters — cmd > out.log 2>&1 sends stdout to the file first, then points stderr at that same, already-redirected destination.

Tricky questions

Does jq just do a smarter regex match on the text?
Most people say

Yes, it's a more powerful grep for JSON-looking text.

Actually

No. jq parses the input into an actual JSON value tree and queries that structure. A comma, a newline or a nested object inside a string value never confuses it, because it never treated a delimiter character as structure in the first place — which is exactly what breaks a comma-split or regex approach.

If a pipeline has three commands and the middle one fails, does the last command even run?
Most people say

No, the pipeline stops at the failed command.

Actually

Yes. All commands in a pipeline start concurrently and each reads whatever the previous one produced; a failing middle command doesn't halt the ones after it, it just may produce less (or no) output for them to work with.

Scenario question

You're handed zcat access.log.gz | grep 5[0-9][0-9] | wc -l, run nightly, alerting if the count is above a threshold. Someone reports it never alerted during an outage where 5xx errors were clearly spiking. How do you find out why, and what do you change?

Model answer

I would first check whether zcat itself failed that night — a corrupted or missing gzip file would make zcat exit non-zero while producing no output, and grep/wc -l downstream would happily report a count of zero from nothing, with the pipeline's default exit status hiding the real failure. I would add set -o pipefail and check the status right after the pipeline, not just the count. I would also stop matching 5[0-9][0-9] against raw text and, if the logs are structured at all, use jq to filter on an actual status field, because a raw regex can match "500" inside something unrelated, like a byte count or a timestamp, and give a count that looks plausible but means nothing. The trade-off is a slightly more verbose script for one that tells the truth about whether it actually ran.

Interview corner

CommonWhat does set -o pipefail actually change?
“I'd point out that by default a pipeline's exit status is just the last command's. Pipefail changes it to the right-most non-zero status in the whole chain, so a failure earlier in the pipe doesn't get silently swallowed by a successful command after it.”
Trickgrep pattern file | wc -l prints 0 and the script exits 0. Is that definitely 'no matches'?
“Not necessarily — it's also what you'd see if grep itself failed to open the file. I'd check grep's own exit status, or turn on pipefail, before trusting a zero count to mean zero matches.”
Experience checkWhere has jq versus a text-based parse actually mattered for you?
“I've had a comma inside a free-text field shift every column after it in a comma-split parse, silently producing wrong values instead of an error. Switching that stage to jq on the actual JSON fixed it permanently, because jq never treats structural characters inside a string as delimiters.”
DesignYou're building a new log pipeline from scratch. Text tools or structured parsing?
“I'd standardize on structured, JSON, log lines at the source and parse with jq or a real JSON library everywhere downstream. grep and awk are still useful for quick, one-off filtering, but I wouldn't build a pipeline's correctness on delimiter guessing.”

Memory hook

A pipe is a bucket brigade where, by default, only the last person's "I'm fine" gets reported back.

If you remember only one thing:

Without pipefail, a pipeline's exit status is only ever the last command's opinion.

Self-check

  1. Without pipefail, what does a pipeline's exit status reflect?

    Only the last command's exit status; every earlier command's failure is invisible to it.

  2. Why is grep's exit code 1 different from a real error?

    Grep returns 1 for "ran fine but found no matches," which needs to be handled deliberately, not treated the same as a crash.

  3. Why does a comma inside a JSON string value break cut -d',' -f3 but not jq -r '.field'?

    cut has no idea the comma is inside a quoted string; it just counts delimiter characters. jq parses the actual JSON structure, so a comma inside a string is just a character, not a field boundary.

Prove it

Write a 10-line NDJSON fixture where at least two "OK" lines have a latency_ms field and at least one "FAILED" line has a reason field containing an embedded comma. Compute, with a real JSON parser, the average latency_ms for OK lines and the list of FAILED order ids. Then compute the same two things with a naive comma-split approach and paste both results side by side, showing where the naive one is wrong.

ce-02-04Card 4 of 6

Run services, scheduled jobs and remote sessions

MediumP3 · Role-dependentTalk about it12 min
systemdcronsshscheduling

Why you care

"It works when I run it by hand" and "it fails every night at 2am" are usually the same script in two different environments, and the gap is almost always PATH, working directory, or an assumption about who's watching for overlap. A cron job inherits almost none of your interactive shell's setup; a systemd service that isn't told what "healthy" means will get restarted into a crash loop instead of investigated; an SSH connection to the wrong host, silently accepted because host-key checking was disabled, is how a credential ends up somewhere it shouldn't. None of this shows up in a stack trace. Interviewers ask about crontab -l, systemctl status and SSH host verification because they are cheap, concrete checks for "has this person actually operated a scheduled job or a remote box, or only ever run scripts at a terminal." This card is the environment every one of your ETL jobs actually runs in, once nobody is watching.

Explain like I'm 5

Running a command yourself is like cooking in your own kitchen: every tool is where you left it. A cron job is someone else cooking in your kitchen at 2am from a note you left, with none of your personal shortcuts set up, in an empty apartment (no terminal, no login shell) — if a knife (a command) isn't exactly where the note says, they can't improvise and find it. A systemd service is a live-in chef the building manager watches: if the chef collapses, the manager restarts them, and keeps restarting them if they collapse again immediately, unless someone tells the manager what "the chef is actually cooking" looks like. SSH host verification is checking a stranger's ID before you hand them your house key — the first time you have no ID to compare against, but every time after that, if the "same" stranger's face has changed, you should refuse, not just assume they got a haircut.

Core concept

  • systemd manages long-running services as units. systemctl status <unit> shows the active state, the main PID, and the most recent log lines; its own exit code follows the LSB convention (0 for active, 3 for not active), which makes it scriptable, not just readable.
  • journalctl -u <unit> shows that unit's log history across restarts, which systemctl status's few recent lines usually don't cover — reach for it the moment "what happened last time" matters more than "what's happening right now."
  • cron runs a command when five fields — minute, hour, day-of-month, month, day-of-week — match. The two day fields are the one genuine trap: if either is restricted from *, cron fires when either matches, not both. crontab -l lists the current user's table; there's no undo, so read before you edit.
  • cron's environment is not your shell's. It sets SHELL=/bin/sh and LOGNAME/HOME from /etc/passwd, and it does not hand a job your interactive PATH. A script that runs a bare command name in your terminal can get "command not found" (exit 127, Explain processes, jobs and signals) under cron for that reason alone.
  • SSH host verification compares the server's host key against ~/.ssh/known_hosts. On first contact there is nothing to compare against, so the default behaviour asks you to confirm and then remembers. If a known host's key later changes, default behaviour refuses the connection outright — that refusal is the point, not a bug to silence with StrictHostKeyChecking=no.

Decision rule: for anything that should always be running, use a service manager, not cron, so restarts and status are supervised; for anything that runs on a schedule and finishes, use cron, but give it an absolute PATH, an explicit working directory, and a way to detect that the previous run is still going before starting a new one. Never disable host-key checking to make a connection error go away — find out why the key is unexpected first.

Minimal example Cron's day fields are OR'd, not AND'd, when both are restricted

Python"run at 02:00 on the 1st, or every Monday" fires for either reasonruns
from datetime import datetime

def cron_matches(spec, when):
    minute, hour, dom, month, dow = spec
    if minute != "*" and minute != when.minute:
        return False
    if hour != "*" and hour != when.hour:
        return False
    if month != "*" and month != when.month:
        return False
    dom_ok = dom == "*" or dom == when.day
    dow_ok = dow == "*" or dow == when.isoweekday() % 7   # cron: 0=Sunday
    if dom == "*" or dow == "*":
        return dom_ok and dow_ok
    return dom_ok or dow_ok   # both restricted: OR, not AND


spec = (0, 2, 1, "*", 1)                          # min hour dom month dow(1=Mon)
monday_not_the_1st = datetime(2026, 9, 28, 2, 0)   # a Monday, day 28
the_1st_not_monday = datetime(2026, 9, 1, 2, 0)    # day 1, a Tuesday

print("Monday, day 28:", cron_matches(spec, monday_not_the_1st))
print("day 1, a Tuesday:", cron_matches(spec, the_1st_not_monday))
Output
Monday, day 28: True
day 1, a Tuesday: True

In production Why the same command resolves differently under cron than at your terminal

PythonPATH lookup: interactive shell vs cron's minimal defaultruns
def which(cmd, path_dirs, existing):
    for d in path_dirs:
        candidate = f"{d}/{cmd}"
        if candidate in existing:
            return candidate
    return None

existing = {
    "/usr/bin/python3",
    "/usr/local/bin/aws",     # installed via a custom setup step, not the OS package manager
}

interactive_path = ["/usr/local/bin", "/usr/bin", "/bin"]
cron_path = ["/usr/bin", "/bin"]           # cron's minimal default: no /usr/local/bin

for label, path_dirs in [("interactive shell", interactive_path), ("cron job", cron_path)]:
    found = which("aws", path_dirs, existing)
    print(f"{label}: aws -> {found or 'not found (exit 127)'}")
Output
interactive shell: aws -> /usr/local/bin/aws
cron job: aws -> not found (exit 127)
Scenario A

In your world

The situation

Your nightly ETL job is a one-line crontab entry: 0 2 * * * ~/load.sh. load.sh calls aws s3 cp ... and psql ... by bare name, and it has run fine every time you tested it by typing ./load.sh at your own terminal.

What actually breaks

At 2am the job fails with aws: command not found, exit 127. Nothing about the script changed; your interactive shell's PATH (set in .bashrc, which cron never sources) included the directory aws lives in, and cron's minimal default PATH doesn't.

What you'd do on Monday

Make the script self-sufficient instead of depending on whoever launches it:

ShellA cron entry and script that don't depend on an inherited PATH (needs a real crontab and Linux host)
#!/usr/bin/env bash
set -euo pipefail
export PATH="/usr/local/bin:/usr/bin:/bin"
cd /opt/etl || exit 1
exec >> /var/log/load.log 2>&1
aws s3 cp "$SRC" ./staging/
psql -f load.sql
Text
# crontab -l
0 2 * * * /opt/etl/load.sh
Scenario B

Imagine you work at Ola

4,000 scheduled jobs across a fleet of ride-dispatch hosts

The numbers

Illustrative: a platform running roughly 4,000 cron-style jobs a night across many hosts for surge-pricing recalculation and driver-payout batches. If even 2% of jobs are missing an explicit PATH or working directory, that's about 80 jobs a night one bad base-image update away from failing identically and simultaneously — not because the logic changed, but because a shared assumption about the environment did.

What the Scenario-A answer does here

The Scenario A fix (hardcode PATH, cd to an absolute directory) still works per job, but "fix it in each script" doesn't scale to 4,000 of them maintained by different teams; a base-image change that removes a tool from /usr/local/bin still breaks all 80 the same night, and nobody notices the common cause until the third team reports the same error.

What flips at this scale

At this scale, environment assumptions move out of individual scripts and into a shared, versioned job-runner wrapper that every scheduled job goes through — one place that sets PATH, working directory, and logging consistently, and one place to fix when the base image changes. Overlap policy (don't start job N+1 if job N is still running) becomes the wrapper's job too, via a lock file or the scheduler's own concurrency setting, not something each script re-implements.

The trade-off you'd defend

I would defend a shared job-runner wrapper over 4,000 independently "fixed" scripts, accepting one more layer of indirection and a dependency on that wrapper being correct, in exchange for one place to fix an environment assumption instead of four thousand.

Illustrative figures chosen for the arithmetic; not Ola-published numbers.

Gotchas

A cron job "disappears" with no error anywhere

Symptom

The job neither runs nor logs anything, ever, even though crontab -l shows it.

Cause

A syntax error in the crontab line (or a missing newline at the end of the file) can make cron silently skip that entry; cron's own errors typically go to the mail spool or syslog, not to a place most people check.

Fix

Validate with crontab -l after editing, check /var/log/syslog or journalctl -u cron for cron's own complaints, and give the job its own log redirection so at least it leaves evidence when it does run.

A restricted day-of-month AND day-of-week is read as AND, not OR

Symptom

A job scheduled for "the 1st of the month and every Monday" fires far more often than the "and" implies.

Cause

Cron combines the two day fields with OR when both are restricted from *, which is the opposite of what most people assume from reading the line.

Fix

If you truly need AND semantics, restrict only one day field and add an explicit date check inside the script.

Disabling host-key checking to make an SSH error go away

Symptom

Host key verification failed is silenced by adding StrictHostKeyChecking=no, and the connection then "just works."

Cause

That refusal exists specifically to catch a host key that changed unexpectedly — a re-provisioned box, or something worse. Disabling the check removes the one signal that would have told you which.

Fix

Find out why the key changed (check with whoever owns the host) before connecting, and if the host was legitimately rebuilt, remove the old entry from known_hosts deliberately rather than turning off verification globally.

Tricky questions

Does a systemd service that keeps restarting mean it's "working," since it never stays down?
Most people say

Yes, the restarts show it recovers on its own.

Actually

A tight restart loop (CrashLoopBackOff-style behaviour) usually means the process fails immediately every time, and systemd is just hiding a permanent failure behind repeated, brief "up" windows. systemctl status and journalctl -u from before the last restart, not the current instant, is where the real cause is.

Is crontab -l showing an empty list the same as "no jobs are scheduled for this box"?
Most people say

Yes, if it's empty, nothing is scheduled.

Actually

crontab -l only shows the current user's table. System-wide jobs live in /etc/crontab and /etc/cron.d/, and other users have their own tables entirely; an empty personal crontab tells you nothing about those.

Scenario question

You're handed a report that a nightly reconciliation job "sometimes runs twice," producing duplicate rows on nights when the source file is unusually large. How do you confirm the cause, and how do you fix it without just making the job faster?

Model answer

I would check whether the job's own runtime, on the slow nights, exceeds the interval between cron firings — if a job scheduled every 15 minutes still takes 20 on a big night, the next scheduled run starts while the previous one is still writing, and that's the duplication, not a bug in the reconciliation logic itself. I would confirm by checking overlapping PIDs or timestamps in the job's own logs on a night it happened. The fix is an explicit overlap guard — a lock file checked and created atomically at the start of the job, with the run refusing to start (and logging why) if the lock is already held — rather than assuming the job will always finish before the next tick. I would not just "make it faster," because that only raises the size threshold at which the same race reappears; the lock removes the race regardless of runtime.

Interview corner

CommonWhat does systemctl status actually tell you, and what doesn't it?
“It shows the current active state, the main PID, and a handful of recent log lines. It's a snapshot, not history — for what happened across earlier restarts I go to journalctl -u for that unit.”
TrickA job works when you run it by hand but fails every night under cron. What's your first guess?
“PATH, almost always. Cron doesn't source your shell's startup files, so anything your interactive PATH added — a custom install directory, a virtualenv — isn't there. I check what the script assumes is on PATH before I look anywhere else.”
Experience checkWhere has an SSH host-key mismatch actually mattered to you?
“I've seen someone silence a host-key warning with StrictHostKeyChecking=no just to get a script running, on a host that had actually been rebuilt by someone else with different access. The warning was correct; disabling it just removed the one check that would have caught it.”
DesignYou need to run 4,000 scheduled jobs across a fleet reliably. Would you just write 4,000 careful cron entries?
“No — I'd put a shared wrapper between the scheduler and every job that fixes PATH, working directory, logging, and overlap handling in one place. Fixing an environment assumption in one wrapper beats finding and fixing it in four thousand independently written scripts.”

Memory hook

Cron is someone else cooking in your kitchen at 2am from a note, with none of your personal shortcuts set up.

If you remember only one thing:

Give a scheduled job an explicit PATH, an explicit directory, and an explicit answer to "what if the last run is still going."

Self-check

  1. Why can a script that runs fine at your terminal fail with exit 127 under cron?

    Cron does not source your interactive shell's startup files, so it uses a minimal default PATH; a command your terminal's PATH could find, cron's often can't.

  2. For a cron line with both day-of-month and day-of-week restricted, how are they combined?

    With OR: the job fires if either field matches, not only when both do.

  3. What should you do when SSH refuses a connection because a known host's key changed?

    Find out why the key changed before connecting — it can mean a legitimate rebuild or something worse — rather than disabling host-key checking to bypass the refusal.

Prove it

Write your own cron_matches-style check for a schedule with both day fields restricted (pick your own spec), and find two real dates in the next 60 days where it fires for a different reason each time — one because of the day-of-month match, one because of the day-of-week match. Paste both dates and which condition triggered each.

ce-02-05Card 5 of 6

Diagnose CPU, disks, descriptors and syscalls

MediumP3 · Role-dependentTalk about it12 min
diagnosisdiskfile-descriptorssyscalls

Why you care

"Disk full" and "job hanging" are two of the most common on-call pages you will get once you own anything running on a real box, and both have at least two unrelated root causes that produce the exact same symptom. A df -h that shows plenty of free space does not mean you have room to create a new file, and a process that "looks stuck" in top might be waiting on I/O, not burning CPU at all. Interviewers ask "how would you debug a hanging process" specifically because a wrong first guess (restart it, add more disk, add more CPU) wastes an incident's most valuable resource — time — on a fix that doesn't address the actual cause. This card is a short, ordered list of "which tool answers which specific question," so your first move in an incident is a check, not a guess.

Explain like I'm 5

Imagine a library with two separate limits: shelf space (bytes) and the number of catalogue cards it owns (inodes), one card per book no matter how thin. You can have huge open shelves and still be unable to add a single new book, because you've run out of catalogue cards — that's a filesystem full of inodes but not bytes, and it looks identical to a full library until you check the card catalogue specifically. Meanwhile, top tells you who's busy running around the library right now; lsof tells you which books a specific librarian is currently holding open and hasn't put back; strace is standing directly behind one librarian, writing down literally every request they make to the building's systems, one at a time.

Core concept

  • top shows live process resource use; CPU saturation (a process pegged near 100% of a core, load average persistently above your core count) is a different problem from blocked I/O, where a process shows low CPU but is stuck waiting — top's CPU column alone won't tell you which, but a process stuck in state D (uninterruptible sleep) is a strong hint it's the latter.
  • df -h reports byte usage per filesystem; df -i reports inode usage. A filesystem can be nowhere near full on bytes and still refuse every new file with "No space left on device" because it is out of inodes — usually from millions of tiny files, not a few huge ones.
  • lsof -p <pid> lists every file descriptor a process currently holds open — real files, sockets, pipes. A steadily growing count for one long-running process, without it ever going down, is the signature of a descriptor leak, distinct from "too much data," and it eventually hits the process's (or the system's) open-file limit.
  • strace traces the actual syscalls a process makes. strace -c summarizes counts and time per syscall (good for "what is this thing even doing"); strace -p <pid> attaches to a running process. It is invasive and slows the target down, so it is a bounded, deliberate tool with explicit access, not a default first step.
  • Bounded observation means: attach for the shortest window that answers the question, on a system you're authorized to touch, and record what you saw before detaching — not leaving strace running indefinitely "just in case."

Decision rule: start with the cheapest, least invasive check that could explain the symptom — df -h and df -i together for anything disk-shaped, top for anything resource-shaped — and only reach for lsof or strace once you have a specific hypothesis about which process and which resource, because both require you to already be looking at a suspect.

Minimal example The same "disk full" error, two different causes

Pythondf -h alone would have missed one of theseruns
def disk_report(name, bytes_total, bytes_used, inodes_total, inodes_used):
    bytes_free_pct = 100 * (1 - bytes_used / bytes_total)
    inodes_free_pct = 100 * (1 - inodes_used / inodes_total)
    if inodes_free_pct < 1:
        cause = "inode exhaustion (too many small files)"
    elif bytes_free_pct < 1:
        cause = "byte exhaustion (too much data)"
    else:
        cause = "healthy"
    return f"{name}: {bytes_free_pct:.1f}% bytes free, {inodes_free_pct:.1f}% inodes free -> {cause}"

print(disk_report("/data", bytes_total=500_000_000_000, bytes_used=120_000_000_000,
                   inodes_total=30_000_000, inodes_used=29_999_500))
print(disk_report("/var/log", bytes_total=100_000_000_000, bytes_used=99_800_000_000,
                   inodes_total=6_000_000, inodes_used=500_000))
Output
/data: 76.0% bytes free, 0.0% inodes free -> inode exhaustion (too many small files)
/var/log: 0.2% bytes free, 91.7% inodes free -> byte exhaustion (too much data)

In production Counting open descriptors the way `lsof -p pid | wc -l` would

PythonA per-file handle that's never closed shows up as a growing fd countruns
class FakeFD:
    """Stands in for a real OS file descriptor so this runs anywhere."""
    open_count = 0

    def __init__(self, name):
        self.name = name
        FakeFD.open_count += 1

    def close(self):
        FakeFD.open_count -= 1


def leaky_export(paths):
    handles = []
    for p in paths:
        handles.append(FakeFD(p))          # never closed until the function returns
    return len(handles)


def safe_export(paths):
    for p in paths:
        fd = FakeFD(p)
        fd.close()                          # closed before the next file opens


paths = [f"batch_{i}.csv" for i in range(500)]

leaky_export(paths)
print("open fds after leaky_export:", FakeFD.open_count)   # what lsof -p pid | wc -l would show

FakeFD.open_count = 0
safe_export(paths)
print("open fds after safe_export:", FakeFD.open_count)
Output
open fds after leaky_export: 500
open fds after safe_export: 0
Scenario A

In your world

The situation

A once-a-week vendor import writes one small status file per record instead of appending to one file, "to make retries easy." It has run for eight months on the same box.

What actually breaks

Overnight the job starts failing every write with OSError: [Errno 28] No space left on device. df -h /data shows 40% free. Nobody believes the disk is actually the problem, so the first response is to restart the job, which fails identically in seconds.

What you'd do on Monday

Check inodes, not just bytes, before assuming the free-space number is the whole story:

ShellCheck both limits before trusting either one (needs a real Linux filesystem)
df -h /data      # bytes: looked fine, so this alone was misleading
df -i /data      # inodes: this is the number that actually explains it
find /data/status -type f | wc -l   # confirm the small-file count is the cause

The real fix is structural, not a bigger disk: batch the per-record status writes into one append-only file or a small database table, so record count stops being the same thing as file count.

Scenario B

Imagine you work at Razorpay

30M payment-status files a month on one host

The numbers

Illustrative: a legacy status-tracking pattern writing one small file per transaction, at 30M transactions a month, on a filesystem provisioned with roughly 60M inodes total. Eight months of accumulation is 240M file-creation attempts against a 60M-inode budget — the filesystem runs out of inodes at around month two, regardless of how much byte capacity is left, because inode budgets are typically fixed at filesystem creation and don't grow with added disk space alone.

What the Scenario-A answer does here

The Scenario A fix (batch writes into one file) still works, but "just delete old status files to free up inodes" is a naive move at this volume: payment-status records often have retention and audit requirements, so deleting them to fix a filesystem problem can trade an outage for a compliance incident.

What flips at this scale

At payment-processing scale, the fix has to separate two concerns that got conflated: durable, queryable status (which belongs in a database, with real indexes and retention rules) and filesystem usage (which should never scale linearly with transaction count in the first place). The flip is treating "one file per record" as a design smell to remove, not a capacity problem to provision around with a bigger inode budget.

The trade-off you'd defend

I would defend moving status tracking into a database over reprovisioning the filesystem with more inodes, accepting a migration effort now, in exchange for a resource (inode count) that no longer scales 1:1 with a business metric (transaction count) that has no natural ceiling.

Illustrative figures chosen for the arithmetic; not Razorpay-published numbers.

Gotchas

df -h looks healthy while every write fails

Symptom

No space left on device, but df -h reports plenty of free bytes.

Cause

The filesystem is out of inodes, not bytes — usually from far more small files than the filesystem was provisioned to expect.

Fix

Always check df -i alongside df -h for this exact error; the fix is almost always fewer, larger files, not more disk.

A process looks "stuck" but isn't burning CPU

Symptom

A job appears hung in top, near 0% CPU, for far longer than it should take.

Cause

It's likely blocked on I/O — a slow disk, an unresponsive NFS mount, a database connection that never got a response — and CPU usage tells you nothing about that.

Fix

Check its process state (D for uninterruptible sleep is the tell) and what it's blocked on with lsof or strace, rather than assuming it's a CPU problem and giving it more cores.

A file descriptor leak looks fine until it isn't

Symptom

A long-running process eventually fails with "Too many open files," seemingly out of nowhere.

Cause

Something in its loop opens a resource (a file, a socket, a cursor) without a matching close on every path, including error paths, and the count creeps up until it hits the process's open-file limit.

Fix

Audit every code path that opens something for a matching close, prefer context managers that close automatically, and confirm the fix by watching lsof -p pid | wc -l stay flat over time instead of climbing.

Tricky questions

If df -h shows 40% free, is there definitely room to create a new file?
Most people say

Yes, 40% free space means there's room.

Actually

Not necessarily. Byte capacity and inode capacity are tracked and can run out separately; a filesystem full of many tiny files can refuse every new file with "no space left" while sitting at 40% free bytes. Check df -i too.

Does strace slow a process down enough to matter, or is it basically free to leave running?
Most people say

It's just observation, so it's safe to attach and leave running.

Actually

strace intercepts every syscall the process makes, which has real overhead, sometimes severe for syscall-heavy workloads. It's a bounded, deliberate diagnostic for a specific question, not something to leave attached "just in case," especially on anything latency-sensitive.

Scenario question

You're handed a report that a data-processing job has been "slow, then eventually hangs" for the past two nights, with no obvious error in its own logs. You have approved, time-limited access to the box. Walk through your first three checks, in order, and what each one would tell you.

Model answer

First, top, to see whether it's actually burning CPU or sitting idle/blocked — that alone tells me whether I'm looking at a compute problem or an I/O problem. If it's low CPU and in state D, I'd check df -h and df -i on whatever filesystem it writes to, because a full disk or exhausted inodes can make writes block indefinitely without throwing a clean error right away. If both look fine, I'd use lsof -p <pid> to see what it currently has open — a leaked or stuck file descriptor, or a socket to a dependency that stopped responding, would show up there. I would only reach for strace -p <pid> for a short, bounded window after that, once I have a specific "what syscall is it stuck in" question, because it's invasive enough that I want a hypothesis first, not a fishing expedition.

Interview corner

Common"No space left on device," but df -h shows free space. What's going on?
“I'd check df -i right away — the filesystem is very likely out of inodes, not bytes, which usually means far too many small files rather than too much data. The fix is changing the write pattern, not adding disk.”
TrickA process shows 0% CPU in top but seems completely hung. Does that rule out a resource problem?
“No — it points at one. Low CPU with a hang usually means it's blocked on I/O, not idle. I'd check its process state and what it currently has open before assuming CPU or memory is even the right thing to look at.”
Experience checkTell me about a time df -h lied to you, or seemed to.
“I've seen a job writing one small status file per record run out of inodes at 40% byte usage, months before the disk was expected to be an issue. df -h alone made it look like a mystery; df -i explained it in one command.”
DesignHow would you diagnose a hang on a production box you don't own, with limited access?
“I'd start with the cheapest, least invasive checks — top, df -h, df -i — and only escalate to lsof or strace once I have a specific hypothesis about which process and which resource, for the shortest window that answers the question, and I'd record what I saw before detaching.”

Memory hook

A full library can still refuse every new book if it's out of catalogue cards, not shelf space.

If you remember only one thing:

Check df -i alongside df -h, and reach for lsof or strace only once you have a specific suspect.

Self-check

  1. Why can df -h show plenty of free space while every write still fails with "No space left on device"?

    Because bytes and inodes are separate budgets; the filesystem can be nowhere near full on bytes while completely out of inodes, usually from too many small files.

  2. Why does 0% CPU usage in top not rule out a resource problem?

    A process stuck waiting on I/O shows low CPU too; CPU usage alone can't distinguish "idle" from "blocked," but process state (D for uninterruptible sleep) is a strong hint.

  3. Why is strace a bounded, deliberate tool rather than a default first check?

    It intercepts every syscall the target makes, which has real overhead, so it's reserved for a specific hypothesis about a specific process, not a first-pass fishing expedition.

Prove it

Extend the disk_report/FakeFD examples into one script that evaluates three services against three checks — byte usage, inode usage, and open-descriptor count — and prints a one-line verdict per service (healthy, byte exhaustion, inode exhaustion, or descriptor leak). Make at least one service trigger each non-healthy verdict. Paste the three verdict lines.

ce-02-06Card 6 of 6

Write a safe Bash automation script

MediumP3 · Role-dependentKnow cold13 min
bashautomationset-etrap

Why you care

A script that "usually works" is worse than one that visibly fails, because the failures it hides show up as corrupted data three steps later, not as a red build. set -e does not mean "stop on any error" — it has specific, well-known exceptions, and a script written as if it had none will keep running past a failed step inside an if, a pipeline, or a && list. A script with no trap leaves a temp file or a partial write behind on every crash. Interviewers ask about set -euo pipefail and trap because writing a script that fails loudly, cleans up after itself, and never traces a secret into a log is a genuinely different skill from writing one that happens to work today. This card is that skill, at Monday-morning-fix size.

Explain like I'm 5

A recipe card that says "stop cooking immediately if anything goes wrong" sounds absolute, but it has quiet exceptions the cook already assumes: tasting a sauce to check if it's ready doesn't shut the kitchen down, and neither does one failed attempt in "try the oven, or if that's broken, the stovetop." set -e is that recipe card — it stops on a failed step taken at face value, not on a failure the script is deliberately checking. A trap is a note taped to the door: "whoever leaves this kitchen wipes the counter first" — it runs whether the meal succeeded or the fire alarm went off.

Core concept

  • Shebang (#!/usr/bin/env bash) picks the interpreter; without it, a script's behaviour depends on how it's invoked, which is its own source of "works for me."
  • Quote every variable expansion ("$file", not $file) and use arrays ("${files[@]}") for anything that's genuinely a list, so a value with a space or a glob character in it is treated as one thing, not several — the same word-splitting trap as Navigate files and permission boundaries, now inside a script instead of at an interactive prompt.
  • set -e (errexit) stops the script on a failing simple command — but not on a command that's part of an if/while/until condition, not on a command before the last one in a pipeline (unless pipefail is also on), not on a command in a &&/|| list except possibly the whole list's own effective status, and not on a negated command (! cmd). set -u (nounset) turns a reference to an unset variable into an immediate error, instead of silently expanding to an empty string. set -o pipefail makes a pipeline's exit status reflect any failing stage, not just the last one — see Build pipelines that preserve meaning.
  • trap 'cleanup' EXIT runs cleanup when the script exits, for any reason: normal completion, an explicit exit, or an uncaught error under set -e. That's the shell equivalent of a finally block, and it's where temp-file removal belongs.
  • Idempotent effects: a script that's safe to re-run after a crash produces the same end state whether it ran once or three times — write to a temp path and rename into place atomically at the end, rather than appending or partially overwriting the real target on every attempt.
  • Never trace secrets: set -x (or any debug logging) prints every command it runs, arguments included — if a credential is ever passed as a command-line argument instead of through an environment variable or a file, set -x puts it straight into your logs.

Decision rule: turn on set -euo pipefail by default, know its exceptions well enough not to be surprised, validate arguments before touching anything, and register trap cleanup before creating what needs cleaning up.

Under the hood

set -e's exceptions follow from what "the exit status of a compound command" means. Bash applies errexit only to the overall status of a construct, and for an if/while condition, a &&/|| list, or a negated command, that overall status is deliberately defined as something other than "did the last simple command succeed" — the shell treats that command's failure as information the script is checking, not as the script's own failure. It's the same reasoning behind why a while read line; do ...; done < file keeps every variable it sets, but cmd | while read line; do ...; done normally doesn't: the pipe puts the while in a subshell, and a subshell's variables die with it.

Minimal example Where `set -e` does and doesn't stop the script

PythonThe four common exceptions to errexit, modeled directlyruns
def would_errexit_stop(context, exit_code):
    if exit_code == 0:
        return False
    if context in ("if_condition", "while_condition", "list_non_last", "negated"):
        return False
    if context == "pipeline_non_last":
        return False   # unless `set -o pipefail` is also on
    return True         # a plain top-level command, or the last stage of a pipeline


cases = [
    ("top_level", 1),
    ("if_condition", 1),
    ("pipeline_non_last", 1),
    ("list_non_last", 1),      # e.g. `false && echo unreachable`
    ("negated", 1),            # e.g. `! false`
]

for context, code in cases:
    print(f"{context} (exit {code}): script stops = {would_errexit_stop(context, code)}")
Output
top_level (exit 1): script stops = True
if_condition (exit 1): script stops = False
pipeline_non_last (exit 1): script stops = False
list_non_last (exit 1): script stops = False
negated (exit 1): script stops = False

In production A trap-shaped cleanup that runs on success and on failure, and only removes what it created

PythonThe Python shape of `trap 'rm -rf "$tmp"' EXIT`runs
import tempfile
from pathlib import Path


def load_batch(rows, should_fail):
    tmp_dir = Path(tempfile.mkdtemp(prefix="orders_load_"))
    tmp_file = tmp_dir / "staging.csv"
    created = []
    try:
        tmp_file.write_text("\n".join(rows))
        created.append(tmp_file)
        if should_fail:
            raise RuntimeError("load failed after staging")
        return f"loaded {len(rows)} rows"
    finally:
        # this is the shape of `trap 'rm -f "$tmp"' EXIT`: cleanup runs on the
        # happy path AND on the exception, and only removes what this run
        # itself created, never a path it didn't create.
        for f in created:
            f.unlink(missing_ok=True)
        tmp_dir.rmdir()
        print("cleanup ran, staging dir removed:", not tmp_dir.exists())


print(load_batch(["1,100", "2,200"], should_fail=False))
try:
    load_batch(["3,300"], should_fail=True)
except RuntimeError as e:
    print("RuntimeError:", e)
Output
cleanup ran, staging dir removed: True
loaded 2 rows
cleanup ran, staging dir removed: True
RuntimeError: load failed after staging

See it happen A safe script's own execution order

callerscripttrap handler./load.sh orders.csvtrap cleanup EXITvalidate $1, mktempload rows (fails on row 4,213)EXIT firestemp file removed, exit 1
  1. set -euo pipefail is the first line executed; every later command is now subject to its rules.
  2. Registered before anything that needs cleaning up is created — a trap set too late can't cover an early failure.
  3. Arguments are checked and the temp path is created only after the trap already covers it.
  4. A plain failing command here is exactly the case errexit is meant to catch.
  5. errexit stops the script; the registered trap still runs, because EXIT fires on every exit path, not just a clean one.
  6. The caller sees a real non-zero exit status, and the filesystem is left exactly as clean as if the script had never run.
Notice: the trap is registered before the risky work starts, which is the only ordering where it can cover a failure that happens on the very next line.
Scenario A

In your world

The situation

Your team's "quick" cleanup script does rm -rf $STAGING_DIR/* at the top of every run, where STAGING_DIR is set from an environment variable, and the script has no set -u.

What actually breaks

A config change removes STAGING_DIR from the environment. The script runs, $STAGING_DIR expands to an empty string, and rm -rf /* — with a leading slash from the trailing / in the original command and an empty variable — starts deleting from the filesystem root before anyone notices what's happening.

What you'd do on Monday

set -u turns that exact mistake into an immediate, loud failure instead of a catastrophe, and validating the variable explicitly makes the intent unmistakable:

ShellFail immediately on an unset critical variable (needs a real shell to demonstrate the difference)
#!/usr/bin/env bash
set -euo pipefail
: "${STAGING_DIR:?STAGING_DIR must be set, refusing to run}"
rm -rf "${STAGING_DIR:?}"/*
Scenario B

Imagine you work at Blinkit

1,200 warehouse batch scripts run nightly across dark stores

The numbers

Illustrative: a grocery-delivery operator (Blinkit, formerly Grofers) running roughly 1,200 nightly batch scripts across dark-store inventory and payout systems, written by different people over several years. If even 10% lack set -u and rely on an environment variable renamed during a platform migration, that's about 120 scripts a night at risk of the same "empty variable in a destructive command" failure on migration night.

What the Scenario-A answer does here

The Scenario A fix (set -euo pipefail plus explicit validation) is correct per script, but adding it to 1,200 scripts individually during a migration is itself a slow, error-prone rollout — some get missed, and you won't know which until one fails destructively.

What flips at this scale

At this scale, safety options belong in a shared template or wrapper every script sources, not something 1,200 authors each remember to add. The wrapper can also assert "the variables I depend on are exactly the ones I expect" before doing anything destructive, turning a silent rename into a loud, immediate failure on the first script that hits it, instead of a slow bleed across the fleet.

The trade-off you'd defend

I would defend a shared template with these safety options baked in, plus a linter flagging any script missing it, over trusting 1,200 scripts to each get it right, accepting a migration cost now for one place to fix a safety gap instead of twelve hundred.

Illustrative figures chosen for the arithmetic; not Blinkit-published numbers.

Gotchas

Piping into a while read loses variables set inside it

Symptom

while read line; do count=$((count+1)); done < <(cmd) keeps count after the loop; cmd | while read line; do count=$((count+1)); done reports it unset or zero.

Cause

A pipe puts the while loop in a subshell; every variable it sets dies with that subshell when the pipe closes.

Fix

Use process substitution (< <(cmd)) instead of piping into the loop, so the loop runs in the current shell.

set -e doesn't catch a failure inside an if

Symptom

A script with set -e keeps running past a command that clearly failed, as long as it was the condition of an if or while.

Cause

Bash deliberately exempts a command used as a condition; its non-zero status is information the script is checking, not an unhandled error.

Fix

If you want that command's failure to stop the script, check its status explicitly and exit yourself; don't rely on errexit to cover a case it exempts.

set -x puts a secret straight into the logs

Symptom

A credential passed as a command-line argument shows up in plain text in the console log, right after someone added set -x to debug an unrelated issue.

Cause

set -x prints every command it executes, arguments included — it cannot distinguish a secret argument from any other.

Fix

Never pass secrets as arguments; use environment variables or files with restricted permissions (Navigate files and permission boundaries), and scope any temporary set -x to the smallest block, not the whole run.

Tricky questions

If a script has set -e, does it stop on literally any non-zero exit code anywhere in it?
Most people say

Yes, that's what "exit on error" means.

Actually

No — it exempts if/while conditions, non-last pipeline stages (without pipefail), commands in a &&/|| list, and negated commands. A script written as if it had no exceptions will be surprised by all four.

Does trap cleanup EXIT only run when the script fails?
Most people say

Yes, it's for handling errors.

Actually

EXIT fires on every way the script ends — including a normal, successful finish and an explicit exit 0 — not only on failure. That's exactly why it's the right place for cleanup that must always happen, not a substitute for error-specific handling (which would use trap ... ERR).

Scenario question

You're handed a deploy script with set -e at the top that is supposed to stop immediately if the health check after deploy fails, but it keeps going and reports success even when the health check clearly returned a failing status in its logs. Where do you look, and what do you change?

Model answer

I would look for how the health check's result is used — if it's the condition of an if, the left side of a &&, or anywhere in a pipeline that isn't the last stage, set -e will not stop the script on its failure, because bash treats that as information the script is deliberately checking, not an unhandled error. I would fix it by checking the exit status explicitly and calling exit 1 myself in the failing branch, rather than assuming errexit covers a case it's designed to exempt. The trade-off is a couple of extra explicit lines instead of one blanket set -e, but it removes the exact ambiguity that caused this bug.

Interview corner

CommonWhat does set -euo pipefail actually buy you, in one sentence each?
“I'd sum it up as: -e stops the script on a failing top-level command, -u turns an unset variable reference into an immediate error instead of a silent empty string, and pipefail makes a pipeline's exit status reflect any stage that failed, not just the last one.”
TrickDoes set -e mean you never need to check a command's exit status yourself?
“No — it has real exceptions: if/while conditions, non-last pipeline stages without pipefail, && or || lists, and negated commands. Anywhere I care about one specific command's failure inside those, I check it explicitly.”
Experience checkTell me about a script you've fixed to be safer.
“I added set -u to a cleanup script after realizing an unset STAGING_DIR would expand to nothing and turn a targeted delete into something much more dangerous. It now fails loudly on an unset variable instead of silently doing the wrong thing.”
DesignYou're rolling out a required environment-variable rename across 1,200 scripts. How do you make that safe?
“I wouldn't trust 1,200 individually edited scripts. I'd put the safety checks and validation in one shared wrapper, add a linter that flags any script not using it, and let the first script that hits the renamed variable fail loudly and immediately, instead of quietly across the fleet.”

Memory hook

set -e is a recipe card that stops on a failed step taken at face value, but not on a failure the script asked for on purpose.

If you remember only one thing:

Know errexit's exceptions, quote every variable, and register your trap before you create anything that needs cleaning up.

Self-check

  1. Name two situations where set -e will not stop a script, even though a command inside it failed.

    Any two of: the command is an if/while condition; it's a non-last stage of a pipeline without pipefail; it's part of a &&/|| list; it's negated with !.

  2. Why does trap cleanup EXIT run even on a successful script?

    EXIT fires on every way the script ends, not only on failure — that's what makes it the right place for cleanup that must always happen.

  3. Why is a credential passed as a command-line argument dangerous once set -x is on?

    set -x prints every command and its arguments before running it, with no way to distinguish a secret argument from any other.

Prove it

Write a real bash script with a shebang, set -euo pipefail, argument validation that fails clearly on zero arguments, and a trap ... EXIT that removes a temp file it created. Run it once successfully, then force a failure partway through (an explicit exit 1 after the temp file exists) and show, from a real terminal, that the temp file is gone in both cases. Paste both transcripts.