Configuration and secrets: precedence and feature flags
Why you care
Every ETL job you have written already has a config precedence, even if nobody designed it: a default in the code, a value in a checked-in file, an environment variable Jenkins sets, maybe a --batch-size flag someone added for a one-off run. If you cannot say which one wins without running the job, you cannot predict its behaviour in a given environment, and neither can the next person who touches it. Interviewers ask "how does your app get its config" because the answer reveals whether you have a contract or a pile of overrides nobody can trace. Secrets and feature flags are the sharpest edge of this: a secret that leaks through the wrong source, or a flag nobody remembers to turn off, is a production incident with your name on the commit.
Explain like I'm 5
Think of a thermostat with four ways to set the temperature: a factory default inside the unit, a weekly schedule programmed once, a phone app that can override today, and someone's hand turning the physical dial right now. If you walk in and the room is freezing, you need to know which of those four wins, in what order, or you will blame the wrong one. Config precedence is exactly that ladder: default, file, environment, and whatever override touched it last. A feature flag is the same dial, but for behaviour instead of temperature: it decides which code path runs right now, and if nobody notes down who set it and why, you are back to standing in a cold room guessing.
Core concept
- An explicit precedence contract: defaults < config file < environment variables < CLI/runtime flags (or whatever order your team picks). The exact order matters less than that it is written down and identical in every environment.
- Secrets are a special config class: never in defaults, never in a checked-in file in plaintext. They arrive via environment or a secret manager at runtime, are logged only in redacted form, and rotate without a code deploy.
- Startup validation: a service should fail fast at boot if a required key is missing or malformed, not three hours into a batch run.
- Feature flags are config with an owner, an expiry and a fallback: who owns it, when it gets removed, and what happens if the flag system itself is unreachable — always define the safe default yourself.
- No universal framework precedence exists. Spring, Django settings,
python-dotenv, and flag SDKs each default to a different order; your job is to make your own order explicit and tested, not assume a library's default matches your mental model.
Use file-based config for values that differ by environment but rarely change (table names, bucket names). Use environment variables for anything injected by the deployment platform (secrets, hostnames). Use a flag system only for things you intend to flip without a deploy, and never for permanent configuration — a flag with no removal date is dead code with extra ceremony.
Minimal example Precedence merge exposes a type bug
import os
defaults = {"timeout": 30, "batch_size": 500}
file_cfg = {"batch_size": 1000}
os.environ["APP_TIMEOUT"] = "5"
env_cfg = {k[4:].lower(): v for k, v in os.environ.items() if k.startswith("APP_")}
resolved = {**defaults, **file_cfg, **env_cfg}
print(resolved["batch_size"], resolved["timeout"], type(resolved["timeout"]).__name__)1000 5 strIn production Config loader: precedence, validation, redaction
import json
import os
from pathlib import Path
Path("config.json").write_text(json.dumps({"batch_size": 2000, "s3_bucket": "orders-raw"}))
REQUIRED = {"db_dsn", "s3_bucket", "batch_size"}
SECRET_KEYS = {"db_dsn"}
def load_config(path, env_prefix="ETL_", cli_overrides=None):
defaults = {"batch_size": 500, "db_dsn": "postgresql://localhost/dev"}
file_cfg = json.loads(Path(path).read_text()) if Path(path).exists() else {}
env_cfg = {
k[len(env_prefix):].lower(): v
for k, v in os.environ.items()
if k.startswith(env_prefix)
}
cli_cfg = cli_overrides or {}
resolved = {**defaults, **file_cfg, **env_cfg, **cli_cfg}
if "batch_size" in resolved:
resolved["batch_size"] = int(resolved["batch_size"])
missing = REQUIRED - resolved.keys()
if missing:
raise RuntimeError(f"missing required config: {sorted(missing)}")
return resolved
def redacted(cfg):
return {k: ("***" if k in SECRET_KEYS else v) for k, v in cfg.items()}
os.environ["ETL_DB_DSN"] = "postgresql://prod-host/orders"
cfg = load_config("config.json", cli_overrides={"batch_size": "5000"})
print(redacted(cfg))
print(cfg["batch_size"], type(cfg["batch_size"]).__name__){'batch_size': 5000, 'db_dsn': '***', 's3_bucket': 'orders-raw'}
5000 intSee it happen Four sources, one resolved value
- Start from defaults baked into the code: timeout=30, batch_size=500.
- The checked-in file overrides batch_size only; timeout is untouched.
- The environment variable overrides timeout, and it arrives as a string, not an int.
- The CLI flag wins last: batch_size becomes 50 for this one run only.
- Nothing validated the type of timeout, so '5' as a string silently reaches code that expects an int.
In your world
The situation
Your settlement job's defaults point at a dev database, a checked-in config.json sets the real table names, and Jenkins injects the prod DB host through an environment variable. Someone, debugging a one-off issue last sprint, added a --db-host flag to the scheduled job's launch script "just for that run" and forgot to remove it.
What actually breaks
The CLI flag outranks the environment variable, so every scheduled run has quietly been writing to a decommissioned read-replica for two weeks. The job logs 48,213 rows processed and exits 0, because writing to the wrong, still-reachable database isn't an error. Finance notices the production totals haven't moved before anyone notices the flag.
What you'd do on Monday
Make precedence visible, not just correct. Log the resolved value and its source for every non-secret key at boot, so an unexpected override (a CLI flag in a scheduled job) is loud instead of silent:
def resolve_with_source(defaults, file_cfg, env_cfg, cli_cfg):
resolved, source = {}, {}
for layer, name in [(defaults, "default"), (file_cfg, "file"), (env_cfg, "env"), (cli_cfg, "cli")]:
for k, v in layer.items():
resolved[k] = v
source[k] = name
return resolved, source
resolved, source = resolve_with_source(
{"db_host": "dev-db"}, {}, {"db_host": "prod-host"}, {"db_host": "debug-replica"}
)
print(resolved["db_host"], "from", source["db_host"])
if source["db_host"] == "cli":
print("WARNING: scheduled job running with a CLI override on db_host")debug-replica from cli
WARNING: scheduled job running with a CLI override on db_hostImagine you work at CRED
The numbers
Illustrative: 400 internal services, each reading config from 4 sources (default, file, env, flag service) is up to 1,600 individual settings. If even 0.5% of env-injected overrides are stale after a migration, that's 8 services quietly pointed at the wrong backing resource at any time, and none of it is visible by reading a repo, because the override lives outside it.
What the Scenario-A answer does here
The Scenario A fix (log the resolved config and source at boot) still works per service, but reading 400 boot logs by hand to find the 8 bad ones does not scale: at 2 minutes a log, that's over 13 hours of manual review to answer one question.
What flips at this scale
At this scale config stops being a per-service concern and becomes a product: a central config/flag service every service pulls from and reports its resolved values to, with an alert when a service's effective config changes outside a deploy. The flip is from "each service logs its own config" to "one system can answer which services are pointed at X right now", because that question is unanswerable from 400 separate log streams.
The trade-off you'd defend
I would defend a central config/flag service with change auditing over per-service files and env vars, trading a new piece of infrastructure (and a new single point of failure to design a safe fallback for) for the ability to answer "who is on the old database" in one query instead of 400 greps.
Illustrative figures chosen for the arithmetic; not CRED-published numbers.
Gotchas
An undocumented override targets the wrong database
The job logs success and writes rows, but finance says the numbers are wrong — the rows landed in a decommissioned read-replica, not the primary.
A --db-host CLI flag left over from a one-off debugging run outranks the environment variable Jenkins sets, and nobody printed which source won at startup.
Log the resolved value and its source for every non-secret key at boot, and treat an unexpected source (a CLI override in a scheduled job) as a startup warning, not silence.
Secrets land in logs
A print(config) or an unhandled exception traceback shows the DB password in Jenkins console output, visible to every logged-in Jenkins user.
Config and secrets are stored in the same dict with no distinction, and generic logging dumps the whole thing.
Mark secret keys explicitly, redact them in any representation that could be logged, and never let str(config) or default exception formatting include them.
A flag never gets removed
Six months later the code still branches on if flag_enabled("new_calc"), both branches have drifted, and nobody is sure which one is actually safe to delete.
The flag was created to de-risk a release, but no removal date or owner was recorded, so cleanup had no trigger.
Record an owner and a remove-by date at creation, and fail a check (or page the owner) when a flag passes its removal date, as shown in Version releases and expire risky flags's production example.
Tricky questions
Yes, env vars are safer than hardcoding, so that's enough.
Env vars beat hardcoding, but they are still visible to anything that can read /proc/<pid>/environ, a crash dump, or container-inspect output. For real secrets, a secret manager with access-controlled retrieval and rotation is the actual answer; env vars are just a delivery mechanism, not a vault.
Yes, that's what defaults are for.
A default is safe only if it is a safe failure value, not a safe-looking placeholder. A default DB host of localhost looks fine in dev and silently wrong if a prod override is missing, which is why startup validation should require certain keys explicitly rather than let them fall through to a default.
Scenario question
You're handed a service where DEBUG=True is hardcoded as the default and prod sets DEBUG=False through an environment variable, but a recent prod incident showed stack traces with internal paths in an API error response. What's going on and what do you change?
Model answer
I'd check whether the environment variable actually reaches the process: a typo in the variable name, a missing export in the entrypoint script, or bool("False") evaluating to True in Python would all leave DEBUG effectively on in prod despite the override looking correct. I'd fix the immediate leak by making the default False and parsing boolean env vars explicitly instead of relying on truthiness, so a misconfigured or missing override fails safe instead of failing open. The trade-off is a stricter default makes local dev slightly less convenient, so I'd also log the resolved value at boot so this is visible, not discovered through an incident.
Interview corner
CommonHow do you decide what goes in a config file versus an environment variable versus a secret manager?
TrickIf an env var and a config file both set the same key, which wins, and why does it matter?
Experience checkTell me about a config bug that reached production.
DesignHow would you design feature flag ownership for a 50-engineer org?
Memory hook
A thermostat has four hands on the dial; config precedence is just naming them in order.
Write the precedence order down, and log which source won for every key at boot.
Self-check
Why did
resolved["timeout"]print as the string'5'instead of the int5in the minimal example?Environment variables are always strings; the merge never coerced the type, so the env override's string value overwrote the int default unchanged.
What's the operational difference between a config value and a secret?
A secret needs access control and rotation and must never appear unredacted in logs; a config value just needs a documented source and precedence.
What should happen when a feature flag passes its removal date?
Something loud: a failing check or a page to the owner, not a silent no-op, so the flag actually gets cleaned up.
Why is
localhosta risky default for a database host?It looks like it works everywhere, so a missing prod override can go unnoticed instead of failing loudly.
Write a resolve_config(defaults, file_cfg, env_prefix, cli_overrides) function that returns both the merged config and a per-key dict of which source won. Feed it a case where an env var and a CLI flag both set batch_size, and print the per-key source map proving the CLI value won and why.