Key management
Why you care
Every SSE-KMS bucket and every Secrets Manager secret sits on one mechanism: envelope encryption. Know only "KMS encrypts things" and you will eventually hit an AccessDeniedException on s3:GetObject where every S3 permission is correct, because the real blocker is a policy on the key, not the bucket. "The identity policy allows it" is the answer of someone who has never debugged a production decrypt failure — the real answer always checks two policies. IAM policy evaluation owns IAM evaluation; this card owns the one resource policy here that is default-deny for everyone, including the account root.
Explain like I'm 5
A KMS key is a vault that never leaves the room. To lock a box you ask the clerk for a one-time combination, handed to you two ways: the plain combination, and the same combination sealed in an envelope only the vault can reopen. You lock the box, throw away the plain combination, and keep only the sealed envelope. Later, to open the box, you hand the envelope back — the clerk only breaks the seal if you're on this vault's approved list, a list the vault owns, not your badge issuer.
Core concept
- Envelope encryption:
GenerateDataKeyreturns a data key as plaintext and ciphertext. You encrypt your payload locally with the plaintext key, store the ciphertext key alongside it, and discard the plaintext immediately. KMS never touches bulk data, only the small key — which is why one mechanism scales from a secret to a 20 GB file. - Decrypt: you send KMS the ciphertext key, not the payload; it returns the plaintext key only if authorized, and you decrypt locally.
- Key policy is default-deny for everyone. No principal — not even root — has any permission unless a key policy, IAM policy, or grant explicitly allows it. The default key policy must explicitly enable IAM delegation, or an IAM allow does nothing on this key.
- Grants are narrow, revocable permissions on a key for a specific principal, used to hand out access without editing the key policy — AWS services use these on your behalf constantly.
- Rotation: a customer managed key can auto-rotate annually, creating new backing material under the same key ID/ARN; old material is kept so old ciphertext stays readable. AWS managed keys rotate yearly with no opt-out.
Decision rule: use an AWS managed key when you don't need to control decrypt access or cross-account sharing. Use a customer managed key the moment you need to name principals, share across accounts, or audit key usage separately.
Under the hood
A KMS key wraps an HSM backing key under a domain key AWS rotates on its own schedule. "Rotating" a customer managed key creates a new backing key version under the same key ID; the old one is kept so prior ciphertext still unwraps. The consequence: rotating a key never re-encrypts anything already written. To re-encrypt existing data under new material, you must re-write the objects yourself.
Minimal example Only the wrapped key is ever kept
plaintext_key = bytes([91, 14, 233, 7]) # KMS would hand you this once
wrapped_key = bytes(b ^ 0xFF for b in plaintext_key) # ciphertext form, safe to store
stored_object = {"ciphertext_key": wrapped_key} # plaintext_key never gets written anywhere
del plaintext_key # the only copy is gone now
print(stored_object)
print("plaintext_key" in locals()){'ciphertext_key': b'\xa4\xf1\x16\xf8'}
FalseIn production The key policy decides decrypt, not just the identity policy
from dataclasses import dataclass, field
@dataclass
class KeyPolicy:
enable_iam_delegation: bool
allowed_principals: set = field(default_factory=set) # explicit allow in the key policy
@dataclass
class IdentityPolicy:
allowed_principals_actions: dict
def can_decrypt(principal: str, key_policy: KeyPolicy, identity_policy: IdentityPolicy) -> str:
identity_allows = "kms:Decrypt" in identity_policy.allowed_principals_actions.get(principal, set())
key_policy_allows = principal in key_policy.allowed_principals or (
key_policy.enable_iam_delegation and identity_allows
)
if not identity_allows:
return "Deny: identity policy does not allow kms:Decrypt"
if not key_policy_allows:
return "Deny: key policy does not grant this principal, and IAM delegation is not enabled"
return "Allow"
identity = IdentityPolicy({"role/etl-runner": {"kms:Decrypt", "kms:GenerateDataKey"}})
key_no_delegation = KeyPolicy(enable_iam_delegation=False)
key_with_delegation = KeyPolicy(enable_iam_delegation=True)
key_explicit = KeyPolicy(enable_iam_delegation=False, allowed_principals={"role/etl-runner"})
print(can_decrypt("role/etl-runner", key_no_delegation, identity))
print(can_decrypt("role/etl-runner", key_with_delegation, identity))
print(can_decrypt("role/etl-runner", key_explicit, identity))Deny: key policy does not grant this principal, and IAM delegation is not enabled
Allow
AllowSee it happen GenerateDataKey and Decrypt round trips to KMS
- Before writing, the job asks KMS for a fresh data key.
- KMS returns both forms; only the ciphertext form is ever stored.
- The payload is encrypted locally; the plaintext key is discarded right after.
- A later read returns the wrapped key alongside the data — KMS has not been involved yet.
- To read it back, the job sends the small wrapped key to KMS, never the payload.
- KMS unwraps and returns the key only if the key policy (and identity policy) allow it.
- The job decrypts the payload itself; KMS never saw the bulk data at either end.
In your world
The situation
Your ETL role has had s3:GetObject/s3:PutObject on settlements-prod/* for a year under an AWS managed key. The platform team migrates the bucket to a customer managed key for a compliance requirement, using a boilerplate Terraform module.
What actually breaks
The next run fails every GetObject with AccessDeniedException. The IAM role hasn't changed. The actual cause: the boilerplate key policy only grants the platform team's own admin role and never delegates to IAM or names your ETL role — your existing s3:GetObject allow is irrelevant to the failure.
What you'd do on Monday
Check the key, not the bucket. Read the current key policy:
import boto3
kms = boto3.client("kms")
key_id = "alias/settlements-prod"
policy = kms.get_key_policy(KeyId=key_id, PolicyName="default")["Policy"]
print(policy) # inspect whether IAM delegation is enabled and whether etl-runner is namedThen add either an explicit statement naming your role, or enable IAM delegation and add kms:Decrypt/kms:GenerateDataKey to the role's identity policy, scoped to this key's ARN — not Resource: "*".
Imagine you work at Juspay
The numbers
Illustrative: 40M transactions/day, each with one encrypted payload, is 40M GenerateDataKey calls plus 40M Decrypt calls — about 80M KMS calls/day, under 1,000/s on average but with sharp retail-hour peaks.
What the Scenario-A answer does here
The Scenario A fix still applies, but one shared customer managed key across every transaction type means a single key-policy mistake locks out every bank integration at once, and all 80M calls compete for the same key's request allowance.
What flips at this scale
At this scale you partition keys by data classification — card tokens, PII, logs — so one bad policy or quota pressure point only affects one data class. On hot paths you also stop calling GenerateDataKey per record and reuse one data key per batch, trading a larger crypto-period for far fewer KMS calls.
The trade-off you'd defend
I would defend per-classification keys plus batched data-key reuse over one key for everything, trading more key inventory for a scoped blast radius and headroom under whatever the account's current KMS limits are.
Illustrative transaction volume; not Juspay-published. KMS request limits change over time, so none are asserted here.
Gotchas
An S3 allow is not a decrypt allow
s3:GetObject succeeds on unencrypted objects but returns AccessDeniedException only for SSE-KMS objects, even with s3:GetObject present.
Reading an SSE-KMS object needs kms:Decrypt, authorized by both the identity policy and the key policy. An S3 permission says nothing about key permissions.
Check the key policy for that object's key, not just the bucket; name the principal explicitly or enable IAM delegation and grant kms:Decrypt scoped to the key ARN.
Rotating a key does not re-encrypt your data
After enabling rotation, old objects still decrypt fine, and a reviewer asks why that doesn't prove "new" encryption.
Rotation creates new material for future calls; it never touches ciphertext already written, which KMS retains so old data stays readable.
If re-encryption under new material is required, read and re-write the objects yourself; rotation alone doesn't do it.
Losing the wrapped data key is losing the data
A bug deletes the ciphertext-key column from a metadata table; the KMS key itself is healthy.
KMS can only unwrap a ciphertext key it's given — a healthy master key is irrelevant if the wrapped key is gone.
Back up the wrapped data key with the same rigor as the payload it protects.
Tricky questions
kms:Decrypt on Resource: "*", can I decrypt any key in the account?Yes, a wildcard resource covers every key.
Only keys whose key policy opts into IAM delegation (or names this principal) are reachable. The key policy remains the gate regardless of the identity policy's wildcard.
Yes, every reference needs updating.
No. The key ID and ARN stay fixed for life; only the backing material rotates. SSE configuration and IAM references keep working unmodified.
Scenario question
You're handed a Lambda reading from an SSE-KMS bucket. Its role has s3:GetObject and kms:Decrypt on Resource: "*", yet every invocation fails AccessDeniedException on GetObject. What do you check, and what's the minimal fix?
Model answer
I would check the key's own key policy, not the IAM role again — a wildcard identity allow is only half the requirement. If the key policy doesn't name this role or enable IAM delegation, the request is denied regardless of the identity policy. I'd fix it narrowly: add a key-policy statement naming the Lambda role's ARN for kms:Decrypt, scoped to this key, rather than widening the identity side further. The trade-off: every new consumer of this bucket now needs an explicit key-policy edit, more friction than a wildcard, but it keeps the blast radius of a compromise to the principals actually listed.
Interview corner
CommonWhat is envelope encryption and why does KMS use it?
TrickMy role has full s3:GetObject, but reading an encrypted object still fails. Why?
Experience checkWhere has a KMS permission issue bitten a deployment you were close to?
DesignHow would you structure KMS keys for a platform with several data classes?
Memory hook
The vault clerk never lets your combination leave the room; you only ever carry the sealed envelope.
s3:GetObject allowed is not kms:Decrypt allowed — always check the key policy too.
Self-check
What two things does KMS return from
GenerateDataKey, and what happens to each?A plaintext data key, used immediately to encrypt locally then discarded, and a ciphertext data key, which is stored alongside the encrypted data permanently.
Why can the account root be denied access to its own KMS key?
Key policies are default-deny for every principal, including root; nothing is allowed unless a key policy, IAM policy, or grant explicitly says so, with no exceptions by identity.
Does automatic rotation re-encrypt existing S3 objects under a customer managed key?
No. Rotation only changes which backing key material future encryptions use; previously encrypted data stays readable using the retained old material.
Extend the production block with a grants dict mapping principal -> {allowed_actions}, separate from KeyPolicy. Write a can_use(principal, action, grants) function and three tests: a principal with a grant for Decrypt only (succeeds for Decrypt, fails for GenerateDataKey), a principal with no grant (fails both), and a revoked grant (remove the entry, confirm it now fails). Paste all three results.