Syncing secrets into Kubernetes with External Secrets Operator
How-to
External Secrets Operator (ESO) watches ExternalSecret resources, fetches
the values they name from an external secrets manager, and writes them into
ordinary Kubernetes Secret objects. Your pods use envFrom or valueFrom
like they always did, and nothing in the pod knows where the values came
from. The whole setup is three commands and one YAML file:
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
-n external-secrets --create-namespace
helm repo add seekrit https://charts.seekrit.dev
helm install seekrit-eso seekrit/seekrit-eso \
-n seekrit-system --create-namespace \
--set seekrit.token=skt_…
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: storefront
namespace: seekrit-system
spec:
refreshInterval: 1m
secretStoreRef:
name: seekrit
kind: SecretStore
target:
name: storefront-secrets
data:
- secretKey: DATABASE_URL
remoteRef:
key: DATABASE_URL
This post walks through each piece, using seekrit as the backend. The ESO half is the same for any provider.
How does External Secrets Operator work?
Three objects:
| Object | Who writes it | What it does |
|---|---|---|
SecretStore / ClusterSecretStore | Platform team, once | Says how to reach a backend and authenticate to it |
ExternalSecret | App team, per app | Names the keys to fetch and the Secret to write them into |
Secret | ESO | The result. A normal Kubernetes Secret, kept in sync |
ESO polls on spec.refreshInterval, compares, and updates the target Secret
when a value changes. Pods pick up the change the way they always have:
mounted files update in place, environment variables need a restart (use
Reloader or a checksum annotation if you want that automated).
Step 1: install ESO
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
-n external-secrets --create-namespace
Once per cluster. Stock chart, no modifications.
Step 2: give the cluster a credential
ESO expects to pull plaintext from a backend. seekrit doesn't serve plaintext;
GET /v1/resolve returns ciphertext plus a data key wrapped to the caller's
token, and decryption happens client-side. So the seekrit-eso chart deploys
a small in-cluster service that holds a service token, resolves and decrypts
locally, caches the result, and exposes it to ESO through a webhook
SecretStore it generates for you.
Mint a token bound to the one environment you want in this cluster:
seekrit token create --name eso --app storefront --env production
The token can read that environment and nothing else. Then install:
helm repo add seekrit https://charts.seekrit.dev
helm install seekrit-eso seekrit/seekrit-eso \
-n seekrit-system --create-namespace \
--set seekrit.token=skt_…
If you'd rather not add a repo, the same chart is published as an OCI artifact
at oci://registry-1.docker.io/seekritdev/seekrit-eso.
For GitOps, don't pass the token with --set. Create the Secret yourself
with sealed-secrets or SOPS and use --set seekrit.existingSecret=<name>.
Step 3: write an ExternalSecret
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: storefront
namespace: seekrit-system
spec:
refreshInterval: 1m
secretStoreRef:
name: seekrit
kind: SecretStore
target:
name: storefront-secrets # the Kubernetes Secret ESO manages
data:
- secretKey: DATABASE_URL # key in the resulting Secret
remoteRef:
key: DATABASE_URL # name in the seekrit environment
- secretKey: STRIPE_API_KEY
remoteRef:
key: STRIPE_API_KEY
One gotcha: a SecretStore is namespaced, so the ExternalSecret has to be in
the namespace you installed the chart into. To reference it from your app's
own namespace, install with --set secretStore.kind=ClusterSecretStore.
Check it worked:
kubectl -n seekrit-system get externalsecret storefront
kubectl -n seekrit-system get secret storefront-secrets
STATUS should be SecretSynced. If it's SecretSyncedError, the events on
the ExternalSecret say why, and kubectl logs deploy/seekrit-eso shows the
backend side.
Step 4: use the Secret
spec:
containers:
- name: api
image: my-api
envFrom:
- secretRef:
name: storefront-secrets
Or one key at a time with valueFrom.secretKeyRef, or as a volume. This is
the point of ESO: the deployment manifest is boring.
How long does a rotated secret take to reach pods?
Two intervals add up. The in-cluster resolver re-fetches from seekrit on its
own refreshInterval (chart value, default 60s), and ESO re-reads it on the
ExternalSecret's spec.refreshInterval. A changed value lands in the
Kubernetes Secret within roughly the sum, then your pods need whatever
restart or reload they normally need.
What happens if the secrets backend is down?
Once running, the resolver keeps serving its last good snapshot from memory
if a refresh fails, so a backend outage doesn't break existing pods. The first
resolve after a restart is fail-closed by default: if it can't reach the API,
it refuses to start, and every ExternalSecret it backs goes down with it.
To survive a restart during an outage, turn on the on-disk cache:
cache:
enabled: true
maxAge: 24h
It stores only the encrypted response, which still needs the token to
decrypt, so the volume is no more sensitive than the token Secret already
in the pod. The trade is that a revoked token keeps working from cache until
maxAge passes while the API is unreachable. A reachable API that refuses
the token clears the cache immediately.
Is this still zero-knowledge?
The seekrit control plane still never sees a plaintext value. The cluster
does: ESO's whole job is writing decrypted values into Secret objects,
which live base64-encoded in etcd. That's inherent to ESO with any backend.
So:
- Turn on etcd encryption at rest.
- Restrict who can reach the resolver. It decrypts everything its token can
read. The chart has a
NetworkPolicythat allows ingress only from ESO's namespace:--set networkPolicy.enabled=true. - One token, one environment, one chart release. To sync a second app,
install a second release with its own token and
secretStore.name. Don't broaden a token.
When not to use ESO
If you don't want plaintext in etcd at all, skip the sync and inject at
runtime instead: run the container through seekrit-run,
which decrypts into the process environment and writes nothing to the
cluster. You lose the "pods don't know about the backend" property and gain
"the cluster never stores the value." Both are reasonable; pick based on who
can read your etcd.
Full chart options: the Kubernetes guide.