seekrit
← all posts

Syncing secrets into Kubernetes with External Secrets Operator

How-to

External Secrets Operator (ESO) watches ExternalSecret resources, fetches the values they name from an external secrets manager, and writes them into ordinary Kubernetes Secret objects. Your pods use envFrom or valueFrom like they always did, and nothing in the pod knows where the values came from. The whole setup is three commands and one YAML file:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace

helm repo add seekrit https://charts.seekrit.dev
helm install seekrit-eso seekrit/seekrit-eso \
  -n seekrit-system --create-namespace \
  --set seekrit.token=skt_…
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: storefront
  namespace: seekrit-system
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: seekrit
    kind: SecretStore
  target:
    name: storefront-secrets
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: DATABASE_URL

This post walks through each piece, using seekrit as the backend. The ESO half is the same for any provider.

How does External Secrets Operator work?

Three objects:

ObjectWho writes itWhat it does
SecretStore / ClusterSecretStorePlatform team, onceSays how to reach a backend and authenticate to it
ExternalSecretApp team, per appNames the keys to fetch and the Secret to write them into
SecretESOThe result. A normal Kubernetes Secret, kept in sync

ESO polls on spec.refreshInterval, compares, and updates the target Secret when a value changes. Pods pick up the change the way they always have: mounted files update in place, environment variables need a restart (use Reloader or a checksum annotation if you want that automated).

Step 1: install ESO

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace

Once per cluster. Stock chart, no modifications.

Step 2: give the cluster a credential

ESO expects to pull plaintext from a backend. seekrit doesn't serve plaintext; GET /v1/resolve returns ciphertext plus a data key wrapped to the caller's token, and decryption happens client-side. So the seekrit-eso chart deploys a small in-cluster service that holds a service token, resolves and decrypts locally, caches the result, and exposes it to ESO through a webhook SecretStore it generates for you.

Mint a token bound to the one environment you want in this cluster:

seekrit token create --name eso --app storefront --env production

The token can read that environment and nothing else. Then install:

helm repo add seekrit https://charts.seekrit.dev
helm install seekrit-eso seekrit/seekrit-eso \
  -n seekrit-system --create-namespace \
  --set seekrit.token=skt_…

If you'd rather not add a repo, the same chart is published as an OCI artifact at oci://registry-1.docker.io/seekritdev/seekrit-eso.

For GitOps, don't pass the token with --set. Create the Secret yourself with sealed-secrets or SOPS and use --set seekrit.existingSecret=<name>.

Step 3: write an ExternalSecret

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: storefront
  namespace: seekrit-system
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: seekrit
    kind: SecretStore
  target:
    name: storefront-secrets        # the Kubernetes Secret ESO manages
  data:
    - secretKey: DATABASE_URL        # key in the resulting Secret
      remoteRef:
        key: DATABASE_URL            # name in the seekrit environment
    - secretKey: STRIPE_API_KEY
      remoteRef:
        key: STRIPE_API_KEY

One gotcha: a SecretStore is namespaced, so the ExternalSecret has to be in the namespace you installed the chart into. To reference it from your app's own namespace, install with --set secretStore.kind=ClusterSecretStore.

Check it worked:

kubectl -n seekrit-system get externalsecret storefront
kubectl -n seekrit-system get secret storefront-secrets

STATUS should be SecretSynced. If it's SecretSyncedError, the events on the ExternalSecret say why, and kubectl logs deploy/seekrit-eso shows the backend side.

Step 4: use the Secret

spec:
  containers:
    - name: api
      image: my-api
      envFrom:
        - secretRef:
            name: storefront-secrets

Or one key at a time with valueFrom.secretKeyRef, or as a volume. This is the point of ESO: the deployment manifest is boring.

How long does a rotated secret take to reach pods?

Two intervals add up. The in-cluster resolver re-fetches from seekrit on its own refreshInterval (chart value, default 60s), and ESO re-reads it on the ExternalSecret's spec.refreshInterval. A changed value lands in the Kubernetes Secret within roughly the sum, then your pods need whatever restart or reload they normally need.

What happens if the secrets backend is down?

Once running, the resolver keeps serving its last good snapshot from memory if a refresh fails, so a backend outage doesn't break existing pods. The first resolve after a restart is fail-closed by default: if it can't reach the API, it refuses to start, and every ExternalSecret it backs goes down with it.

To survive a restart during an outage, turn on the on-disk cache:

cache:
  enabled: true
  maxAge: 24h

It stores only the encrypted response, which still needs the token to decrypt, so the volume is no more sensitive than the token Secret already in the pod. The trade is that a revoked token keeps working from cache until maxAge passes while the API is unreachable. A reachable API that refuses the token clears the cache immediately.

Is this still zero-knowledge?

The seekrit control plane still never sees a plaintext value. The cluster does: ESO's whole job is writing decrypted values into Secret objects, which live base64-encoded in etcd. That's inherent to ESO with any backend. So:

  • Turn on etcd encryption at rest.
  • Restrict who can reach the resolver. It decrypts everything its token can read. The chart has a NetworkPolicy that allows ingress only from ESO's namespace: --set networkPolicy.enabled=true.
  • One token, one environment, one chart release. To sync a second app, install a second release with its own token and secretStore.name. Don't broaden a token.

When not to use ESO

If you don't want plaintext in etcd at all, skip the sync and inject at runtime instead: run the container through seekrit-run, which decrypts into the process environment and writes nothing to the cluster. You lose the "pods don't know about the backend" property and gain "the cluster never stores the value." Both are reasonable; pick based on who can read your etcd.

Full chart options: the Kubernetes guide.