SF Compute (rented GPU nodes)

SF Compute sells H100 time on a market: you buy a contract for a few hours, nodes spin up in about five minutes, and when the contract ends they are gone. You consume it two ways — VM nodes you SSH into (h100v), or a namespace in a Kubernetes cluster (h100i, with InfiniBand).

Both hand you a fresh machine with nothing on it, which makes the secrets question sharper than it is on a long-lived server. There is exactly one thing you get to put on a VM node before it boots, and that thing is a cloud-init startup script — a file you hand the provider, stored with the order and replayed onto every node it creates.

The rule for this page: that script carries one credential, not twenty. Everything else it fetches.

Why not just paste the keys into the startup script

A training node is not a one-secret machine. It usually wants a Hugging Face token to pull weights, a Weights & Biases key to log to, object-storage credentials for checkpoints, and often a model-provider key for evals. Putting those five in the startup script means:

• Five plaintext values in a file you no longer control, for the life of the order — not the life of the node.
• The same five on every node the order creates, including ones that come up after you stopped paying attention.
• Rotation means a new order. The script is fixed at purchase; changing a key means the running fleet is stale and you cannot fix it without buying again.
• No audit trail. Nothing records which node read which credential, because nothing read anything — the values were already there.

Bootstrap with one credential instead, and let the node resolve the rest at start. That is the same trade the sandbox guides make and the same one seekrit-run exists for.

Which shape to use

The first two columns are the normal case: it is your own training code, and giving it the keys it needs is fine. Reach for the third only when the thing running on the GPU is not something you would have run anyway — an agent, a user-submitted job, an eval harness executing generated code.

VM nodes

1. Mint a token for this contract

The token is bound to one application environment and can read that and nothing else. Mint it per contract so revoking it at teardown is a complete cutoff:

seekrit token create --name sfc-2026-09-12 --app training --env production
# prints:  skt_XXXXXXXX_…    (save it now — shown once)

2. Write a startup script that fetches, not one that carries

SF Compute requires the startup script to configure SSH access, so you are writing one regardless. Add two things to it: the static seekrit-run binary, and the token.

#!/bin/bash
set -euo pipefail

# SF Compute requires this — without it you cannot reach the node.
mkdir -p /root/.ssh
echo "ssh-ed25519 AAAA… you@laptop" >> /root/.ssh/authorized_keys
chmod 700 /root/.ssh && chmod 600 /root/.ssh/authorized_keys

# The one credential this file carries.
echo 'SEEKRIT_TOKEN=skt_XXXXXXXX_…' > /etc/seekrit.env
chmod 600 /etc/seekrit.env

# ~2 MB, statically linked, no Node and no system CA bundle needed.
curl -fsSL https://run.seekrit.dev/install.sh | sh

# Everything else arrives here, decrypted on the node.
cd /workspace
seekrit-run --env-file /etc/seekrit.env --cache -- python train.py

seekrit-run authenticates with the token, fetches the environment's ciphertext, decrypts it locally, and execs your command with HF_TOKEN, WANDB_API_KEY, and the rest set. The seekrit API never sees a plaintext value — decryption happens on the GPU node, the same as it does in the browser and the CLI.

Note what the training process ends up holding: the resolved secrets and SEEKRIT_TOKEN itself, since the --env-file overlay is part of the environment seekrit-run hands down. That is fine for code you wrote. It is the reason the untrusted case below uses the proxy instead — a job that can read its own environment can re-resolve the whole thing, not just use the values it was given.

3. Buy the nodes

sf buy -d '1h' -t h100v
sf vm list
sf vm ssh root@156

Or for a reserved block with the startup script attached:

sf nodes create -n 8 -d "24h" -s "tomorrow at 9am"

4. Revoke when the contract ends

This is the step that makes the pattern hold. The node is gone; the token in that order's startup script is not.

seekrit token revoke skt_XXXXXXXX

seekrit token list shows role, status, and last-used time, so an order you forgot to clean up is visible rather than silent.

Per-run overrides with a branch

If each run needs its own values — a different checkpoint bucket, a per-experiment dataset URL — use an ephemeral branch with its TTL set to the contract length, instead of a new environment you have to remember to delete:

seekrit branch create run-0912 --app training --from production --ttl 12h
seekrit secrets set CHECKPOINT_URI "s3://runs/0912" --app training --env production --branch run-0912

Then set SEEKRIT_BRANCH=run-0912 alongside the token in /etc/seekrit.env. The branch expires on its own, taking its overrides and key grants with it.

Kubernetes

sf clusters list prints the clusters you can reach, their Kubernetes API endpoint, and your namespace — you are a tenant in a shared cluster, not its administrator:

sf clusters list
sf clusters users add --cluster alamo --user myuser
kubectl get pods
# No resources found in sf-jensen namespace.

That constraint decides the approach. seekrit's External Secrets Operator guide installs an operator and cluster-wide CRDs, which a namespaced tenant cannot do. So on SF Compute, put the token in a Secret and run the workload through seekrit-run:

kubectl create secret generic seekrit-token --from-literal=token=skt_XXXXXXXX_…

apiVersion: batch/v1
kind: Job
metadata:
  name: nanogpt
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: train
          image: ghcr.io/acme/train:latest
          command: ["seekrit-run", "--"]
          args: ["python", "train.py"]
          env:
            - name: SEEKRIT_TOKEN
              valueFrom:
                secretKeyRef:
                  name: seekrit-token
                  key: token
          resources:
            limits:
              nvidia.com/gpu: 8

Bake the launcher into the image rather than curling it at pod start — it is a single static file and the image then holds a launcher and no values:

FROM seekritdev/run:latest AS seekrit
FROM nvcr.io/nvidia/pytorch:24.10-py3
COPY --from=seekrit /seekrit-run /usr/local/bin/seekrit-run

Watch it come up the usual way — kubectl get pods, then kubectl logs -f <pod-name>. Every replica resolves independently, so an 8-node distributed job needs no shared state beyond the one Secret.

When the GPU job is not your code

An agent, an eval harness running generated code, or a user-submitted job can read os.environ. If that is what is on the node, injecting the values is the wrong shape — run seekrit-proxy on the node and give the workload placeholders:

# In the startup script, before the workload starts.
set -a; . /etc/seekrit.env; set +a      # SEEKRIT_TOKEN, for the proxy only
seekrit-proxy --config seekrit-proxy.toml &

OPENAI_API_KEY='{{seekrit:OPENAI_API_KEY}}' \
  OPENAI_BASE_URL=http://127.0.0.1:8080/openai \
  python agent_eval.py

The proxy resolves and decrypts once at startup, then substitutes the real value into requests to allowlisted hosts on the way out. The job holds a name; the credential exists only inside the proxy and in the request to the upstream. See agent access policy for bounding which hosts, methods, and paths it may reach.

Checklist

• Startup script carries the token and the SSH key — no other credentials
• Token bound to one application environment, never --admin
• seekrit-run installed on the node or baked into the image
• --cache on, so a network blip doesn't start a long job uncredentialed
• Token revoked when the contract ends (seekrit token revoke)
• Untrusted job? Proxy, not injection

See also

• seekrit-run launcher — install, flags, and the graceful-degradation rules
• Service tokens — what a token can reach, and how to scope it
• Branches — ephemeral per-run config with a TTL
• Kubernetes (ESO) — the declarative path, when you own the cluster
• Credential broker — the credential the workload names but never holds
• CI/CD & containers — the same service-token pattern everywhere else