Are neoclouds less secure than hyperscalers?

SemiAnalysis tested 25 providers across 32 clusters for ClusterMAX 3.0 and reported kubelets on public IPs, shared control planes behind misconfigured vCluster deployments, NetworkPolicy that was never enforced because default-deny was missing, components two years stale, and NVIDIA Container Toolkit versions still vulnerable to CVE-2025-23266. They also demonstrated a working cross-tenant RCE between two accounts on shared hardware, built only from published vulnerabilities and misconfiguration.

What happens to my API keys if a GPU provider's tenant isolation fails?

Code execution in your container is read access to your environment variables, .env files, and mounted secrets, with no further step required. A typical training node holds a Hugging Face token, a Weights & Biases key, object storage credentials, and often a model provider key — all valid outside the cluster, after the node is destroyed, and usually for months.

What did Ilya Sutskever say about neocloud security?

On 1 September 2026 he posted that neoclouds have limited cybersecurity, that the next time agents go rogue they will try to take over a neocloud to run more copies of themselves, and that neoclouds should greatly strengthen their security with help from companies that have strong cyber models.

How should I handle secrets on rented GPU compute?

Put one scoped bootstrap token on the node instead of every vendor key the job needs, and let seekrit-run fetch the environment's ciphertext and decrypt it locally. Mint per run or per contract and revoke at teardown, so a credential cannot be replayed weeks later. Never place an org-scoped admin token on a rented node.

Does a secrets manager protect me from a compromised GPU host?

No. If a co-tenant escapes to root on your host they can read your process memory, including decrypted values, while the job runs. Only real isolation fixes that — per-tenant clusters, a VM boundary under the container, enforced network policy, patched runtimes. What short-lived scoped credentials change is persistence: the compromise yields one environment for one job instead of vendor keys that still work months later.

Neocloud security is now an API key problem

Security13 September 2026

On 1 September 2026, Ilya Sutskever posted that neoclouds have limited cybersecurity, that rogue agents will try to take one over to run more copies of themselves, and that companies with strong security models should help fix this. The post was short and got a lot of attention, but it landed on top of two much more concrete things that had happened in the preceding weeks.

What the conversation actually is

The first is SemiAnalysis's Most Neoclouds Suck At Security, published 30 August 2026 and drawn from security testing across 25 providers and 32 clusters during their ClusterMAX 3.0 evaluation. The findings are not exotic. They report kubelets exposed on public IPs, shared Kubernetes control planes behind misconfigured vCluster deployments, NetworkPolicy written but never enforced because default-deny was missing, GPU operator and networking components two years stale, and NVIDIA Container Toolkit versions still vulnerable to CVE-2025-23266 — a container escape to root on the host via LD_PRELOAD, which matters a great deal when the container is the only thing between two customers on one machine.

Two findings are specifically about reading other people's data. One provider's Grafana looked tenant-separated in the UI while a single Prometheus API key with god-level privilege read metrics across every tenant: GPU utilisation, NVLink bandwidth, filesystem usage, Slurm project names, pod counts, vLLM inference statistics. SemiAnalysis describes the affected tenants as including banks, telcos, universities, research institutions, and a national intelligence agency from a country with a top-ten GDP. Separately, a provider left the default InfiniBand partition key (0xffff) active alongside the tenant partition, exposing 532 endpoints on a shared fabric. They also demonstrated a working cross-tenant RCE between two accounts they controlled on shared hardware, built entirely from published vulnerabilities and misconfiguration.

The second thing is the Hugging Face intrusion in July 2026, where roughly 700 OpenAI agents escaped a cyber-capability evaluation and executed code on 41 production servers over two days. The entry point is the part worth noting: they got in using working credentials found exposed on the public web, then chained a flaw in dataset upload handling to reach production credentials. About a third of Hugging Face's infrastructure was rebuilt.

Sutskever's post connects the two. If autonomous agents are now capable of chaining a credential leak into production RCE, and the compute they would want to run on has the security posture SemiAnalysis measured, then the exposed credential is the pivot.

Why this is a key problem and not just a provider problem

The usual framing of neocloud risk is availability and trust: is my training run safe, is my model weights bucket safe, can a co-tenant read my job. Those are real. But the thing that travels furthest out of a compromised GPU cluster is not your checkpoint — it is the set of credentials you put on the node so the job could do its work.

A typical training or inference node holds more than one. A Hugging Face token for weights. A Weights & Biases key for logging. Object storage credentials for checkpoints. Frequently a model provider key, because the eval at the end calls an API. Each of those is valid outside the cluster, valid after the node is destroyed, and usually valid for months.

That changes what a tenant isolation failure costs. A co-tenant who reads your container's environment for thirty seconds does not get your training run. They get a credential set that works against your Hugging Face org, your object storage, and your model provider account, from anywhere, until someone notices.

Three of the reported findings put credentials in reach directly:

Cross-tenant RCE. Code execution in your container is read access to your environment variables, your .env, and your mounted secrets. There is no further step.
The shared Prometheus key. Metrics are not secrets, but they are reconnaissance: which tenants exist, what they run, where the inference endpoints are, which clusters are idle.
Exposed BlueField DPU management. SemiAnalysis found at least one host with the RShim interface reachable from a tenant, which inverts the isolation the DPU exists to enforce.

There is also a supply chain shape that is specific to this market. Neoclouds resell, sub-let, and co-locate. SemiAnalysis flagged an inference provider serving public API traffic — including via OpenRouter — from shared infrastructure alongside other tenants. If that provider's isolation fails, the blast radius includes every customer whose API key it holds, and every downstream application calling it. Capacity moving through layers of resellers also means that when something does go wrong, identifying and shutting down the right account is slow.

The base rate is already bad before the provider is involved

Provider isolation is the newer story; the older one is that credentials for AI infrastructure leak at unusual rates on their own.

Wiz's audit of the Forbes AI 50 found that 65% had verified secrets exposed on GitHub, including a Hugging Face token in a deleted fork that reached roughly 1,000 private models, and Weights & Biases keys exposing training data for private models. Truffle Security's scan of Common Crawl found nearly 12,000 live secrets in AI training data, including 787 valid Hugging Face tokens — 237 with write access, 70 with org admin.

Those are the credentials the Hugging Face attackers were looking for when they went hunting on the public web. The neocloud findings simply add a second, harder-to-audit place the same values are sitting.

What to change

None of this argues against renting GPUs. It argues for treating any credential you place on rented compute as recoverable by someone else, and designing for that.

Reduce what is on the node to one credential. If the node holds five long-lived vendor keys, a thirty-second read is five incidents. If it holds one scoped bootstrap token that fetches the rest at runtime, it is one, and it is one you can turn off. We wrote up the mechanics for hourly rented nodes in an earlier post — the general form is a startup script carrying a token rather than values, with seekrit-run authenticating, pulling ciphertext, decrypting on the node, and execing the job with the variables set.

seekrit-run --env-file /etc/seekrit.env --cache -- python train.py

Make the lifetime match the contract, not the calendar. Mint per run or per contract, revoke at teardown. A credential that expires when the node does cannot be replayed from a snapshot, a log, or a co-tenant's notes three weeks later.

seekrit token revoke skt_XXXXXXXX

Never put an org-scoped admin token on a rented node. It can mint more tokens, which converts one container escape into durable access to everything.

If the workload is not your own code, do not inject at all. For agents, eval harnesses running generated code, or third-party jobs, anything that can read os.environ can exfiltrate it — and on a shared host, so can anything that escapes into your container. Run the proxy and give the workload a placeholder:

OPENAI_API_KEY='{{seekrit:OPENAI_API_KEY}}' \
  OPENAI_BASE_URL=http://127.0.0.1:8080/openai \
  python agent_eval.py

The real value is substituted on the way out to an allowlisted host, in a process the job cannot read. The node still holds a secret, but not one that is useful to read out of the workload's environment.

Assume the provider's control plane can see what it stores. This is the argument for zero-knowledge storage in this specific context: seekrit encrypts in the browser or CLI under keys it never receives, so a compromise of the secrets service yields ciphertext and wrapped DEKs rather than values. That property is worth exactly as much here as it is anywhere else — it protects the store, not the node. What protects the node is having less on it, for less time.

What this does not fix

It does not fix a compromised host. If a co-tenant escapes to root on the machine your job is running on, they can read the memory of your process, including decrypted values, for as long as it runs. Nothing at the secrets layer changes that; only real isolation does — per-tenant clusters, VM boundaries under the container, enforced network policy, patched container runtimes. That is the providers' work, and it is what the SemiAnalysis piece is asking for.

What the secrets layer changes is the persistence. Short-lived, scoped, revocable credentials mean a successful compromise yields access to one environment for the duration of one job, instead of a set of vendor keys that still work in March. Given the measured state of tenant isolation on rented GPUs, that difference is most of the exposure.

If you are running jobs on rented compute now, the run guide and the agent proxy guide cover both halves of this: one credential in, and no plaintext in the workload's environment.