# HashiCorp Nomad

Nomad's `secret` block lets a job **name** a credential instead of holding one.
seekrit ships a secret provider plugin for it, so a job file looks like this:

```hcl
job "billing-api" {
  namespace = "billing"

  secret "seekrit" {
    provider = "seekrit"
    path     = "billing-prod"
  }

  group "api" {
    task "server" {
      driver = "docker"

      env {
        DATABASE_URL      = "${secret.seekrit.DATABASE_URL}"
        STRIPE_SECRET_KEY = "${secret.seekrit.STRIPE_SECRET_KEY}"
      }
    }
  }
}
```

Nothing in that file is a credential, and nothing in Nomad's state is a
plaintext.

## How it works

Nomad execs the plugin on the **client node** that will run the task, and reads
one JSON object from its stdout. The plugin holds the service token, calls
`GET /v1/resolve`, and decrypts in a process that lives for one fetch.

```
job `secret` block ─▶ Nomad client ──exec──▶ seekrit plugin ──/v1/resolve──▶ seekrit API
  (names a profile)    (your host)            (decrypts here)                (ciphertext only)
```

> **Note:** The seekrit API never sees plaintext, and neither does the Nomad server. Decryption stays on your client node, exactly as it does for the CLI and [`seekrit run`](/docs/guides/run).

## Prerequisites

Nomad **1.11 or later** — the `secret` block and the secret provider plugin
framework arrived in that release.

A seekrit service token bound to the app environment the jobs need (see
[Service tokens](/docs/guides/service-tokens)):

```bash
seekrit token create --name nomad-billing --app billing --env production
# prints:  skt_XXXXXXXX_…    (save it now)
```

## 1. Install the plugin on every client

```bash
curl -fsSL https://nomad.seekrit.dev/install.sh | sh
```

or, with the CLI already installed:

```bash
seekrit nomad install
```

> **Warning:** **The installed filename is the provider name.** Nomad matches a job's `provider` value against the plugin executable's filename, so the file must be called `seekrit` — the installers handle this, but a hand-copied binary under its artifact name (`seekrit-nomad-secrets`) registers nothing. There is no error and no log line; jobs simply cannot resolve their secrets.

The plugin has to land in the `secrets/` subdirectory of the agent's plugin
directory, and the agent has to be told where that is:

```hcl
# /etc/nomad.d/client.hcl
client {
  enabled           = true
  common_plugin_dir = "/opt/nomad/plugins"
}
```

→ `/opt/nomad/plugins/secrets/seekrit`

## 2. Give the client a credential

The plugin reads its configuration from `/etc/seekrit/nomad.json` on the client.
`seekrit nomad init` prints the whole set of files for the machine you run it
on:

```bash
seekrit nomad init --profile billing-prod --namespace billing
```

Profiles are **named credentials the node operator declares**. A job picks one by
name; it cannot introduce one, widen one, or supply its own token.

```json
{
  "profiles": {
    "billing-prod": {
      "token_file": "/etc/seekrit/tokens/billing",
      "namespaces": ["billing"],
      "jobs": ["billing-api", "billing-worker"]
    }
  }
}
```

```bash
# The token itself lives in its own file, not in the config.
sudo install -m 600 /dev/null /etc/seekrit/tokens/billing
echo "skt_…" | sudo tee /etc/seekrit/tokens/billing > /dev/null
sudo chmod 600 /etc/seekrit/nomad.json
```

> **Warning:** **A plugin is shared by every job on the client.** Without `namespaces` or `jobs`, any job scheduled on that node can use the profile — which is a weaker boundary than the job file suggests. Restrict every profile that holds something you would not give the whole client.

| Key | Meaning |
| --- | --- |
| `token_file` | Read the token from this file. Preferred — it keeps the credential out of both the config and the process table. |
| `token_env` | Read it from a named variable in the **agent's** environment. |
| `token` | Inline. Works; puts a credential in a file that would otherwise be safe to check in. |
| `namespaces` | Nomad namespaces allowed to use this profile. Absent = any. `[]` = none. A trailing `*` globs. |
| `jobs` | The same, matched against `NOMAD_JOB_ID`. |
| `api_url` | Per-profile API base. |
| `branch` | Resolve a [branch config](/docs/guides/branches) layered on the token's environment. |
| `allow_branch_override` | Let a job choose the branch via `SEEKRIT_BRANCH` in its `secret.env` block. Off by default. |
| `interpolate` | Expand [`${OTHER_SECRET}` references](/docs/guides/references). On by default. |

Unknown keys are a hard error, deliberately: a misspelled `namespaces` would
otherwise parse as "no restriction" and silently open the fence.

## 3. Restart the agent

```bash
sudo systemctl restart nomad
```

> **Warning:** **Plugins are fingerprinted once, at client startup.** Installing into a running agent does nothing until it restarts — and, again, without an error anywhere.

## 4. Check your work

```bash
seekrit nomad check --path billing-prod --namespace billing --job billing-api
```

```
nomad plugin check
installed    ok — /opt/nomad/plugins/secrets/seekrit
fingerprint  ok — 0.1.0
fetch        ok — 8 secrets
```

`check` runs the plugin exactly the way Nomad does, including the namespace and
job id it would pass, so a profile your job is fenced away from fails here
rather than at 3am. It prints a count, never names or values.

## Paths

The first segment of `path` is **always a profile name**, never a secret name.

| `path` | Resolves to |
| --- | --- |
| *(empty)* | the `default` profile, whole environment |
| `billing-prod` | that profile, whole environment |
| `billing-prod/STRIPE_API_KEY` | one secret from that profile |

A whole-environment fetch is the usual choice, because Nomad interpolates
`${secret.<block>.<KEY>}` per key. Naming a single secret narrows what the job's
templates can reach:

```hcl
secret "stripe" {
  provider = "seekrit"
  path     = "billing-prod/STRIPE_SECRET_KEY"
}
```

## Trying it on one client

For a single node, you can skip the config file entirely: put a token in the
Nomad agent's own environment and use the `default` profile.

```ini
# /etc/systemd/system/nomad.service.d/seekrit.conf
[Service]
Environment=SEEKRIT_TOKEN=skt_…
```

```hcl
secret "seekrit" {
  provider = "seekrit"
  # no `path` — the default profile, whole environment
}
```

Every job on that client can then read that one environment. Good for trying the
plugin; not the shape for a shared cluster.

## Availability

Nomad resolves secrets **once per task placement**, so a transient API blip
during a placement storm would otherwise fail every task on the node at once.
Turn on the last-known-good cache to survive that:

```json
{
  "cache": { "dir": "/var/lib/seekrit/nomad-cache", "max_age": "15m" }
}
```

Only the **still-encrypted** resolve body is written to disk — the same bytes
the API served, no more sensitive than the token next to it. Only a *transport*
failure falls back: a `401` does not, because serving a cached payload after the
API refused the token would quietly extend a credential that was revoked.
`max_age` bounds how long that can last.

## Telemetry

Nomad shows a plugin failure as one string on a task event and discards the
plugin's stderr, so the plugin exports OpenTelemetry to **your** collector if you
configure one:

```ini
Environment=OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
```

It records the profile name, the calling namespace and job id, a secret *count*,
whether the cache was used, and an error kind — never a value, a token, or
ciphertext. See [Telemetry](/docs/guides/telemetry).

## Troubleshooting

Every way to get a Nomad plugin wrong fails **silently**, so work down this list
rather than looking for an error message:

| Symptom | Cause |
| --- | --- |
| The job reports an unknown provider | The executable is not named `seekrit`, or is not in `<common_plugin_dir>/secrets/`. |
| Nothing happens at all | `common_plugin_dir` is unset in the agent's `client` block. |
| It worked before the install | The agent has not restarted; plugins are fingerprinted at startup. |
| `may not use the "…" profile` | The job's namespace or id is not in that profile's `namespaces`/`jobs`. |
| `no seekrit profile named …` | The `path`'s first segment does not match a profile on that client. |

`seekrit nomad check` distinguishes the first three from the last two.

## Reference

- CLI commands: [`seekrit nomad init`](/docs/reference/cli#seekrit-nomad-init),
  [`seekrit nomad install`](/docs/reference/cli#seekrit-nomad-install),
  [`seekrit nomad check`](/docs/reference/cli#seekrit-nomad-check).
- Minting and scoping tokens: [Service tokens](/docs/guides/service-tokens).
- Nomad's own docs: [`secret` block](https://developer.hashicorp.com/nomad/docs/job-specification/secret)
  and [secret provider plugins](https://developer.hashicorp.com/nomad/plugins/author/secret-provider).
- Runtime injection instead of a plugin: [`seekrit run`](/docs/guides/run).