# Telemetry (OpenTelemetry)

The seekrit components that run on **your** infrastructure emit
**OpenTelemetry** traces, metrics, and logs over OTLP/HTTP, configured entirely
through the standard `OTEL_*` environment variables. Point them at the collector
you already run and they appear alongside the rest of your services.

This covers the five self-hosted components:

| Component               | What it is                                   |
| ----------------------- | -------------------------------------------- |
| `seekrit-run`           | The launcher that injects secrets and execs   |
| `seekrit-proxy`         | The agent egress proxy                        |
| `seekrit-sdk-server`    | The Kubernetes/ESO sidecar                    |
| `seekrit-kms`           | The AWS KMS-compatible gateway                |
| `seekrit-provisioner`   | The self-hosted temporary-access executor     |

> **Note:** This telemetry goes to **your** collector, never to seekrit. It is a different thing from [audit log export](/docs/guides/audit-export), which streams the server-side audit trail from seekrit's API to your SIEM. These services export their own operational signals directly.

## Turn it on

Set an OTLP endpoint. That is the whole configuration — there is no seekrit-specific
telemetry setting:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
seekrit-proxy --config seekrit-proxy.toml
```

With no endpoint set, **nothing is exported**: no exporters are created, no
background threads start, and no connections are attempted. Unlike the OpenTelemetry
default, these binaries do not assume a collector on `localhost:4318` — they run
in containers and CI where there usually isn't one.

The usual variables all work, because the standard SDK handles them:

| Variable                                   | Effect                                                        |
| ------------------------------------------ | ------------------------------------------------------------- |
| `OTEL_EXPORTER_OTLP_ENDPOINT`              | Base endpoint; `/v1/traces`, `/v1/metrics`, `/v1/logs` appended |
| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`       | Per-signal override (full URL)                                |
| `OTEL_EXPORTER_OTLP_HEADERS`               | Collector credentials, e.g. `api-key=…`                       |
| `OTEL_SERVICE_NAME`                        | Override the default service name                             |
| `OTEL_RESOURCE_ATTRIBUTES`                 | Add resource attributes, e.g. `deployment.environment=prod`    |
| `OTEL_TRACES_EXPORTER=none`                | Disable one signal (also `METRICS`, `LOGS`)                   |
| `OTEL_SDK_DISABLED=true`                   | Disable everything                                            |

Only **OTLP over HTTP** is supported (`http/protobuf`). There is no gRPC exporter —
it would roughly double the size of binaries that ship on `scratch`. Every
collector accepts OTLP/HTTP on port 4318.

## What gets recorded — and what never does

These processes hold decrypted secrets in memory. Telemetry leaves the process
and lands somewhere with different access controls, so it is held to the same
rule as the audit log:

> **Warning:** Spans and metrics carry secret **names**, counts, upstream hosts, durations, status codes, and error kinds. They **never** carry secret values, ciphertext, data keys, private keys, service tokens, `Authorization` headers, request or response bodies, or the SQL a provisioning command runs.

This is enforced by tests, not convention: each service has a `tests/telemetry.rs`
that drives a real request carrying a sentinel secret value through the real
handler and fails if that value appears anywhere in the exported spans.

### Signals by component

**`seekrit-proxy`** — a span per proxied request (method, matched route, upstream
host, injected secret names, status) plus:

- `seekrit.proxy.requests` — by `plane` (reverse/forward) and `outcome`
  (`forwarded`, `denied`, `no_route`, `bad_request`, `upstream_error`)
- `seekrit.proxy.injections` — substitutions performed, by upstream
- `seekrit.proxy.upstream_duration` — upstream latency, by upstream

A rising `outcome="denied"` rate is the one worth alerting on: it means a workload
is asking for a secret its route has no claim to.

**`seekrit-sdk-server`** — a span per API read, plus
`seekrit.sdk_server.secret_reads` (by `hit`/`miss`/`unauthorized`),
`seekrit.sdk_server.refreshes` (by `ok`/`error`), and
`seekrit.sdk_server.snapshot_size`. Alert on refresh errors — they mean the
snapshot is going stale and ESO will keep syncing old values.

**`seekrit-kms`** — a span per KMS operation, plus `seekrit.kms.operations` and
`seekrit.kms.operation_duration`, both labelled by `operation` and by `outcome`
(the AWS error code, e.g. `NotFoundException`, or `ok`).

**`seekrit-provisioner`** — a span per provisioning command (kind, lease id,
statement count) plus `seekrit.provisioner.commands` and
`seekrit.provisioner.command_duration`. Each rejection stage is its own outcome —
`signature_invalid`, `stale`, `replay`, `sql_error` — so "someone is forging
commands" is distinguishable from "the database is down".

**`seekrit-run`** — one span per invocation with the secret count, the command
name, and `seekrit.run.degraded`. That last flag is the useful one: it is `true`
when the resolve failed and the command ran with only `.env` plus the live
environment. A fleet quietly running without its managed secrets looks healthy
otherwise.

## Connecting traces across the boundary

`seekrit-run` reads `TRACEPARENT` from its environment, so if your CI is
instrumented (the Jenkins OpenTelemetry plugin, GitLab, `otel-cli`) the run
attaches to the pipeline step that launched it instead of starting a detached
trace.

The long-running services read `traceparent` from inbound requests and continue
the caller's trace.

`seekrit-proxy` can also **inject** trace context into the requests it forwards,
which is **off by default**:

```toml
propagate_trace_upstream = true
```

It is off because the proxy's upstreams are usually third-party APIs you don't
operate, where propagation buys nothing and just hands an outside party a
correlatable identifier. Turn it on when the upstream is your own instrumented
service and you want one continuous trace.

## Kubernetes

The [`seekrit-eso` chart](/docs/guides/kubernetes) takes an `otel` block:

```yaml
otel:
  endpoint: http://otel-collector.observability:4318
  serviceName: seekrit-sdk-server
  resourceAttributes: deployment.environment=prod
  headers: ""
```

Leave `endpoint` empty (the default) and the sidecar exports nothing.

> **Note:** If you enable the chart's `networkPolicy`, remember it restricts **ingress** to the sidecar. Egress to your collector is unaffected — but a cluster-wide default-deny egress policy of your own will block export until you allow it.

## Docker

```bash
docker run --rm \
  -e SEEKRIT_TOKEN=skt_… \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318 \
  -e OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod \
  seekritdev/proxy --listen 0.0.0.0:8080
```

## Building without it

Telemetry is compiled in by default and costs 350–540 KiB depending on the
binary. If you build from source and want it gone entirely — most relevant for
`seekrit-run`, the smallest binary and the one distributed standalone:

```bash
cargo build --release --no-default-features
```

The instrumentation call sites remain and become no-ops; no OpenTelemetry SDK or
exporter is linked.