Telemetry (OpenTelemetry)

The seekrit components that run on your infrastructure emit OpenTelemetry traces, metrics, and logs over OTLP/HTTP, configured entirely through the standard OTEL_* environment variables. Point them at the collector you already run and they appear alongside the rest of your services.

This covers the five self-hosted components:

Component	What it is
`seekrit-run`	The launcher that injects secrets and execs
`seekrit-proxy`	The agent egress proxy
`seekrit-sdk-server`	The Kubernetes/ESO sidecar
`seekrit-kms`	The AWS KMS-compatible gateway
`seekrit-provisioner`	The self-hosted temporary-access executor

note

This telemetry goes to your collector, never to seekrit. It is a different thing from audit log export, which streams the server-side audit trail from seekrit's API to your SIEM. These services export their own operational signals directly.

Turn it on

Set an OTLP endpoint. That is the whole configuration — there is no seekrit-specific telemetry setting:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
seekrit-proxy --config seekrit-proxy.toml

With no endpoint set, nothing is exported: no exporters are created, no background threads start, and no connections are attempted. Unlike the OpenTelemetry default, these binaries do not assume a collector on localhost:4318 — they run in containers and CI where there usually isn't one.

The usual variables all work, because the standard SDK handles them:

Variable	Effect
`OTEL_EXPORTER_OTLP_ENDPOINT`	Base endpoint; `/v1/traces`, `/v1/metrics`, `/v1/logs` appended
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`	Per-signal override (full URL)
`OTEL_EXPORTER_OTLP_HEADERS`	Collector credentials, e.g. `api-key=…`
`OTEL_SERVICE_NAME`	Override the default service name
`OTEL_RESOURCE_ATTRIBUTES`	Add resource attributes, e.g. `deployment.environment=prod`
`OTEL_TRACES_EXPORTER=none`	Disable one signal (also `METRICS`, `LOGS`)
`OTEL_SDK_DISABLED=true`	Disable everything

Only OTLP over HTTP is supported (http/protobuf). There is no gRPC exporter — it would roughly double the size of binaries that ship on scratch. Every collector accepts OTLP/HTTP on port 4318.

What gets recorded — and what never does

These processes hold decrypted secrets in memory. Telemetry leaves the process and lands somewhere with different access controls, so it is held to the same rule as the audit log:

caution

Spans and metrics carry secret names, counts, upstream hosts, durations, status codes, and error kinds. They never carry secret values, ciphertext, data keys, private keys, service tokens, Authorization headers, request or response bodies, or the SQL a provisioning command runs.

This is enforced by tests, not convention: each service has a tests/telemetry.rs that drives a real request carrying a sentinel secret value through the real handler and fails if that value appears anywhere in the exported spans.

Signals by component

seekrit-proxy — a span per proxied request (method, matched route, upstream host, injected secret names, status) plus:

seekrit.proxy.requests — by plane (reverse/forward) and outcome (forwarded, denied, no_route, bad_request, upstream_error)
seekrit.proxy.injections — substitutions performed, by upstream
seekrit.proxy.upstream_duration — upstream latency, by upstream

A rising outcome="denied" rate is the one worth alerting on: it means a workload is asking for a secret its route has no claim to.

seekrit-sdk-server — a span per API read, plus seekrit.sdk_server.secret_reads (by hit/miss/unauthorized), seekrit.sdk_server.refreshes (by ok/error), and seekrit.sdk_server.snapshot_size. Alert on refresh errors — they mean the snapshot is going stale and ESO will keep syncing old values.

seekrit-kms — a span per KMS operation, plus seekrit.kms.operations and seekrit.kms.operation_duration, both labelled by operation and by outcome (the AWS error code, e.g. NotFoundException, or ok).

seekrit-provisioner — a span per provisioning command (kind, lease id, statement count) plus seekrit.provisioner.commands and seekrit.provisioner.command_duration. Each rejection stage is its own outcome — signature_invalid, stale, replay, sql_error — so "someone is forging commands" is distinguishable from "the database is down".

seekrit-run — one span per invocation with the secret count, the command name, and seekrit.run.degraded. That last flag is the useful one: it is true when the resolve failed and the command ran with only .env plus the live environment. A fleet quietly running without its managed secrets looks healthy otherwise.

Connecting traces across the boundary

seekrit-run reads TRACEPARENT from its environment, so if your CI is instrumented (the Jenkins OpenTelemetry plugin, GitLab, otel-cli) the run attaches to the pipeline step that launched it instead of starting a detached trace.

The long-running services read traceparent from inbound requests and continue the caller's trace.

seekrit-proxy can also inject trace context into the requests it forwards, which is off by default:

propagate_trace_upstream = true

It is off because the proxy's upstreams are usually third-party APIs you don't operate, where propagation buys nothing and just hands an outside party a correlatable identifier. Turn it on when the upstream is your own instrumented service and you want one continuous trace.

Kubernetes

The seekrit-eso chart takes an otel block:

otel:
  endpoint: http://otel-collector.observability:4318
  serviceName: seekrit-sdk-server
  resourceAttributes: deployment.environment=prod
  headers: ""

Leave endpoint empty (the default) and the sidecar exports nothing.

note

If you enable the chart's networkPolicy, remember it restricts ingress to the sidecar. Egress to your collector is unaffected — but a cluster-wide default-deny egress policy of your own will block export until you allow it.

Docker

docker run --rm \
  -e SEEKRIT_TOKEN=skt_… \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318 \
  -e OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod \
  seekritdev/proxy --listen 0.0.0.0:8080

Building without it

Telemetry is compiled in by default and costs 350–540 KiB depending on the binary. If you build from source and want it gone entirely — most relevant for seekrit-run, the smallest binary and the one distributed standalone:

cargo build --release --no-default-features

The instrumentation call sites remain and become no-ops; no OpenTelemetry SDK or exporter is linked.