Observability + Rate Limiting — Design

Date: 2026-07-06 · Status: Approved (brainstorm)
Module: CORE platform (no license gate). Queue #3 (after LDAP ✅ + SCIM ✅; then Backup System #4).

Goal

Deepen Obscura's operability for on-prem/air-gapped deployments: richer Prometheus metrics
and a dependency-aware readiness probe, opt-in OpenTelemetry tracing (off by default —
nothing leaves the box unless configured), and admin-tunable rate limits. Builds on the
existing baseline (RED metrics at /metrics, /healthz+/readyz, three in-memory
limiters, slog request logging) — extends, does not replace.

Baseline (already shipped — do not rebuild)

• internal/httpapi/metrics.go: Prometheus registry with obscura_http_requests_total
  / _duration_seconds / _in_flight + obscura_login_total. Served at GET /metrics
  (local-scrape, never pushed).
• wire.go: /healthz (static 200) + /readyz (DB Ping + protection engine Ready).
• ratelimit.go: in-memory fixed-window limiters — per-user (600/min), per-auth-IP
  (60/min), per-TSA (600/min) — hardcoded consts; plus publicSignLimiter in
  handlers_public_sign.go. slog requestLog middleware.
• DB is pgxpool.Pool (platform/db/db.go) — exposes Stat().

Phase 1 — Richer metrics + dependency-aware /readyz

New Prometheus series (extend metrics.go; client_golang already a dep — NO new dep):
- DB pool — a custom prometheus.Collector reading pool.Stat() on scrape:
obscura_db_pool_total, _acquired, _idle, _max, _acquire_wait_total,
_acquire_wait_seconds_total.
- Dependency health — obscura_dependency_up{dep} gauge (1/0) for
postgres|minio|extract|embed|ai|gotenberg, set by the shared probe (below).
- Scheduler jobs — obscura_scheduler_runs_total{task,result} +
obscura_scheduler_run_duration_seconds{task}, incremented where the scheduler executes
a due task (wrap the run in internal/scheduler).
- Rate-limit rejections — obscura_rate_limit_rejected_total{limiter} bumped wherever a
limiter denies (the four limiters).
- AI tokens — obscura_ai_tokens_total{feature,direction} exported from the existing
ai_usage table via a scrape-time collector (sum tokens_in/out per feature).

Deep /readyz — a DependencyProbe (new small type) checks each dependency with a
per-dep short timeout (~2s): postgres Ping; MinIO bucket-exists/stat; extract/embed/ai
sidecars GET /healthz; Gotenberg reachability; protection engine Ready. Returns
200 {ready:true, deps:{...}} when all up, 503 {ready:false, deps:{dep:err}} when any is
down. The same probe result feeds the dependency_up gauges (probe runs on /readyz hit
and on a periodic background tick so metrics stay fresh without a scrape depending on a
request). /healthz stays a static cheap 200 (liveness only — never blocks on deps).

Phase 2 — OpenTelemetry tracing (opt-in, air-gapped-safe)

• Off by default. Spans are only created + exported when OTEL_EXPORTER_OTLP_ENDPOINT
  is set (plus optional OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME default
  obscura, OTEL_TRACES_SAMPLER/ratio). Unset → a no-op TracerProvider: zero span
  allocation, nothing leaves the box. This preserves the air-gapped posture as the default.
• New Go deps (pinned in go.mod; air-gap builds fine — deps are cached): go.opentelemetry.io/otel,
  .../sdk, .../exporters/otlp/otlptrace/otlptracehttp, .../otelhttp (or a hand-rolled
  middleware span to avoid the net/http wrapper if lighter).
• Instrumentation: one server span per HTTP request in the request middleware (attrs:
  route, method, status, user id when present); DB query spans via a pgx tracer
  (pgx.QueryTracer) on the pool; outbound spans around sidecar / Gotenberg / Peruri /
  OIDC / SMTP calls. Context propagation through the existing ctx.
• Log correlation: inject trace_id/span_id into the slog requestLog fields so logs
  and traces line up even with no collector running (and this is useful on its own).
• Config lives in platform/config (fail-closed only if an endpoint is set but malformed).
  A helper obs.SetupTracing(cfg) (shutdown func, error) wired in main/wire.go;
  shutdown flushes on exit.

Phase 3 — Admin-configurable rate limits

• Migration rate_limit_settings singleton (id=1 CHECK) with columns for each limit
  (user_per_min, auth_ip_per_min, tsa_per_min, public_sign_per_min) + updated_at,
  seeded with today's constants so behavior is byte-unchanged until an admin edits.
• Live-apply: limiters read the active values via an atomic.Pointer[rateLimitConfig]
  refreshed on save (the license hot-swap pattern) — no restart. A safety floor
  (e.g. each limit ≥ 10) is validation-enforced so an admin can't lock everyone (including
  themselves) out; and the auth-IP limit has a sane min so brute-force protection can't be
  disabled to 0.
• API: GET/PUT /api/v1/admin/rate-limits (perm: the same admin perm the other system
  settings use) + OpenAPI + gen:api.
• UI: an Admin → Observability tab (new ObservabilityTab.tsx registered in
  AdminPage): current limits (editable NumberInputs with the floor), live rejection counts
  per limiter (read from the Phase-1 metric via a small /admin/rate-limits/stats read or
  parsed from an admin metrics summary), and a read-only dependency-health strip mirroring
  /readyz. en/id.

Error handling

/readyz deep-probe uses a bounded total timeout so a hung dependency can't stall a load
balancer; a probe error marks that dep down, never panics. Tracing setup failure with an
endpoint set logs loudly and falls back to no-op (never blocks boot). Rate-limit save
validates the floor and rejects out-of-range with a clear error; a malformed stored row
falls back to the seeded defaults.

Testing (repo discipline: never go test on the live DB; NEVER docker compose down -v)

Build/vet + tsc/vite; curl e2e on the deployed stack: GET /metrics asserts the new series
present (db_pool, dependency_up, scheduler, rate_limit_rejected, ai_tokens); /readyz
returns 200 with all deps up, and 503 naming the dep when one is stopped (test by pausing a
sidecar via docker compose stop <svc> — NOT down -v — then restarting); rate-limit PUT
lowers a limit → 429 observed → raise → recovers, and the floor rejects a too-low value;
tracing with OTEL_EXPORTER_OTLP_ENDPOINT set to a throwaway otel/opentelemetry-collector
debug receiver shows spans (or assert trace_id appears in the slog request logs). Assert
/me 5 modules intact and tracing OFF by default (no endpoint → no spans, /metrics still
serves). Deploy from repo root; single-service teardown via docker compose stop.

Out of scope (YAGNI)

Metrics push/remote-write (local-scrape only), log shipping/aggregation (slog to stdout is
the seam), per-route custom rate limits (the four classes suffice), OTel metrics/logs
signals (traces only), a bundled collector (operator brings their own), alerting rules
(that's the operator's Prometheus/Grafana).