think
16px
820px
← all files

Audit identity & trace-id uniqueness — completion notes

Date: 2026-07-08
Spec: specs/2026-07-08-audit-identity-and-trace-uniqueness-design.md
Plan: plans/2026-07-08-audit-identity-and-trace-uniqueness.md
Status: implemented, gated green, deployed to staging + observatory-verified
(commit e12ae68, 2026-07-08).

Batches the observatory team's two integration prompts (trace-id uniqueness +
identity/X-User-Id) — they share edit sites and both are audit-header semantics.

What changed

Trace-id uniqueness (prompt 1)

  • packages/streams/src/native-provider.ts — mints X-Request-Id once per
    sendMessage as `${sessionId}-t${turn}-${nonce}`; nonce injectable via
    NativeProviderConfig.nonce (default crypto.randomUUID()).
  • apps/public-web/.../orchestrate.ts — the pure turnRequestId became the
    exported makeTurnTraceId(req, nonce?). The trap is closed: the trace id
    is minted once at the top of runOrchestrator and threaded through
    buildLlms(req, turnTraceId) and buildTools(…, turnTraceId, req.sessionId),
    so planning, synthesis, and the DataTool share one id.
  • Untouched (verbatim forward): dash/gateway.py, tools/data.ts X-Request-Id
    passthrough, llm/client.ts Idempotency-Key (still deterministic
    ${sessionId}:t${turn}:…).

Identity — X-User-Id (prompt 2)

  • Agno agents + embedders (dash/gateway.py, byte-identical both apps):
    added user_id_var; RequestIdMiddleware now seeds it from the incoming
    X-User-Id; GatewayHeaders._snapshot() injects X-User-Id. Because both
    agents.py model calls and embedder.py retrieval already route through
    gateway_default_headers(), both now carry the per-actor id with no code
    change — only the stale # TODO(gateway) / embedder comments were updated.
  • Public data path (tools/data.ts + orchestrate.ts): DataTool forwards
    X-User-Id; the orchestrator passes req.sessionId (the same non-PII
    per-session id its direct LLM calls already send).
  • Staff data path (api/dash-proxy + new auth/actor-id.ts): identity is
    injected server-side in the proxy from the verified JWT session and
    overrides any client-sent X-User-Id (no spoofing). The value is
    opaqueStaffId(email) = staff-<HMAC-SHA256(email, appSecret)[:16]> — stable
    per person, non-reversible, no PII, no DB lookup.

The agno 2.4.7 per-run-header finding (report-back)

agno 2.4.7 has no per-run header seam — Agent.run(user_id=…) never reaches
the HTTP layer and OpenAILike only takes client-level default_headers. The
landing (already proven by the audit-hardening GatewayHeaders): a
ContextVar-backed Mapping that openai-python re-reads on every request build
(guarded by test_openai_client_resolves_request_id_per_request). Identity now
rides the exact mechanism the trace id already used — per-request (hence
per-turn/per-user) attribution through the one cached client, seeded by the ASGI
middleware from the incoming header. Confirmed: embedder + orchestrator paths
carry the same id as the chat call they serve.

Identity contract — interpretation notes

  • Public accounts (logged-in citizens) still attribute by req.sessionId,
    not a per-account id — the public OrchestrateRequest carries only
    sessionId. Stable per-session grouping satisfies the prompt; per-account
    attribution would need the account id threaded into the request (future).
  • Non-PII for staff: the client User object has no numeric id (only
    email/role/name), so the opaque id is derived server-side from the email
    via keyed HMAC rather than shipping any raw identifier to the browser.

Dormancy

Additive X-* headers only. With MODEL_GATEWAY_URL unset the model/embedding
egress still hits plain vLLM, which ignores them; request URLs, bodies, and all
non-X-* headers are byte-identical. The DataTool userId and dash-proxy
injection only add a header value.

Tests (TDD)

  • streams: prefix+nonce assertion; two same-(session,turn) sends → different
    trace (friendly-errors.test.ts).
  • public-web: trace-id.test.ts — one shared X-Request-Id across
    planning/synthesis/DataTool + same X-User-Id; two runs → different trace but
    same Idempotency-Key base. data-tool.test.ts — X-User-Id forward/omit.
  • agents: test_gateway.py — user_id_var injection + middleware seeding (both
    apps).
  • internal-web: actor-id.test.ts (stable, non-PII); dash-proxy-identity.test.ts
    (server-injected, overrides client value, omitted when unauthed).

Gate: pnpm check green (typecheck + streams 17 / internal-web 59 / public-web
128 + conventions OK); agent pytest green (gateway 10×2, embedder 5) in the
agno==2.4.7 / openai==2.16.0 scratch venv.

Out of scope (flagged, per prompt 1)

  • Synthetic-traffic hygiene: the colliding rows are a smoke/eval/demo
    harness (surface=internal, empty user_id) in another repo reusing a fixed
    session. Recommend it mint a fresh session per run and/or tag itself with a
    distinct X-Surface (smoke/eval) so the observatory can exclude it from
    real-traffic error rates.
  • Staff/native Idempotency-Key: native chat still sends none; a
    deterministic ${sessionId}:t${turn} key would let the gateway dedupe staff
    retries. Separate ticket.

Deploy + verification (done 2026-07-08)

Shipped all 4 images (tag e12ae68) via the build split (per-image builds to
stay under the 10-min window; in-image agent pytest green — public 41 / internal
62); deploy-staging.sh recreated all four stacks. All four app/agent
containers healthy on the new image. Container health checks bind to the LAN
IP, so verify with curl http://192.168.83.20:3500/api/health (public ok:true)
and curl -I http://192.168.83.20:3510/ (staff 307→login) — the deploy script's
127.0.0.1 "PUBLIC UNHEALTHY" line is the known false negative.

Observatory confirmed (public path). Staging MODEL_GATEWAY_URL points at the
audit-emitting gateway (:8200, status:ok). A real public turn wrote 4
ai_calls rows (ahu-observatory-db on Server 2), all sharing one trace_id in
the new shape and all attributed:

trace_id = deploy-smoke2-<sid>-t0-e2f66fac-d05f-4577-99c6-0c25ea609119   (×4, one per sub-call)
user_id  = deploy-smoke2-<sid>                                            (was empty pre-deploy)
surface  = public

Pre-deploy rows (08:50, surface=internal) show the old unattributed pattern
(gateway-assigned ULID trace, empty user_id) for contrast.

Staff path (internal/staff-<hmac>) not yet driven live — needs an
authenticated staff chat turn; it reuses the same agent-side user_id_var
mechanism proven above plus the unit-tested dash-proxy injection. To confirm:
one staff chat, then check ai_calls for surface=internal with a
staff-<hex16> user_id.

Deploy quirk for this control container: no rsync → used a tar-over-ssh
rsync shim in /tmp/binshim (with the pnpm shim) so deploy-staging.sh runs
unmodified.