think
16px
820px
← all files

Audit identity & trace-id uniqueness — design

Date: 2026-07-08
Status: spec
Source: two integration prompts from the observatory team
(INTEGRATION-chatbot-traceid-uniqueness.md, prompt-ai-ahu-chatbot-identity.md).
Both close observability gaps against ahu-gpu-manager/docs/CONVENTIONS.md §2
and touch the same audit-header edit sites, so they ship as one batch.

Problem

The AI Observatory ingests X-Request-Id as the audit trace_id and
X-User-Id as the actor. Two defects degrade forensics:

  1. Trace-id collisions. X-Request-Id is derived deterministically as
    `${sessionId}-t${history.length}` at every egress. That is unique
    only per (session, turn) — not globally. A reused session id with reset
    history (persisted session, or a smoke/eval harness replaying turn 0)
    regenerates identical ids, so one trace_id returns two unrelated request
    chains. Real evidence: the same …-t0 id logged 4 hours apart for
    different tasks.

  2. Unattributed actors. The observatory's per-actor security layer keys on
    engine / surface / user_id. Chatbot traffic sends a blank X-User-Id on
    almost every path, so every actor collapses to
    ahu-chatbot/{public,internal}/unattributed. The security rules can point
    at a surface but never a person/session.

Goals

  • X-Request-Id = `${sessionId}-t${turn}-${nonce}` — globally unique per
    turn-execution, shared by every sub-call of that turn, human-parseable
    (keeps the ${sessionId}-t${turn} prefix).
  • Idempotency-Key stays deterministic (${sessionId}:t${turn}:…) —
    unchanged. Trace uniqueness and idempotency determinism are decoupled.
  • Every gateway egress path carries a stable, non-PII X-User-Id:
    authenticated staff → opaque stable staff id; anonymous public → the existing
    per-session id. Never PII (name/email/NIK), never random-per-call.

Non-goals / out of scope

  • Synthetic-traffic hygiene (the colliding rows are a smoke/eval harness in a
    different repo). Flagged in completion notes, not fixed here.
  • Adding an Idempotency-Key to the staff/native chat path (separate ticket).
  • Gateway changes — none required; the gateway already reads both headers.

Design

Trace uniqueness (prompt 1)

Two egress entry points mint the turn trace id; each gets an injectable nonce
generator (default crypto.randomUUID()) so tests are deterministic:

  • Public orchestrator (apps/public-web/.../orchestrate.ts): the pure
    turnRequestId(req) becomes makeTurnTraceId(req, nonce?). The trap: it was
    called twice (buildLlms, buildTools). Fix — compute the trace id once
    at the top of runOrchestrator and thread that single value through
    buildLlms(req, turnTraceId) and buildTools(…, turnTraceId) so planning,
    synthesis, and the DataTool share one id.
  • Staff/native (packages/streams/src/native-provider.ts): mint once per
    sendMessage, nonce injected via NativeProviderConfig.nonce.

Verbatim-forward paths are left untouched: dash/gateway.py
(RequestIdMiddleware / GatewayHeaders), tools/data.ts, llm/client.ts
(Idempotency-Key).

Identity — X-User-Id (prompt 2)

Three egress paths, one rule set:

  1. Public orchestrator direct LLM calls — already send
    X-User-Id: req.sessionId (a non-PII per-session id). Confirmed + tested.
  2. Public data agent (via DataTool) — DataTool gains a userId param
    and forwards X-User-Id; the orchestrator passes req.sessionId, so the
    agent's own model + embedding egress attribute to the same per-session actor.
  3. Staff data agent (via NativeProvider → /api/dash-proxy → agent) —
    identity is injected server-side in the proxy from the verified JWT
    session, never trusting a client-sent header. The value is an opaque HMAC
    of the email     keyed by the app secret (staff-<hex16>): stable per person,
    non-reversible, no PII, no DB lookup.

Agent side (the agno 2.4.7 finding). agno 2.4.7 has no per-run header hook,
BUT the audit-hardening work already made GatewayHeaders a ContextVar-backed
Mapping that openai-python re-reads per request. So identity flows the same
way trace id already does: RequestIdMiddleware seeds a new user_id_var from
the incoming X-User-Id header, and GatewayHeaders._snapshot() injects
X-User-Id. This covers both agents.py model calls and embedder.py
retrieval egress with no change to either (they already route through
gateway_default_headers()); only the stale # TODO(gateway) comments update.
The documented landing is per-request (hence per-turn/per-user) attribution.

Dormancy

Additive X-* headers only. With MODEL_GATEWAY_URL unset the current
upstreams ignore them, so request URLs/bodies and all non-X-* headers stay
byte-identical. The staff-id HMAC and DataTool userId only produce header
values; they never alter the request otherwise.

Test plan (TDD)

  • streams: two sendMessage with identical (sessionId, history.length)
    → different X-Request-Id, both startsWith('…-t2-').
  • orchestrate: within one run, planning + synthesis + DataTool share one
    X-Request-Id starting ${sessionId}-t${turn}-; two runs with the same
    (session, turn) → different trace but same Idempotency-Key base.
  • orchestrate identity: direct calls carry X-User-Id === req.sessionId;
    DataTool forwards X-User-Id.
  • gateway.py: user_id_var injects X-User-Id; RequestIdMiddleware
    seeds it from the incoming header; absent header → no X-User-Id.
  • internal-web: opaqueStaffId is stable + contains no email; dash-proxy
    injects X-User-Id from the session and overrides any client-sent value.

Acceptance

  • [ ] X-Request-Id = ${sessionId}-t${turn}-${nonce}, unique per turn, shared
    across sub-calls; Idempotency-Key unchanged.
  • [ ] All three egress paths send a stable non-PII X-User-Id.
  • [ ] pnpm check green; agent pytest green; dormancy preserved.