Integration task (ahu-ai-chatbot) — make the per-turn trace id (X-Request-Id) globally unique

Paste everything below into a Claude Code session running inside the ahu-ai-chatbot repo.
It is self-contained: context, root cause, the exact edit sites, the invariants to preserve,
the required tests, and the repo's own workflow gates.

Background — what's wrong (real evidence)

The AI Observatory shows the same trace_id reused for unrelated executions hours apart. Example (all engine=ahu-chatbot, surface=internal, empty user_id):

2026-07-08 04:38 UTC   trace_id = 019f4001-f31d-738f-8cee-56e6ad2d843e-t0   (ok/error)
2026-07-08 04:43 UTC   …-t2
2026-07-08 04:46 UTC   …-t4
2026-07-08 04:49 UTC   …-t6
2026-07-08 08:34 UTC   019f4001-f31d-738f-8cee-56e6ad2d843e-t0   ← SAME id, 4h later, different task

The base UUIDv7 019f…843e decodes to a mint time of 2026-07-08 04:34:57 UTC, so at 08:34 it is a reused 4-hour-old session id, not a fresh one. The turn index is even (t0/t2/t4/t6) because it is history.length, which grows by 2 per round (one user + one assistant message).

Root cause

X-Request-Id (which becomes the audit trace_id, per ahu-gpu-manager/docs/CONVENTIONS.md §2) is derived deterministically as:

`${sessionId}-t${history.length}`

That is unique only per (session, turn) — not globally. When a session id is reused with a reset/empty history (a persisted session, or a smoke/eval/demo harness that replays turn 0), the turn indices repeat and the trace ids collide. Result: searching one trace_id returns two or more unrelated request chains, defeating audit forensics.

The fix — decouple trace uniqueness from idempotency determinism

These are two different headers with two different jobs (CONVENTIONS §2):

Change: mint the per-turn trace id once per turn-execution at the outermost entry point as:

`${sessionId}-t${turn}-${nonce}`      // nonce = uuidv7() / crypto.randomUUID()

Keep the ${sessionId}-t${turn} prefix (so the observatory can still filter/group by session + turn and it stays human-parseable), and append a fresh nonce so two sends that compute the same prefix still differ. Leave Idempotency-Key untouched → a resubmitted turn still dedupes at the gateway, but each attempt is independently traceable and different sessions/turns never collide. This aligns better with §2 ("generate at the outermost entry point; reuse downstream") than the current deterministic-regeneration behavior.

Invariants you MUST preserve

1. One turn = one trace, shared by all its sub-calls. Within a single user turn, the plan call, synthesis call, data-agent call, and embedding calls all carry the same X-Request-Id. The agents' RequestIdMiddleware + GatewayHeaders (apps/{public,internal}-agent/dash/gateway.py) forward the incoming id verbatim to model/embedding egress — do not touch that, and do not mint per-sub-call ids (that would shatter the per-turn trace).
2. Idempotency-Key stays deterministic — no nonce.

Exact edit sites

1. packages/streams/src/native-provider.ts:77 (staff/native outermost send)

headers: { "X-Request-Id": `${sessionId}-t${history.length}` },

→ mint once per send as ${sessionId}-t${history.length}-${nonce}. Update the CONVENTIONS comment just above it (lines ~72–76).

2. apps/public-web/src/lib/orchestrator/orchestrate.ts:44 (turnRequestId) — ⚠️ the trap

turnRequestId(req) is currently pure and called twice (line 54 in buildLlms, line 152 in buildTools). If you make it mint a fresh nonce on each call, planning/synthesis and the DataTool get different ids → the per-turn trace fragments (invariant #1 broken).

Do this instead: compute the turn trace id once at the top of the orchestrate run (e.g. const turnTraceId = makeTurnTraceId(req)) and thread that single value through buildLlms and buildTools. Update the comment at lines 38–46 (it currently claims "a resubmitted turn regenerates the same values" for X-Request-Id — that is the behavior being intentionally changed).

No change (forward verbatim — leave alone):
- apps/{public,internal}-agent/dash/gateway.py — RequestIdMiddleware / GatewayHeaders forward the incoming id verbatim. ✅
- apps/public-web/src/lib/orchestrator/tools/data.ts — forwards this.requestId verbatim. ✅
- apps/public-web/src/lib/orchestrator/llm/client.ts — Idempotency-Key derivation stays ${idempotencyBase}:${++callSeq}. ✅

Testability: inject the nonce generator (parameter defaulting to uuidv7() / crypto.randomUUID()) so tests can stub it and assert deterministically.

Tests

Update:
- packages/streams/tests/friendly-errors.test.ts (~line 139): expect(headers["X-Request-Id"]).toBe("sesi-7-t2") → assert .startsWith("sesi-7-t2-") and a non-empty nonce suffix.
- Reword the two changed code comments (above).

Add:
- Streams — uniqueness: two sends with identical (sessionId, history.length) produce different X-Request-Id.
- Public orchestrate — per-turn sharing (invariant #1): within one orchestrate run, the planning client, synthesis client, and the DataTool all receive the same X-Request-Id, and it starts with ${sessionId}-t${turn}-.
- Public orchestrate — decoupling proof: two orchestrate runs with the same (sessionId, history.length) produce different X-Request-Id but the same Idempotency-Key base (${sessionId}:t${turn}:plan / :syn).

Keep unchanged (they test verbatim forwarding — still valid):
- apps/{public,internal}-agent/tests/test_gateway.py (sess-1-t3, sess-1-t4, sess-9-t2).
- apps/public-web/tests/orchestrator/data-tool.test.ts (forwards a configured id).

Out of scope — but flag these (do not implement here)

1. Synthetic-traffic hygiene (likely a different repo/harness). The colliding rows are surface=internal with empty user_id — a smoke/eval/demo harness reusing a fixed session. Recommend the harness mint a fresh session per run and/or tag synthetic traffic with a distinct X-Surface (e.g. smoke/eval) so the observatory can exclude it from real-traffic error rates. Leave a docs/superpowers/ note or ticket.
2. Native/staff chat sends no Idempotency-Key (CONVENTIONS §2 says "recommended on chat"). Adding a deterministic ${sessionId}:t${turn} key would let the gateway dedupe staff retries. Separate ticket.

Workflow & gates (this repo's CLAUDE.md)

• This changes audit-trace semantics across two surfaces → treat as feature-sized: short spec → plan → completion notes under docs/superpowers/{specs,plans}. TDD.
• pnpm check (typecheck + all suites + scripts/check-conventions.mjs) must pass before anything ships.
• Respect the app boundary: shared code lives in packages/@ahu/*; public-web and internal-web never import each other. (The two entry points are edited independently; any shared nonce helper goes in a package, not a cross-app import.)
• Do not deploy without Efran's explicit go-ahead. Staging is the only deployed env; deploy via infra/deploy/build-and-ship.sh → infra/deploy/deploy-staging.sh when authorized.

Verification (after an authorized staging deploy)

Send two turns in a staff conversation, then start a fresh conversation reusing the same session id (or run the harness twice), and query the observatory DB:

SELECT ts, trace_id FROM ai_calls
WHERE engine='ahu-chatbot' AND ts > now() - interval '15 minutes'
ORDER BY ts;

Confirm: every distinct turn has a distinct trace_id (unique suffix), no two unrelated turns share one, and all sub-calls within a single turn still share one trace_id.

Acceptance criteria

• [ ] X-Request-Id = ${sessionId}-t${turn}-${nonce}, unique per turn-execution, identical across all sub-calls of that turn.
• [ ] Idempotency-Key unchanged / still deterministic.
• [ ] Two runs with the same (session, turn) → different trace, same idempotency base (test proves it).
• [ ] pnpm check green; spec / plan / completion notes present.