Verification prompt — observatory audit trail for `ahu-chatbot`

Paste into a Claude Code session that can reach the deployed platform stack on
the GPU host (ai-ahu · 192.168.83.20) — ideally run inside the
ahu-ai-observatory repo, with ssh access to the host for docker/curl.
This is a verification + gap-report task, not a code change. Read
ahu-gpu-manager/docs/CONVENTIONS.md (audit schema §5, headers §2) first.

Context

The ai-ahu-chatbot engine is fully flipped onto the gateway (:8200):
planning, synthesis (/synthesis), knowledge, and embeddings all route
through it, and it sends the full §2 header set (X-Tenant-Id: ahu-chatbot,
X-Surface, X-User-Id, X-Request-Id, X-Priority, Idempotency-Key).
Two recent chatbot changes make the audit trail complete:

Embedding calls are now attributed (client_params.default_headers) —
previously engine=unknown for retrieval traffic.
Per-turn X-Request-Id now rides the agent fan-out too (orchestrator +
public_dash/data-agent model calls + embeddings), so one user turn is one
trace_id. Derivation: ${sessionId}-t${turn}.

The gateway emits one audit event per AI call to Redis Stream ahu.ai.audit;
the observatory ingests it (consumer group, starts at 0 so backlog is
captured) into Timescale and serves it on the query API (:8300) + dashboard
(:8310).

Timing: chat attribution (planning/synthesis) has been live since the
2026-07-04 flip and should already be visible. Embedding attribution + per-turn
trace grouping across the agent fan-out appear only after the chatbot's next
deploy — if that hasn't shipped yet, note it and check what you can.

What to verify

1. The pipeline is alive

Gateway is emitting: redis-cli XLEN ahu.ai.audit grows under traffic; the
gateway's /metrics shows gateway_requests_total{tenant="ahu-chatbot"}
climbing.
Observatory is deployed and consuming: curl -s :8300/healthz → ok;
:8300/metrics — the consumer-group unacked/pending gauge stays low
(not growing unbounded = the ingester keeps up); no poison-message spikes.
If the observatory stack isn't up, bring it up per its README
(deploy/observatory.yaml tokens set; needs the gateway stack's
ahu-platform_default network) and report that it was missing.

2. Chatbot events land, correctly attributed

Using an operator/auditor token (Authorization: Bearer <token> from
deploy/observatory.yaml):
- GET /api/calls?engine=ahu-chatbot&limit=20 returns recent events. Confirm
surface is public/internal as expected, upstream is qwen-35b
(planning/knowledge) and ext-dashscope-397b (synthesis), status ok.
- GET /api/summary?engine=ahu-chatbot (and /api/series) show non-zero
volume with a sane latency/error breakdown.

3. The two new attributions (post-deploy)

Embeddings: filter calls for operation=embed (or upstream=tei-embeddings).
They should carry engine=ahu-chatbot — not unknown. unknown
embedding calls = the chatbot embedder-header change hasn't deployed yet
(or EMBEDDER_BASE_URL isn't the gateway). Report which.
Per-turn trace grouping: pick one recent chatbot trace_id and
GET /api/calls?trace_id=<id>. A single public turn should group the
orchestrator's planning + synthesis calls and the public_dash agent's
model call(s) and the embedding call under one trace_id. If agent calls
sit under their own random trace ids instead, the per-turn X-Request-Id
isn't reaching the agent — report it (expected only after the chatbot deploy).

4. It's visible to a human

Dashboard (:8310) renders: an engine/tenant filter includes ahu-chatbot,
and a trace drill-down shows the grouped fan-out from §3. If the dashboard
can't filter by tenant or open a trace, that's the gap that makes the audit
work unusable — flag it.

Report back

Pipeline health (stream growing, ingester lag, observatory + dashboard up?).
Chat attribution present? Embedding attribution present or still unknown?
Does one turn's trace_id group orchestrator + agent + embedding calls?
Any schema/field mismatch between what the gateway emits and what the
observatory stores or the dashboard shows (e.g. engine vs tenant naming,
missing trace_id).
Concrete next actions split by owner: gateway (ahu-gpu-manager),
observatory (ahu-ai-observatory/dashboard), or chatbot (needs a deploy).

Verification prompt — observatory audit trail for ahu-chatbot