AHU AI Platform — Team Handoff (2026-07-07)

One page to orient a new engineer: what each repo is, what runs where, and how it all connects. Canonical wire contract: ahu-gpu-manager/docs/CONVENTIONS.md v1.0 — read it before touching any model/OCR call path; on any conflict it wins.

The one-paragraph story

Three AI engines (chatbot, OCR, doc-classifier) serve users. As of 2026-07-07, every AI call they make — LLM chat, embeddings, OCR jobs, Azure DI, document classification — flows through one Go gateway (ahu-gpu-manager, :8200) that enforces priority queueing per GPU upstream, on-prem policy, idempotency, and emits one audit event per call onto a Redis stream. The observatory (ahu-ai-observatory, :8300) ingests that stream into TimescaleDB and serves a query API; the dashboard (ahu-observatory-dashboard, :8320) renders it as role-aware BI. Engines are dormant-integrated: unset the gateway env vars and each engine talks to its backends directly again (instant rollback).

Repositories (GitHub org: Virtue-Digital-Indonesia)

Model servers on ai-ahu (not repos — deployed containers the gateway fronts): ahu-vllm :8001 (Qwen3.6-35B-A3B-FP8, the main LLM), ahu-cleanup-llm :8003 (3B cleanup model), tei-qwen3-embed :8100 (Qwen3-Embedding-4B), plus the gpu-server's own vLLM VLM (qwen-vlm). External: DashScope qwen3.5-397b-a17b (dev-only, class: external_dev).

Topology

Audit / BI data flow

Key operational facts

• Host: ai-ahu = 192.168.83.20 (2× H100 NVL 96 GB, 12 cores, 122 GB RAM, disk ~92% full). SSH key auth as efran. Host nginx serves x056.ahu-azure.val.id (+ OCR demo vhost). Deploy survey: ahu-gpu-manager/docs/deploy/2026-07-04-gpu-host-survey-and-deploy.md.
• Deploys are compose-based per service: gateway ~/ahu-gpu-manager/deploy (project ahu-platform), dashboard ~/ahu-observatory-dashboard/deploy, OCR ~/ahu-ai-staging/ocr/infra (project ahu-ocr-staging, env in env/shared.env, ship script deploy/build-and-ship.sh).
• Engine→gateway wiring is env-only (rollback = restore one env line + recreate): chatbot MODEL_GATEWAY_URL + EMBEDDER_BASE_URL; OCR GPU_SERVER_URL, AZURE_ON_PREM_BASE_URL, CLASSIFIER_URL (all → http://192.168.83.20:8200; backups shared.env.bak-*). GPU_SERVER_DIRECT_URL keeps cache-ops direct.
• Queue semantics (CONVENTIONS §3): 429/503 + Retry-After = graceful degradation, not an error; job polls distinguish queued vs processing; 404 {"code":"JOB_UNKNOWN"} → resubmit with the same idempotency key.
• Per-engine integration prompts (how each engine was wired, and how to wire the next one): ahu-gpu-manager/docs/integration/.

Update (2026-07-07, later): PaddleOCR flip + P2.8 cache + BI additions

• PaddleOCR is now the LIVE OCR engine for all per-doc-type extraction (Azure DI is per-page billed; paddle is free on-prem). Flipped via env: OCR_LAYOUT_PROVIDER=paddleocr + KTP/NPWP/CONTACT_INFO/DOMISILI/BUKTI_SETOR/SP_PENDIRIAN_PP/PASSPORT_EXTRACTION_SOURCE=paddleocr + PADDLE_OCR_URL={gateway} + CLEANUP_LLM_URL={gateway}/v1 (3B cleanup for paddle card paths, model cleanup-3b). The Azure DI façade stays deployed as an instant fallback (revert = shared.env.bak-prepaddle). Watch extraction quality per doc type.
• P2.8 response cache (tag p2.8): the gateway caches 200s for cache_ttl_s-enabled sync façade paths (live: paddle /layout, classifier /api/classifier/classify, 7d TTL) keyed by tenant + operation + query + a multipart-aware content fingerprint — document rescans skip queueing and upstream work entirely (verified 2.7ms vs 71ms) and audit as cache_hit.
• BI: timestamps + chart tooltips render fixed GMT+7 (WIB); /api/summary now returns per-row p50/p95/p99 plus an exact ungrouped overall distribution (fleet tile on the executive view); dashboard-native alert banners (error-rate spike ≥20% / queue saturation ≥10s over a 15-min window).

Update 2 (2026-07-08): BI buildout + GPU telemetry

• Dashboard now has 4 views (all roles): Overview, SLA & Throughput (fleet p50/p95/p99 from an exact ungrouped aggregate, per-engine p95 trends, error-rate/queue trends, per-GPU utilization panels), Analytics (calls by operation/model, tokens by engine, cache-hit rate, per-tenant rollup), Report (print/PDF operational report + CSV exports, WIB).
• DCGM GPU telemetry (Health Monitoring): the observatory scrapes the host dcgm-exporter (:9400) every 15s into gpu_samples and serves /api/gpu/now + /api/gpu/series; the dashboard raises a GPU-saturation banner at ≥95% util.
• Observatory API: /api/series now supports group_by=operation|status and returns per-bucket p95_total_ms + cache_hits; /api/summary rows and overall carry cache hits; audit cache_hit is ingested (migration 002).
• Success-status vocabulary is {ok, completed} on every view (job-path OCR calls report completed).

Update 3 (2026-07-08): GPU-health enforcement (tag p2.9)

The gateway now couples real GPU health (host DCGM :9400) into routing + admission — the enforcement half of Health Monitoring (the visibility half shipped in Update 2):
- Gateway /metrics exposes gateway_gpu_{util,mem_used_pct,temp_c,faulted} per GPU + gateway_gpu_shed_total.
- Fault-coupling LIVE (couple_faults: true): a thermally-critical GPU (≥90°C) drops its pinned upstreams from the endpoint rotation (fail-open on stale/missing DCGM data — never drops on no-data). Not firing today (both GPUs 40°C, XID 0).
- XID coupling opt-in (couple_xid: false): DCGM XID_ERRORS latches benign codes, so XID-as-fault is deliberately off until the live XID stream is understood — thermal is the safe default signal.
- Batch-shed built but OFF (shed_batch: false): when enabled, a sustained-saturated (≥97% × 3 polls) upstream sheds BATCH-class admissions (429 GPU_SATURATED); interactive/system are never shed.
- GPU→upstream pinning (verified): gpu 0 → qwen-35b; gpu 1 → cleanup-3b, tei-embeddings, gpu-server, doc-classifier-svc. Per-upstream gpu_ids: in gateway.yaml; dormant for any upstream without it.

Update 4 (2026-07-08): Security analytics — the "catch bad actors" layer

• Observatory: /api/actors (per-actor behavior rollup: calls, errors, token egress, off-hours WIB activity, ops/model breadth — actor = engine/surface/user, blank user = "unattributed" until X-User-Id is enforced), /api/security-warnings (24h rule evaluation: token-egress outlier vs peer median [critical], error/refusal probing, volume outlier, off-hours activity, bulk body-reading — all evidence-citing, tiny-sample guarded), /api/body-access-stats (auditor-only watcher rollup). Gated operator+auditor.
• Dashboard: new Security view (operator+; hidden from executive) — warning banners with explicit all-clear, the per-actor table (toned error-rate/egress/off-hours), the auditor-only "watchers" panel, and an unattributed-share tile that makes the identity gap visible.
• Live on day one it flagged real traffic: an egress outlier + an error-probing pattern on the chatbot-internal actor (test traffic — but exactly the detection path a real incident follows: banner → actors table → Calls explorer → body inspection → verify-chain).
• To sharpen it: enforce X-User-Id + tenant tokens (roadmap #5) so actors resolve to real people; later, post-hoc body content-scanning (LLM-as-judge) for exfil/PII patterns.

Update 5 (2026-07-08): Gateway-proxied RAG-search route (tag rag-proxy-v1)

• Why: post production-split, the public chatbot (DMZ) calling ai-ahu-document-rag (Internal) directly would cross the zone boundary. Migration decision #1 chose a Gateway proxy over a firewall exception. This is that route.
• What: a new upstream type rag-http lets the gateway's existing generalized sync-facade adapter proxy RAG's POST /search verbatim (same SearchRequest/{query,hits[]} shape), audited as operation=search (new value, added to CONVENTIONS §5 — observatory stores operation as free-text, ingests/groups it fine). Reuses all existing sync-facade machinery: tenant resolution, on-prem enforcement, pool admission, priority (search defaults to interactive), verbatim relay, audit emission. Zero new handler code — the change is a config-validation widening + one audit constant. e2e test TestRAGSyncFacadeSearchEndToEnd proves forward+audit; opus adversarial review returned SHIP after one doc-drift fix.
• Dormant until configured: with no rag-http upstream in deploy/gateway.yaml, the gateway is byte-identical to before.
• ENGINE FLIP DONE + VERIFIED 2026-07-08: chatbot landed the header work (commits cf49dbb+bed6b29, scoped RAG_SEARCH_URL seam so RAG_BASE_URL still serves /admin/ingest + admin search direct); audited PASS (trace sharing, deterministic idempotency key, 429/503-vs-500 backpressure, dormancy, 5 tests). Flipped the live env RAG_SEARCH_URL=http://192.168.83.20:8200 in ahu-ai-staging/chatbot/infra/env/public.env (backup public.env.bak-pre-ragflip), recreated only public-web (agent untouched). Verified: a real public Tanya query produced one turn where the operation=search (ai-ahu-rag) event and the synthesis chat events SHARE ONE trace_id — retrieval + answer stitched. Rollback: remove the env line, recreate public-web.
• RAG /sessions cross-zone path CLOSED BY DELETION 2026-07-08 (commit f42e355): investigation found it dead (RAG endpoints are stubs returning []; thread history moved to public-web's local ThreadStore; the UI gates the call to Akun-only so guests never reach it). Rather than build a gateway GET-proxy for a dead path, the chatbot's two api/threads/doc routes had their cookie-less RAG passthrough removed (prompt-ai-ahu-chatbot-drop-rag-sessions.md) — cookie-less callers now return empty directly, no DMZ→Internal fetch. Behavior-neutral (guests already got empty); Akun history unchanged. threads-upstream lib KEPT (still used by internal-web staff Data-Agent threads). This completes the RAG cross-zone boundary work: /search proxied+live, /sessions closed.
• Engine flip (hands-off): docs/integration/prompt-ai-ahu-chatbot-rag-proxy.md — point the public orchestrator's RagTool base URL (RAG_BASE_URL) at the gateway + add the §2 headers (the /search path and body are unchanged). Also flags the browser-direct NEXT_PUBLIC_DOC_RAG_API_URL path as a separate DMZ-entry question.
• DEPLOYED LIVE 2026-07-08 13:33 UTC: ai-ahu-rag upstream added to the live deploy/gateway.yaml (backup: gateway.yaml.bak-pre-rag), endpoint http://192.168.83.20:8110 (host-published — docker DNS won't resolve across networks), gateway rebuilt + restarted (9 upstreams, 3 sync paths). End-to-end smoke verified: real POST /search through the gateway returned live corpus hits in 39 ms, and the observatory recorded operation=search · engine=ahu-chatbot · surface=public · model=rag-search · upstream=ai-ahu-rag · traffic_class=interactive · status=ok with the request's trace_id. Post-restart LLM path re-verified (200 on /v1/chat/completions). Remaining: the engine-side flip via prompt-ai-ahu-chatbot-rag-proxy.md (RAG_BASE_URL swap + headers).

Update 6 (2026-07-09): EWS out-of-band notifications + per-person identity gap-closing prompt

• Observatory EWS notifier (tag obs-ews-notify-v1, DEPLOYED DORMANT): a background loop evaluates the same security + early-warning warnings the dashboard shows, and dispatches NEW/escalating ones to email (SMTP) + webhook with dedup (per stable warning subject), escalate-once, and severity routing. Config block notifications: in observatory.yaml; secrets (SMTP password, webhook token) come from ENV by name (password_env/token_env), never yaml. Dormant until enabled:true + ≥1 channel — byte-identical startup otherwise. Adversarial review caught + fixed a critical fingerprint-storm bug (was hashing volatile evidence values → an email every tick; now hashes ID|Metric|Subject only, regression-tested), plus dedup-window/interval bound validation, ctx-bounded SMTP (no goroutine/socket leak), and recover() so a bad tick can't crash the observatory. Fills EWS Notification (40→~85) + Escalation (0→~75). To activate: add a notifications: block to the live observatory.yaml, set SMTP_PASSWORD/WEBHOOK_TOKEN in the compose env, docker compose restart observatory — Efran's call (real recipients).
• Escalation → self-hosted Keep (keephq), DEPLOYED + LIVE + verified end-to-end 2026-07-09: decided against a bespoke escalation UI (reinvents PagerDuty/OnCall); then found Grafana OnCall OSS was archived Mar 2026, so pivoted to Keep (maintained, on-prem). Keep runs on ai-ahu (VPN-only: UI 192.168.83.20:8330, API :8331, ws :8332; ~/keep-deploy/compose.keep.yml, SQLite in ./state, telemetry off, NO_AUTH). The observatory EWS notifier is enabled + wired (notifications.webhook.url → http://192.168.83.20:8331/alerts/event, min_severity=warning, every 5m, webhook-only so Keep owns escalation/ack); the webhook emits Keep's AlertDto (tag obs-keep-webhook-v1) with our stable fingerprint → Keep dedups + auto-resolves. Verified: real sec_error_probing + sec_volume_outlier warnings landed in Keep, grouped by fingerprint; firing→resolved confirmed. Runbook + hardening backlog (enable auth [top], Postgres, pin images, Keep outbound providers, edge DNS): ahu-ai-observatory/docs/deploy/2026-07-09-keep-deployment.md. Escalation policies / on-call schedules / ack are configured in the Keep UI.
• Identity per-person (security view): platform side (observatory + dashboard Security view) done + live; public chat now resolves to real users. Two gaps remain engine-side (internal staff surface 66/68 blank; embeddings unattributed) — closed by prompt docs/integration/prompt-ai-ahu-chatbot-identity-internal-embed.md (hands-off). RAG quality monitoring is parked pending Efran's Fable signal.

Update 7 (2026-07-09): RAG Quality Monitor — LIVE (tag obs-rag-quality-v1)

• What: post-hoc grades the PUBLIC chatbot's answer quality from the audit trail — zero chatbot changes. The observatory reads each synthesis turn's stored body (question + inlined HASIL ALAT retrieved context + answer), computes deterministic metrics (had_context, num_passages, top_score, refused, parse_ok) + an on-prem LLM-judge (via the gateway, X-Priority: batch, deterministic idem-key) scoring faithfulness / answer-relevance / context-relevance. Stored in rag_quality; served at /api/quality/{summary,recent,series}; new "RAG Quality" dashboard view (operator+); degradation folds into the EWS→Keep pipe (rag_low_faithfulness, rag_ungrounded_spike).
• Verified live: 4 real public turns graded (faithfulness 1.0, answer 1.0, context 0.95 w/ Indonesian reasons); parse_ok_rate=1.0 (synthesis-scoped, excludes planning calls). Dormant unless rag_quality.enabled.
• Adversarial review (opus) caught + fixed before enable: judge 429/503 backpressure was terminal → permanent grading holes (now retryable, left un-graded for next tick, per CONVENTIONS §3); parse/refusal rates diluted by non-synthesis planning rows → added is_synthesis column + synthesis-scoped denominators; grader body reads now recorded in body_access_audit (PDP, actor rag-quality-evaluator); config validation; question truncation.
• Deploy gotchas (documented): (1) judge model must be the FULL vLLM id Qwen/Qwen3.6-35B-A3B-FP8 — the qwen-35b gateway alias 404s (vLLM only accepts its real id). (2) Qwen3 is a reasoning model — the judge sends chat_template_kwargs:{enable_thinking:false} or it burns the token budget on thinking and emits no JSON. (3) docker compose up -d after build may NOT recreate the container with the new image — use --force-recreate. (4) YAML: quote marker values with trailing colons (question_marker: "PERTANYAAN:").
• Coverage caveat: grades the orchestrator synthesis path (which inlines context into the prompt) — covers public answers broadly. Phase 2 (separate specs): staff/internal surface + a golden Q&A regression set on the same judge harness.

Update 8 (2026-07-10): Golden Q&A regression set — LIVE (tag obs-golden-v1)

• What: RAG Quality phase 2b — a goldeneval CLI (in ahu-ai-observatory/cmd/goldeneval) that drives the public chatbot with a curated question set, scores each answer (a deterministic must_include/must_not_include fact gate + the on-prem LLM judge from phase 1), writes golden_results, and gates deploys (non-zero exit below pass-rate, on a verified-item regression, or on an unverified set without -allow-unverified). GET /api/quality/golden + a "Golden set" panel on the dashboard RAG Quality view show the latest run.
• Why not live-grade the internal chatbot: the internal/staff chatbot is a text-to-SQL data agent, not doc-RAG — its real risk is query correctness (wrong SQL, faithfully restated), which live faithfulness can't catch. A golden set with SME-verified answers is the right gate for both surfaces; internal live-faithfulness grading was deprioritized.
• HONESTY: the shipped golden set is an example (verified:false) — the harness + a starter set + an SME guide (docs/golden-set-guide.md); the authoritative Q&A content is AHU SMEs' to author/verify. Both the CLI report and the dashboard flag an unverified set as non-authoritative, and an unverified set can't exit-0 green.
• Verified live 2026-07-10: a run drove 7 example items through the real public chatbot; the deterministic gate correctly FAILED an item the judge scored 1.00 (missing required phrase) — proving the fact-floor is judge-independent; 1 genuine pass; below-gate → exit 1 + non-authoritative banner; rows + dashboard panel populated.
• Adversarial review (opus) caught + fixed: regression gate was counting transient (429/503 backpressure) errors as "verified regressions" → would spuriously block deploys; a deterministic fact-fail was discarded when the judge call errored → false green; an all-unverified set could exit 0. All fixed + tested.
• Deferred: internal data-agent golden drive (needs the internal query endpoint + drift-stable questions); CI/deploy-hook wiring (the binary is the integration point).

Deferred roadmap (updated 2026-07-08 — build next, in this order)

1. ~~Predictive Intelligence Engine harness~~ — DONE (2026-07-08, tag obs-predict-v1). Folded into the observatory (Go) — not a new service — querying TimescaleDB directly (no new container/token/HTTP hop). New pure internal/predict package (Holt-Winters -> Holt -> drift -> naive forecasting, z-score/IQR anomaly, evidence-citing early-warning rules) + internal/store.QueryMetricSeries (recombination-safe reductions) + endpoints /api/forecast, /api/anomalies, /api/early-warnings (metric allowlist rejects percentiles). New dashboard "Prediksi" view (/forecast). Honest throughout: insufficient_data over fabrication; success vocabulary {ok,completed}. Moves Time Series Forecasting / Anomaly Detection / Early Warning System off 0%. Data source is the only DWH seam.
2. Enable shed_batch once saturation behavior is observed; enable couple_xid after triaging the live XID stream.
3. GPU Auto-scaling Engine MVP (readiness sheet: 20%): a scaler loop over the signals we already record (queue depth, admission wait, 429 rate, DCGM util) driving the levers this fleet actually has — dynamic pool resizing, model warm/cold on the second H100, load-shedding hints — and emitting scale-recommendation metrics a future k8s HPA can consume. Design constraint: single-box until the new GPU servers arrive.
4. Gateway-registry GPU-health coupling: mark an upstream degraded (stop admitting batch) when its GPU is saturated — the enforcement half of Health Monitoring (the visibility half shipped).
5. Per-tenant fair-share quotas (Resource Scheduler): weighted-fair admission within a pool so one tenant cannot monopolize slots; per-tenant daily token/call budgets with 429 QUOTA_EXCEEDED.
6. Scheduled reports: the /report view emailed daily/weekly (needs an SMTP/channel decision); plus report presets per role.
7. Prod-hardening flip (unchanged, awaiting new GPU servers): synthesis → on-prem, allow_external_upstreams: false, tenant bearer tokens, HA pair (coordination: redis).
8. RAG quality monitoring (unchanged): chatbot-side telemetry → RAGAS-style eval harness → golden-set regression.

Known gaps / next steps (in priority order)

1. Prod hardening flip (config, awaiting go): live gateway still has allow_external_upstreams: true and synthesis → ext-dashscope-397b (cloud). Prod = repoint synthesis to on-prem qwen-35b + set false.
2. Tenant tokens: tenants are header-trust (token: ""); real bearer tokens exist in the machinery but aren't issued.
3. HA: coordination: local — single gateway instance; redis-backed coordination is built, not deployed. Server specs for MVP/HA already sent to DevOps.
4. RAG quality monitoring: observatory sees calls, not RAG semantics (no retrieved-doc ids / groundedness). Planned next initiative — starts with chatbot-side telemetry.
5. Small logged follow-ups: gateway façade events lack upstream_class; Azure completion traffic_class mirrors the poll; streaming relay for very large Azure results; dashboard chart hover-tooltips still Carbon-default time (axes are WIB); p95/p99 percentiles + alerting.