think
16px
820px
← all files

Plan C Completion Notes — Polish & Backlog (2026-07-02)

Companion to 2026-07-01-monorepo-split-plan-c-polish-and-backlog.md. Records what
shipped, what broke along the way, and what is deliberately deferred to Plan D.

Phase summary

Phase Scope Outcome
A — RAG unblock Locate corpus, verify Milvus/embedder, ingest, live smoke Shipped. Live prod answers real Indonesian legal questions with references (93 deltas on smoke).
B — Admin scope routing /admin/{public,internal,shared,observe} layouts, ScopeBadge/ScopePill, audit log, jobs API Shipped. Audit sqlite at /data/audit.sqlite (table audit); first mutation row: efran@val.id / shared / job.enqueue / queue:nightly-eval:1.
C — SSE resume Redis checkpoints + Last-Event-ID replay Shipped. Verified live: interrupt at id 70 → replay resumes at 71. Known nuance: reconnecting after done re-runs orchestration; acceptable because real clients stop on done.
D — Gateway headers X-Tenant-Id / X-Surface / X-User-Id / X-Priority on LLM calls Shipped.
E — BullMQ workers nightly-eval + knowledge-reingest queues, esbuild CJS bundle, ahu-workers compose stack Shipped. End-to-end proven: enqueue via /api/admin/jobs → ahu-ai-workers consumes → returnvalue stored.
F — sql-guard TS port Validator + tests Shipped. 10/10 vitest (within 47-test public-web suite) + 3/3 pytest isolation. Runtime wiring into DataTool deferred (Plan D).
G — Dual-model streaming native-provider single/dual-mode pipeline Shipped. Mode detection via RunIntermediateContent / OutputModelResponseStarted; single-model buffers to stream_finalize, dual-model streams live.
H — Eval env unification Evals read orchestrator env schema Shipped. MODEL_REGISTRY now targets Qwen/Qwen3.6-35B-A3B-FP8 via MODEL_GATEWAY_URL; LLM_API_KEY_LOCAL falls back to PLANNING_API_KEY, OPENAI_API_KEY to SYNTHESIS_API_KEY.
I — Physical rename ai-ahu-chatbot → ahu-ai-chatbot Shipped (user-approved 2026-07-02, 9f45e73). ahu-ai-chatbot is the real dir; compat symlink ai-ahu-chatbot → ahu-ai-chatbot stays until no session/shell pins the old path.

Bug log (what broke and how it was fixed)

  1. Test Reference shape drift — orchestrator tests used old {title, source};
    unified to {name, meta_data:{source}} (0ca558d).
  2. Eval env skips — LLM_API_KEY_LOCAL missing → added PLANNING_API_KEY=EMPTY
    to shared.env (vLLM accepts any bearer).
  3. Agents couldn't reach MySQL — deployed agents lacked DB_AHU_*; added to
    public.env/internal.env (Phase-1 caveat: public borrows internal read-only creds
    until a public replica exists) (ba8678d, templates only).
  4. Retired model 404 — vLLM now serves Qwen3.6-35B; eval registry updated (2e04fdf).
  5. Staff surface exposed — app auth is client-side only (localStorage JWT, mock
    HS256 signer; middleware does not cover /api/admin/*). Publishing the staff
    subdomain changed the threat model: admin APIs answered unauthenticated from the
    internet. Stopgap: nginx Basic Auth on the staff vhost (f03ad45). Server-side
    session auth is Plan D.
  6. htpasswd written empty — here-string overrode the pipe as stdin for
    sudo tee; fixed by temp-file + sudo cp. (401-with-creds symptom.)
  7. Worker 422 — nightly-eval worker POSTed JSON; Agno /runs expects multipart
    FormData. Fixed to FormData, same shape as NativeProvider (5af260f).
  8. Agent direct runs said "OPENAI_API_KEY not set" — with no /data/provider.json
    the agent fell back to OpenAIResponses(gpt-5.2) because _resolve_model only read
    legacy LLM_BASE_URL vars while deployed env carries
    MODEL_GATEWAY_URL/LLM_PLANNING_MODEL/PLANNING_API_KEY. Wired unified-env
    fallback for both planning and synthesis tiers; refreshed stale 27B defaults (5af260f).
  9. native-provider tests unblocked — first attempt died on hand-rolled reader
    mocks; rewritten with real Response/ReadableStream (Node 22 natives). 8/8
    passing (d7ca9aa): think-split buffering, dual-mode detection, references,
    chunk-boundary reassembly, FormData shape, HTTP errors.
  10. MySQL DB name case — eval run surfaced Unknown database 'ahu_badan_hukum';
    the real name on 192.168.72.101 is AHU_BADAN_HUKUM (MySQL on Linux is
    case-sensitive). Env fixed + agents recreated (e2b34b2).

Success criteria status

  1. ✅ Live prod returns real Indonesian legal answers (RAG loaded)
  2. ✅ Admin scope routing + ScopePill
  3. ✅ Audit log has ≥1 mutation (job.enqueue row verified)
  4. ✅ SSE resume via Last-Event-ID (verified live)
  5. ✅ Gateway headers on LLM calls
  6. ✅ BullMQ job runs via /admin/shared/jobs (see verification below)
  7. ✅ sql-guard tests green
  8. ✅ native-provider single/dual-mode test coverage (8/8)
  9. ✅ In-container eval runs without env skips (see verification below)
  10. ✅ Physical rename complete (user-approved; compat symlink retained until live sessions close)

Deferred to Plan D

  • Server-side session auth for the staff surface (replace edge Basic Auth stopgap).
  • sql-guard validator wiring into the orchestrator DataTool runtime path.
  • Durable ahu-net attach for ai-ahu-rag (declare external network in its compose;
    current docker network connect does not survive recreate).
  • Local embedding endpoint for dash knowledge hybrid search (embedder currently
    warns on OPENAI_API_KEY and returns 0-dim).
  • Public DB replica → flip PUBLIC_DB_URL, retire shared read-only creds.

Operational notes

  • Deploy loop: ./infra/deploy/build-and-ship.sh → ./infra/deploy/deploy-staging.sh.
  • Staff surface: https://x056.ahu-demo.chatbot-neo-staff.val.id behind Basic Auth
    (user ahu-staff; password on the edge box at /tmp/staff-basic-auth.txt).
  • Ports 3500/3510 bind to 192.168.83.20, not 127.0.0.1 — health checks must use the LAN IP.
  • Audit store: sqlite /data/audit.sqlite, table audit (not audit_log).
  • Agent model resolution precedence: /data/provider.json → unified env
    (MODEL_GATEWAY_URL + LLM_PLANNING_MODEL / SYNTHESIS_GATEWAY_URL +
    LLM_SYNTHESIS_MODEL) → legacy LLM_BASE_URL → OpenAI (dev only).