think
16px
820px
← all files

Plan D Completion Notes — Deferred Backlog (2026-07-02)

Companion to 2026-07-02-monorepo-split-plan-d-backlog.md. Records what shipped,
what broke along the way, and what remains deliberately deferred. Deploy order was
D3 → D4 → D2 → D1; all 17 tasks and the final checklist are green.

Item summary

Item Scope Outcome
D3 — Durable RAG net attach ai-ahu-rag compose on Server 2 declares ahu-net external Shipped (Server 2 compose edit; backup compose.yaml.bak-plan-d). --force-recreate preserves the attach; public doc query green afterwards. No repo commit — the compose lives on Server 2.
D4 — Local embeddings Env-gated TEI embedder (tei-qwen3-embed) in both agents + pgvector re-ingest Shipped (f39147c, 1f8c463, dde0527). Re-ingested at 2560 dims via load_knowledge --recreate; hybrid search returns non-zero scores through TEI; agent logs clean (2aacf95).
D2 — sql-guard enforcement sqlglot pre-execution guard in the Python public agent, SURFACE=public gated; public stack flipped to public_dash via DASH_VARIANT=public Shipped (f711af6, 079988c, cf8225e, 6c7b348). 23 in-container pytest green; live forbidden pattern refused; live aggregate passes with per-year table (2024: 171.943 / 2025: 220.839 / 2026: 127.055) matching a direct DB probe; run_sql_query confirmed in session tool_calls. Staff surface untouched (data-agent + reasoning-data-agent still registered, spot-check run COMPLETED).
D1 — Staff server-side auth SQLite staff_users + bcrypt, httpOnly JWT cookie (ahu_staff_session, HS256, 8h) with tv (token_version) claim, Edge-safe middleware on /admin/* + /api/admin/*, Node-side mutation guard re-checking token_version, users CRUD API + /admin/shared/users page, login rewired to the server route, mock accounts deleted Shipped (0171258 → 721c370). Live-verified from the internet: no-cookie 401 / login 200 / with-cookie 200; disabled user's unexpired session gets 401 "Sesi tidak berlaku lagi" on mutations while reads render until expiry; audit rows (user.create, auth.login, user.update, user.disable) confirmed on /data/audit.sqlite.
Cutover — Basic Auth removal Comment out auth_basic on the staff vhost (local nginx edge) Shipped (499d9af). Backup at sites-available/x056.ahu-demo.chatbot-neo-staff.val.id.pre-plan-d-backup; htpasswd retained. Post-cutover internet checks: no-cookie 401, login page 200, with-cookie 200, page redirect 307.

Bug log (what broke and how it was fixed)

  1. Orchestrator returned all-empty answers while the agent run was healthy (D2) —
    evaluate.ts sliced the data-tool response with .slice(0, 800) and compose.ts
    with .slice(0, 1200). The public_dash agent streams inline reasoning before
    its final answer (all as RunContent deltas), so the judge and synthesis only ever
    saw reasoning, never the conclusion. Fixed via TDD to tail slices
    .slice(-800) / .slice(-1200) (cf8225e).
  2. Misdiagnosis trap en route to (1) — a POST access-log line 0.5s after request
    start suggested the agent "completed instantly" (suspected guard short-circuit).
    uvicorn logs access lines at response start, not completion; reconstructing the
    SSE deltas showed the run had succeeded with the full table.
  3. DataTool spoke the wrong wire format (D2) — the orchestrator's DataTool needed
    AgentOS multipart form fields + RunContent event parsing to talk to the agent
    (079988c).
  4. Transient Error during hybrid search: 'op' — benign agno filter-DSL KeyError
    when the model emits an unsupported knowledge-filter; retries follow. One post-fix
    live run failed on it; subsequent runs passed. Known noise, not a regression.
  5. Turbopack dev panic — pnpm dev in internal-web ("Next.js package not found").
    Workaround for all local smoke tests: ./node_modules/.bin/next dev -p 3501
    (no turbopack; 3501 avoids the staging-port convention on 3500).
  6. Guard-insertion script corrupted a multi-line import — the Task 12 script that
    edited all 16 admin routes inserted the guard import inside anon-limit/route.ts's
    multi-line import { ... } from "@/lib/anon/limits". Caught immediately by
    tsc --noEmit; moved below the import block. The other 15 files were clean.
  7. test:streams pointed at pre-split paths — internal-web's script referenced
    scripts/test-*.ts relative to the app (they live at repo root and import the old
    root src/, which moved to packages/streams in Plan A). Repointed the script at
    the migrated packages/streams vitest suite, 8/8 (9f4d926).
  8. Live verification 401'd at nginx, not the app — /tmp/staff-basic-auth.txt
    holds STAFF_BASIC_PASS=<value>, not a bare password; $(cat …) sent the whole
    line as the password. Parse with cut -d= -f2.
  9. First Task 16 build died mid-image — the background build was killed when the
    previous CLI session exited. Re-run completed from docker layer cache.
  10. In-container better-sqlite3 audit check — plain require('better-sqlite3')
    fails in the deployed image; use the absolute pnpm path
    /app/node_modules/.pnpm/better-sqlite3@11.10.0/node_modules/better-sqlite3.
  11. No headless browser on the build box — Task 14's browser check was substituted
    with API-level exercise of every page action (create / role change / disable /
    reset password / last-admin guard) plus SSR-shell parity with the known-good jobs
    page. Task 16 Step 5's browser flow was likewise driven with two curl cookie jars.
  12. No passwordless sudo — the nginx cutover (Task 17 Steps 1–2) was run by the
    operator via a single prepared sudo sh -c '…' command; verified from the session
    afterwards.

Success criteria status (final checklist)

  1. ✅ Internet /api/admin/* without cookie → 401; after login → 200 (post-cutover).
  2. ✅ Disabled user's existing session cannot mutate — token_version verified live
    (scratch admin uji@ahu.local, left disabled by design).
  3. ✅ /admin/shared/users CRUD works, writes audit rows; Basic Auth removed.
  4. ✅ sql-guard pytest green in-container (23); live forbidden refused; allowed
    aggregate passes.
  5. ✅ RAG --force-recreate preserves the ahu-net attach; public doc query green.
  6. ✅ Hybrid search non-zero via TEI; re-ingest via load_knowledge --recreate.
  7. ✅ Suites green: internal-web typecheck + vitest 10/10 + streams 8/8;
    public-web 53/53 (incl. 13 sql-guard cases); shipped via build-and-ship +
    deploy-staging (final tag 721c370).

Still deferred

  • SSO Kemenkumham — by design, SSO replaces only the /api/auth/login route;
    cookie, middleware, mutation guard, and staff_users stay as-is.
  • Public DB replica flip — blocked; no replica exists. Public agent keeps
    borrowing internal read-only MySQL creds (Plan C caveat) until one does.

Operational notes

  • Staff console: https://x056.ahu-demo.chatbot-neo-staff.val.id — session auth only
    (no Basic Auth). Seed admin is admin@ahu.local; password lives in git-ignored
    infra/env/internal.env (STAFF_ADMIN_PASSWORD) and should be rotated via
    /admin/shared/users → Reset sandi after handoff.
  • Seeding only runs when staff_users is empty — rotating the env password later
    does NOT change the DB credential; use the UI reset.
  • Rotating NEXTAUTH_SECRET now logs every staff session out (signer + verifier).
  • Basic Auth rollback: restore *.pre-plan-d-backup over the vhost (or uncomment the
    two auth_basic lines) + nginx -t && systemctl reload nginx; htpasswd file kept.
  • Mutation revocation model: middleware checks JWT signature/expiry only (Edge-safe);
    writes re-check the DB row (requireMutationSession). Read-only staleness is
    bounded by the 8h cookie — accepted trade-off.
  • uji@ahu.local (id 2) is a disabled scratch admin from live verification; user
    delete is unsupported by design (audit trail), leave it disabled.
  • Dev-server smoke recipe: from apps/internal-web,
    STAFF_DB=/tmp/staff-dev.sqlite STAFF_ADMIN_EMAIL=admin@ahu.local STAFF_ADMIN_PASSWORD=dev-admin-123 AUDIT_DB=/tmp/audit-dev.sqlite ./node_modules/.bin/next dev -p 3501.