Full-result data pipeline & rich tables — completion notes

Date: 2026-07-08 · Spec: specs/2026-07-08-full-result-data-pipeline-design.md
· Plan: plans/2026-07-08-full-result-data-pipeline.md · Commit: d2c9184
(admin-UI fixes from the same session: 558148a)

What shipped

Driver was the real thread 019f4001-f31d-738f-8cee-56e6ad2d843e ("jumlah PT
per provinsi dan per kota"): the agent's multi-thousand-row result reached the
user as 5 inline rows + truncated markdown top-N tables, CSV export contained
only those 5 rows, and the answer closed with a "keterbatasan ruang" apology.

Agent (apps/internal-agent)

• app/results_store.py — stdlib-only spill store. run_sql_query persists
  the COMPLETE result (cap RESULT_SPILL_MAX_ROWS=10k) as JSON under
  RESULTS_DIR (default /data/results, named volume), keyed by 32-hex
  result_id; TTL GC (RESULT_SPILL_TTL_DAYS=14) on save.
• dash/tools/sql.py — tool returns {result_id, row_count, columns, truncated, preview[≤200], note}; graceful inline-only fallback when the
  store fails. Public agent tree untouched.
• app/results.py — GET /results/{id} (offset/limit ≤1000) +
  GET /results/{id}/csv (streamed, UTF-8 BOM, RFC 4180 + formula-injection
  guard). Router is dash-free for testability.
• dash/agents.py DATA_VIZ contract: table.result_ref (WAJIB for
  query-backed tables), tables[] for multi-table answers, inline ≤15 rows,
  total = full row_count, ban on space apologies.

Web (apps/internal-web)

• /api/data/results/[id] + /csv — proxies with in-route staff-session
  check (lib/auth/require-session.ts; middleware only covers /admin/*).
• lib/data-table/core.ts — pure engine: id-ID numeric parse ("17.795",
  "+18,4", "1.234,5"), numeric-aware sort (empties sink), all-column filter,
  pagination, server-payload mapping.
• components/chat/data/DataTableCard.tsx — sortable/filterable/paginated
  sticky-header table; "Muat semua N baris" pulls the full server result
  (client cap 5k rows; CSV uncapped); "Ekspor .csv lengkap" downloads the
  complete server CSV. ResultTable is now a thin wrapper (API preserved).
• MdTableCard + lib/data-table/md-extract.ts — ReactMarkdown table
  override upgrades narrative markdown tables to the same interactive card;
  retrofits OLD threads.
• Parser/types: tables[], result_ref, allTables() normalizer.

Verification (staging, live model + prod data)

Re-ran the reference scenario through the deployed data-agent:

• DATA_VIZ carried 2 tables, each with result_ref; per-kota table:
  10 inline rows, 585 on the server, CSV = 585 rows exactly; BOM present;
  narrative clean of apologies.
• Web proxy: 401 without session; with session → 39-row JSON + 586-line CSV
  over HTTPS; traversal attempt → 404.
• UI (headless Chromium, temp staff user, removed after): "10 dari 585 baris
  dimuat" → "Muat semua 585 baris" → pager "1-10 dari 585" with sort ▼ on
  jumlah_pt; old reference thread's markdown Tabel 1 renders as an
  interactive 50-row card with filter + CSV.
• Gates: 59 agent pytest, 183 JS vitest, all typechecks, conventions OK;
  in-image pytest ran in build-and-ship.

Operational notes

• Spill lives in the agent's internal-agent-data volume; results expire
  after 14 days — "Muat semua" on an expired ref toasts a clear
  "kedaluwarsa, jalankan ulang" message. Inline rows always still render.
• Env knobs documented in infra/env/internal.env.example (defaults are
  live values; no staging env edit needed).
• Old threads keep working: no result_ref ⇒ no load-all button, honest
  "N dari M baris dimuat" subtitle.

Deferred

• Grouped/stacked multi-series charts (kategori × provinsi in ONE chart) —
  read the dataviz skill before building; charts[] array plumbing is a
  natural follow-up.
• Settings sub-sections still on ad-hoc Tailwind (Orchestrator, Synthesis,
  DataDash provider) — flagged during the admin-UI fix.
• /api/dash-proxy and /api/threads remain session-unchecked
  (pre-existing); consider extending requireStaffSession there.