Semantic Folder Suggestion (AI smart-filing) — Design

Date: 2026-06-30
Status: Approved (design); pending implementation plan.

Goal

When a user uploads a file, recommend the folder it should go into — ranked by semantic
similarity of the file's content to each folder's existing contents (and an optional admin-set
folder purpose). Suggestion-only (the user accepts or ignores; never auto-files). Delivered as a
new, separately-sellable premium module gated end-to-end by the licensing system.

Approach (decided)

• Mechanism: a pgvector embedding index over documents. A folder's "profile" is the centroid
  of its members' embeddings (blended with the embedding of its admin description). At upload, the
  file's content is embedded and ranked against folder centroids by cosine similarity. (NOT the
  lighter LLM-prompt "auto-profile" approach — chosen for deeper, content-accurate matching; pgvector
  is already in the stack.)
• License: a NEW dedicated module semantic (5th premium module, alongside
  correspondence/watermarking/ai/esign) so it can be sold separately from the chat-style ai module.
  Gated end-to-end. The same index also powers future semantic search (out of scope here).
• Embedder: a swappable Embedder port with three adapters — sidecar (default), openai,
  mock — chosen by EMBED_PROVIDER. Default model: multilingual-e5-small (384-dim,
  Indonesian+English) served via a fastembed/ONNX sidecar (no PyTorch/GPU; small image; CPU-fast).

Components

1. The semantic license module

Add semantic to: the signed-license module set, the module registry (requireModule), the GA
build-pinning, and the UI enabled_modules gate — identical plumbing to the existing four. The demo
license gains semantic so the feature works in the demo. A deployment without the license runs none
of it (no sidecar required, no embeddings written, no suggestions, no UI). The embedding pipeline,
the suggest endpoint, and the UI affordances are ALL gated on semantic.

2. Embedder seam + sidecar

New port:

type Embedder interface {
    Embed(ctx context.Context, texts []string) ([][]float32, error) // batch
    Info() EmbedInfo // { Model string; Dim int }
}

Adapters, selected by EMBED_PROVIDER (default sidecar):
- sidecar — HTTP client to a new embed-sidecar service (FastAPI + fastembed, ONNX-quantized
multilingual-e5-small). POST /embed {texts:[...], kind:"query"|"passage"} → {vectors:[[...]], model, dim} (the sidecar applies the e5 query:/passage: prefixes). Added to deploy/docker-compose.yml
(CPU-only; model baked into / mounted in the image). Mirrors the planned stego-sidecar pattern.
- openai — text-embedding-3-small (cloud; for connected/SaaS deployments).
- mock — deterministic hash→vector (tests/dev). NOTE: mock is non-semantic, so the DEMO runs the
real sidecar to show meaningful suggestions.
- Config: EMBED_PROVIDER, EMBED_SIDECAR_URL, OPENAI_API_KEY/EMBED_MODEL, and EMBED_DIM
(must match the pgvector column + the active model).

Operational constraint (documented): pgvector columns are fixed-dimension. A deployment commits
to one embedder dimension; switching to a different-dim model later requires a re-embed + a dim
migration. The port switches provider freely (sidecar↔openai at the same dim); it does not make dim
a runtime toggle.

3. pgvector index + embedding pipeline

• Migration: CREATE EXTENSION IF NOT EXISTS vector; +
  sql CREATE TABLE document_embeddings ( document_id uuid PRIMARY KEY REFERENCES documents(id) ON DELETE CASCADE, version int NOT NULL, model text NOT NULL, embedding vector(384) NOT NULL, updated_at timestamptz NOT NULL DEFAULT now() ); -- ANN index for cosine search: CREATE INDEX document_embeddings_ann ON document_embeddings USING hnsw (embedding vector_cosine_ops);
• Embed-on-upload: hook the existing version/extraction path (where a document's text is extracted
  for search/AI — the AddVersion → set-content-text flow). After a new version lands AND semantic
  is licensed, embed the extracted text (truncated to a token budget) and UPSERT into
  document_embeddings (version, model, vector). Best-effort + async (do not block the upload). For
  non-text content (images) with no extractable text, embed the title (or skip).
• Backfill sweep: a scheduled job (the established scheduler pattern, like the disposition sweep)
  that embeds documents whose document_embeddings row is missing or stale (older model/version).
  Covers pre-existing docs and the moment semantic is first licensed. Page-capped + logged.

4. Folder centroids + the suggest flow

• Folder profile = the centroid (avg(embedding)) of the folder's member documents, blended with
  the embedding of the folder's admin description (§5) when present; for an empty/sparse folder the
  description embedding carries it. v1 computes centroids on the fly via a SQL aggregate over
  document_embeddings ⋈ documents grouped by folder_id, restricted to the caller's
  write-accessible folders (ACL); materializing centroids (a folder_embeddings table refreshed on
  a sweep) is the documented scale path.
• Suggest endpoint: POST /api/v1/semantic/suggest-folder (requireModule("semantic")). Input:
  the uploaded file (multipart) OR {title, text}. It extracts text ad-hoc from the bytes (reusing the
  DMS extractor; no document is created), embeds it (kind:"query"), ranks the caller's writable
  folders by cosine similarity to their centroids, and returns the top-K
  [{folder_id, path, score}] (e.g. K=3, score = 1 − cosine distance). Empty index / no writable
  folders → empty result (the UI just shows nothing).
• UX: in NewDocumentModal, once a file is picked (and semantic is enabled), call suggest and
  render "Suggested folder: /Contracts/Vendors — 87% match" (top result, with the next 1-2 as
  alternates). Accepting pre-selects that folder in the existing folder picker; ignoring changes
  nothing. Suggestion-only — never auto-files. Also (optional, low-cost) a "Suggested folder" hint +
  one-click move on an already-filed document's detail.

5. Optional admin folder description

• Add a description text column to folders (migration) + an admin/edit affordance ("what this
  folder is for"). The field is harmless core metadata (settable anytime); its use — embedding it
  into the folder profile to sharpen suggestions (especially for new/empty folders) — is
  semantic-gated. When semantic is licensed, the description is embedded on set/change and stored
  in its OWN table folder_description_embeddings(folder_id uuid PRIMARY KEY REFERENCES folders(id) ON DELETE CASCADE, embedding vector(384), updated_at timestamptz). The suggest query blends this
  with the on-the-fly member-document centroid (e.g. average the two when both exist; use whichever
  exists otherwise) before ranking.

6. Phasing (for the implementation plan)

1. Module + embedder seam: add the semantic license module (registry, gating, demo license,
   GA pin) + the Embedder port + the mock adapter + config. (No behavior yet; gating + seam.)
2. Sidecar + pipeline: the embed-sidecar (fastembed e5-small) in compose + the sidecar/openai
   adapters; the pgvector migration + embed-on-upload + the backfill sweep.
3. Suggest: folder centroids + POST /semantic/suggest-folder + the NewDocumentModal chip
   (+ optional doc-detail hint). The demo shows real suggestions after this phase.
4. Admin descriptions: the folder description field + admin UI + its embedding blend.

Out of scope

• Semantic search (the same index would power it later; not built here).
• Auto-filing (suggestion-only by decision).
• Re-embedding/dim migration tooling for switching embedder dimension (documented manual op).
• Multi-replica shared embedding queue (single-node on-prem target; the backfill sweep suffices).

Gating summary (end-to-end, the licensing requirement)

• Backend: requireModule("semantic") on the suggest endpoint; the embed-on-upload hook + backfill
  sweep no-op when semantic is unlicensed; the license schema + registry + GA build-pin include it.
• Frontend: the suggested-folder chip + the admin description field render only when semantic ∈
  enabled_modules. The existing demo license is regenerated to include semantic.

Testing / verification

Per repo discipline: never go test (writes the live demo Postgres). Verify via
go build ./... && go vet ./...; cd web tsc + vite build; and a deployed e2e: bring up the
embed-sidecar, file a few documents into distinct folders (so they embed), then upload a new file
semantically similar to one folder's contents and assert POST /semantic/suggest-folder returns that
folder as the top suggestion with a sensible score; assert it returns nothing / is hidden when
semantic is unlicensed; assert the upload modal renders the chip. After every deploy assert
/me enabled_modules now includes semantic (demo) plus the original four; clean up test docs.