think
16px
820px
← all files

Semantic search — Design

Date: 2026-07-01
Status: Approved (design); pending implementation plan.

Goal

Let users search documents by meaning, not just keywords, by ranking the existing pgvector
document_embeddings index against an embedded query. Delivered as a Keyword | Smart toggle
on the existing /search page, gated by the existing semantic module (so that module now
covers both smart filing and semantic search — one "Semantic Intelligence" value prop over one
embeddings engine). Fully air-gapped: embeddings only, no LLM.

Decisions (locked during brainstorming)

  1. Toggle, not replace or blend. A Keyword | Smart mode switch on /search. Keyword =
    today's tsvector search (core, unchanged). Smart = meaning-based (premium). Hybrid
    keyword+semantic blending is a documented future, not v1.
  2. Gated by the existing semantic module — reuses the same document_embeddings index,
    fastembed sidecar, and backfill pipeline that module already provisions. When semantic is
    unlicensed the Smart toggle is hidden and search is keyword-only (no change, no new plumbing).
  3. Reuse the suggest-folder pattern (query → Embedder.Embed(kind=query) → cosine rank) — but
    this time the HNSW index is usable because we rank stored per-document embeddings, not
    on-the-fly centroids.
  4. ACL-filtered to readable docs, mirroring keyword search (aclPredicate, access_mode ≥ 1).
  5. Filters (Classification, ExpiredOnly) carry into Smart mode; results show a match-%
    badge per row.

Reused (existing) systems — do NOT rebuild

  • Keyword search: GET/POST /api/v1/documents/search (Search title-only + SearchAdvanced
    structured tsvector over title/content_text/tags/doc_type/classification, ACL-filtered) — stays
    as the Keyword mode, untouched.
  • Search UI: the global CommandBar (header → /search?q=) and SearchResultsPage.tsx
    (useSearch → POST /documents/search, Classification + ExpiredOnly filters, DocumentRow
    results table). The toggle is added here.
  • Embeddings: document_embeddings(document_id pk, version, model, embedding vector(384), updated_at) with the HNSW cosine index document_embeddings_ann USING hnsw (embedding vector_cosine_ops) — ORDER BY embedding <=> $query::vector uses it directly.
  • Semantic context: internal/semantic/app.Service (Embedder, Store, SuggestFolders,
    cosine, the live Enabled() license predicate) + internal/semantic/adapters Store + the
    formatVector/parseVector codec — extended, not duplicated.
  • ACL: aclPredicate(subjects, bypass, n) (dms adapters) = the EXISTS over document_acl_read
    at access_mode >= 1 that keyword search already uses; the semantic query reuses the same shape.

Component 1 — Backend: rank documents by embedding

Service method on semantic.Service (mirrors SuggestFolders):
SearchDocuments(ctx, subjects []kernel.Subject, bypass bool, q SearchQuery, k int) ([]domain.Document, map[string]float64, error)
where SearchQuery{Text string; Classification string; ExpiredOnly bool} — it returns the ranked
documents (top-k) plus a document_id → score map. Empty query or empty index → empty.

Flow: embed q.Text (kind=query) → hand the query vector (as pgvector text) to the dms ranking
method below → the caller's readable docs come back ranked by cosine, over-fetched, trimmed to k
in the Service. (The Service owns the embedding step + trim + score shaping; dms owns the
ACL-correct ranked lookup.)

dms Store/Service method (new — dms owns documents + document_acl_read + aclPredicate):
SearchDocumentsByEmbedding(ctx, subjects, bypass, queryVectorText string, q SearchQuery, limit int) ([]domain.Document, []float64, error) — returns the ranked full documents (reusing the existing
documentColumns select list, so there is no N+1 hydration) and a parallel score slice. The SQL
joins document_embeddings → documents, applies the ACL predicate + filters + deleted_at IS NULL,
and orders by cosine distance:

SELECT <documentColumns>, 1 - (e.embedding <=> $query::vector) AS score
FROM documents d
JOIN document_embeddings e ON e.document_id = d.id
WHERE d.deleted_at IS NULL
  AND <acl predicate on d>            -- access_mode >= 1 for the caller's subjects (or bypass)
  AND ($classification = '' OR d.classification = $classification)
  AND ($expiredOnly = false OR (d.expires_at IS NOT NULL AND d.expires_at < now()))
ORDER BY e.embedding <=> $query::vector
LIMIT $overfetch
  • ANN + restrictive-filter under-fetch (the one real gotcha): HNSW returns the nearest rows by
    vector, then the ACL/filters prune some, so a plain LIMIT k can return fewer than k readable
    results. Mitigation for v1: over-fetch (overfetch = min(k * 5, cap)), raise
    hnsw.ef_search for the query, then trim to k in Go. pgvector iterative-scan is the documented
    scale path.
  • Score = 1 - cosine_distance (cosine similarity), clamped ≥ 0, for a match-% badge.

ACL placement: the ranking query needs both the pgvector table and the dms aclPredicate /
documents table. Cleanest boundary: the query lives where the ACL predicate lives — the
SearchDocumentsByEmbedding method above is a dms method (dms owns documents +
document_acl_read + aclPredicate + documentColumns). The semantic Service reaches it through a
new method on its existing DocSource port (the narrow interface it already uses to call into dms),
so the semantic package never imports dms internals. The semantic Service owns the embedding step +
trim + score map; dms owns the ACL-correct ranked lookup. (Alternative: the semantic adapter imports
the ACL predicate — rejected; keeps ACL logic in one place.)

Endpoint: POST /api/v1/semantic/search (requireModule("semantic")), body
{query: string, classification?: string, expired_only?: bool, k?: int}. Response reuses the
documents DTO plus scores: {documents: [<raw domain.Document>...], scores: {<doc_id>: <float>}}
(documents in ranked order; scores keyed by id for the badge). Reuses nonNilDocs + the same
document serialization keyword search returns, so the frontend reuses ApiDocument → DocumentRow.

Component 2 — Frontend: the toggle + Smart results

  • SearchResultsPage.tsx gains a Carbon ContentSwitcher Keyword | Smart, rendered only when
    moduleEnabled(me.data?.enabledModules, 'semantic') (else keyword-only, unchanged). Default =
    Keyword. Mode is local state (optionally reflected in the URL as &mode=smart).
  • New hook useSemanticSearch(criteria) (mirrors useSearch + useSuggestFolder): POSTs
    /api/v1/semantic/search with {query, classification, expired_only, k: 50}, maps the response
    to DocumentRow[] (same mapper as keyword) and attaches score per row from the scores map.
  • Smart results render in the same table (same columns), ranked by relevance, with a subtle
    match-% badge (reuse the folder-suggestion badge style). The Classification + ExpiredOnly
    filters apply in both modes.
  • Empty/short query and empty results show the existing empty state.

Gating & degradation

  • Backend: requireModule("semantic") on POST /semantic/search; the ranking is a no-op path when
    the module is unlicensed (the route 403s). Keyword search is untouched and always available.
  • Frontend: the Smart toggle is hidden unless semantic ∈ enabled_modules; keyword remains the
    default. This matches the fail-closed posture of the semantic module.
  • Only documents that have an embedding appear in Smart results; the existing backfill sweep keeps
    the index current, and anything not-yet-embedded is still findable via Keyword.

Air-gapped

Uses the embeddings sidecar only (kind=query) — no LLM, no cloud. Fully on-prem, consistent with
the semantic module.

Out of scope (documented futures)

  • Hybrid keyword+semantic blend (reciprocal-rank fusion).
  • Semantic search over folders / attachments / letter bodies.
  • LLM re-ranking or query expansion (would pull in the ai module).
  • Per-version embeddings (embeddings are per-document today).

Testing / verification

Per repo discipline: never go test (writes the live demo Postgres). Verify via
go build ./... && go vet ./...; cd web tsc + vite; and a deployed e2e:
- File a few documents with distinct content into the demo, ensure they embed (backfill), then
POST /semantic/search {query: "<concept matching one doc>"} and assert that doc ranks top with a
sensible score; assert an ACL-restricted doc does NOT appear for a user who can't read it; assert
Classification/ExpiredOnly filters narrow the results; assert the endpoint 403s when semantic is
unlicensed (license hot-swap) and the Smart toggle is hidden.
- After every deploy assert /me enabled_modules unchanged (the 5 modules) and the demo intact;
clean up test docs. Regenerate web/src/api/schema.ts (npm run gen:api) after the OpenAPI edit.