Semantic Search Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add a Keyword | Smart toggle to the existing /search page, where Smart ranks documents by meaning against the pgvector document_embeddings index — ACL-filtered, air-gapped, gated by the existing semantic module.

Architecture: The semantic context owns query embedding (EmbedQuery); the dms context owns the ACL-correct HNSW-ranked SQL over document_embeddings ⋈ documents (reusing aclPredicate + documentColumns); the httpapi handler orchestrates both and returns the existing documents DTO + a per-doc score map. The frontend adds a toggle (hidden when unlicensed) that swaps useSearch for a new useSemanticSearch, rendering the same table with a match-% badge.

Tech Stack: Go modular monolith (go/internal, chi, pgx, pgvector HNSW), React + Carbon (web/src), openapi-fetch, react-i18next.

Working discipline (applies to EVERY task)

NEVER run go test (test DSN = live demo Postgres). Verify Go via cd go && go build ./... && go vet ./....
Verify frontend via cd web && npx tsc --noEmit && npx vite build; run npm run gen:api after any api/openapi.yaml change; i18n en/id parity is tsc-enforced — add both.
Deploy only from repo root: docker compose -f deploy/docker-compose.yml --env-file deploy/mekari.env up -d --build. After every deploy assert /me enabled_modules == ['ai','correspondence','esign','semantic','watermarking'] and the demo is intact; clean up test docs.
Commit per task locally on main. Do NOT push unless asked.
Fail-closed: the endpoint is requireModule("semantic") (403 when unlicensed) and the Smart toggle is hidden when semantic ∉ enabled_modules. Keyword search is untouched.

Post-deploy assertion (reuse verbatim)

TOKEN=$(curl -s -XPOST localhost:38080/api/v1/auth/dev-login -H 'content-type: application/json' -d '{"email":"admin@obscura.local"}' | python3 -c 'import sys,json;print(json.load(sys.stdin)["token"])')
curl -s localhost:38080/api/v1/me -H "authorization: Bearer $TOKEN" | python3 -c 'import sys,json;print(sorted(json.load(sys.stdin)["enabled_modules"]))'

File Structure

Phase 1 (backend):
- Modify go/internal/dms/adapters/pg.go — formatVector helper + Store.SearchDocumentsByEmbedding (the ACL-ranked SQL + custom scan).
- Modify go/internal/dms/app/service.go — Repository port entry + Service.SearchDocumentsByEmbedding (over-fetch + trim).
- Modify go/internal/semantic/app/service.go — Service.EmbedQuery.
- Create go/internal/httpapi/handlers_semantic_search.go — the SemanticSearch handler; modify go/internal/httpapi/server.go (route) + api/openapi.yaml (+ npm run gen:api).

Phase 2 (frontend):
- Modify web/src/api/types.ts — optional score on DocumentRow.
- Modify web/src/api/search.ts — add useSemanticSearch (+ a shared row mapper).
- Modify web/src/features/search/SearchResultsPage.tsx — the toggle + badge + gating.
- Modify web/src/i18n/locales/en.ts + id.ts — toggle/badge strings.

PHASE 1 — Backend

Task 1.1: dms — the ACL-ranked embedding query

Files:
- Modify: go/internal/dms/adapters/pg.go (aclPredicate at :287, documentColumns at :232, scanDocument at :234 — reuse; add near SearchDocuments at :356)
- Modify: go/internal/dms/app/service.go (Repository interface near the search methods; a Service wrapper)

[ ] Step 1: Add a formatVector helper to the dms adapter (dms formats its own pgvector param; ~10 lines, self-contained). Add to pg.go (near the top of the file's helpers, after the imports):

// formatVector renders a []float32 as pgvector's text input form "[a,b,c]", for binding as a
// text param cast to ::vector in a query. (The semantic context has its own copy; dms keeps a
// local one so it stays self-contained.)
func formatVector(v []float32) string {
    var b strings.Builder
    b.Grow(len(v)*8 + 2)
    b.WriteByte('[')
    for i, x := range v {
        if i > 0 {
            b.WriteByte(',')
        }
        b.WriteString(strconv.FormatFloat(float64(x), 'f', -1, 32))
    }
    b.WriteByte(']')
    return b.String()
}

Confirm strings and strconv are imported in pg.go; add whichever is missing.

[ ] Step 2: Add SearchDocumentsByEmbedding to the Store in pg.go (reuse aclPredicate + documentColumns; custom scan = the 16 scanDocument fields plus score). Note the table stays unaliased documents because aclPredicate references documents.id:

// SearchDocumentsByEmbedding ranks live, readable documents by cosine similarity of their stored
// embedding to queryVec, applying the same classification/expired filters as advanced search,
// nearest-first, capped at limit. Returns the ranked documents + a parallel score slice
// (1 - cosine distance). limit is the caller's over-fetch (the ACL/filter WHERE post-prunes the
// ANN candidates, so the caller trims to the requested k). Empty queryVec → empty.
func (s *Store) SearchDocumentsByEmbedding(ctx context.Context, subjects []kernel.Subject, bypass bool, queryVec []float32, classification string, expiredOnly bool, limit int) ([]domain.Document, []float64, error) {
    if len(queryVec) == 0 || limit <= 0 {
        return nil, nil, nil
    }
    acl, aclArgs := aclPredicate(subjects, bypass, 5)
    args := append([]any{formatVector(queryVec), classification, expiredOnly, limit}, aclArgs...)
    rows, err := s.db.Exec(ctx).Query(ctx,
        `SELECT `+documentColumns+`, 1 - (e.embedding <=> $1::vector) AS score
           FROM documents
           JOIN document_embeddings e ON e.document_id = documents.id
           WHERE documents.deleted_at IS NULL
             AND `+acl+`
             AND ($2 = '' OR documents.classification = $2)
             AND ($3 = false OR (documents.expires_at IS NOT NULL AND documents.expires_at < now()))
           ORDER BY e.embedding <=> $1::vector
           LIMIT $4`, args...)
    if err != nil {
        return nil, nil, fmt.Errorf("dms search by embedding: %w", err)
    }
    defer rows.Close()
    var docs []domain.Document
    var scores []float64
    for rows.Next() {
        var d domain.Document
        var ref *string
        var score float64
        if err := rows.Scan(&d.ID, &d.FolderID, &d.Title, &d.CurrentVersion, &d.Classification, &d.CreatedAt, &d.DeletedAt, &d.LegalHold, &d.RetentionUntil, &d.DocType, &d.InheritAccess, &d.OwnerID, &d.Tags, &d.ExpiresAt, &d.Status, &ref, &score); err != nil {
            return nil, nil, err
        }
        if ref != nil {
            d.Reference = *ref
        }
        docs = append(docs, d)
        scores = append(scores, score)
    }
    return docs, scores, rows.Err()
}

ANN scale note (documented, not a v1 blocker): pgvector HNSW returns up to hnsw.ef_search (default 40) candidates before the WHERE/LIMIT apply, so on a large corpus a big LIMIT won't exceed ~40 post-filtered rows. The demo corpus is well within 40. Scale path: raise hnsw.ef_search (a session/tx GUC via SET LOCAL inside a uow.Do) or use pgvector iterative scan. Left out of v1 to keep the read path transaction-free.

[ ] Step 3: Add the method to the Repository interface in service.go (near SearchDocuments/SearchAdvanced):

    // SearchDocumentsByEmbedding ranks readable live documents by cosine similarity to queryVec
    // (with classification/expired filters), nearest-first, capped at limit; returns docs + scores.
    SearchDocumentsByEmbedding(ctx context.Context, subjects []kernel.Subject, bypass bool, queryVec []float32, classification string, expiredOnly bool, limit int) ([]domain.Document, []float64, error)

[ ] Step 4: Add the Service wrapper in service.go (over-fetch, then trim to k):

// SearchDocumentsByEmbedding ranks readable documents by cosine similarity to queryVec, applying
// the classification/expired filters, returning the top k (default 50, cap 100) documents + a
// parallel score slice. Over-fetches (k*5, capped) to absorb ACL/filter post-pruning, then trims.
func (s *Service) SearchDocumentsByEmbedding(ctx context.Context, subjects []kernel.Subject, bypass bool, queryVec []float32, classification string, expiredOnly bool, k int) ([]domain.Document, []float64, error) {
    if len(queryVec) == 0 {
        return nil, nil, nil
    }
    if k <= 0 {
        k = 50
    }
    if k > 100 {
        k = 100
    }
    overfetch := k * 5
    if overfetch > 200 {
        overfetch = 200
    }
    docs, scores, err := s.repo.SearchDocumentsByEmbedding(ctx, subjects, bypass, queryVec, classification, expiredOnly, overfetch)
    if err != nil {
        return nil, nil, err
    }
    if len(docs) > k {
        docs, scores = docs[:k], scores[:k]
    }
    return docs, scores, nil
}

[ ] Step 5: Verify + commit

cd /home/efran/remote-development/obscura/go && go build ./... && go vet ./...
cd /home/efran/remote-development/obscura && git add go/internal/dms
git commit -m "feat(dms): SearchDocumentsByEmbedding — ACL-ranked HNSW cosine query"

Expected: build/vet clean.

Task 1.2: semantic — embed the query

Files:
- Modify: go/internal/semantic/app/service.go (add near SuggestFolders; reuse clipEmbed, aiapp.EmbedQuery, s.dim)

[ ] Step 1: Add EmbedQuery

// EmbedQuery embeds a free-text search query and returns its vector (nil for empty text or an
// unexpected embedding shape). The semantic-gated search route uses it to rank documents; the
// route's requireModule("semantic") is the enforcement, so this does not re-check Enabled().
func (s *Service) EmbedQuery(ctx context.Context, text string) ([]float32, error) {
    text = strings.TrimSpace(text)
    if text == "" {
        return nil, nil
    }
    vecs, err := s.embedder.Embed(ctx, aiapp.EmbedQuery, []string{clipEmbed(text)})
    if err != nil {
        return nil, err
    }
    if len(vecs) != 1 || len(vecs[0]) != s.dim {
        return nil, nil
    }
    return vecs[0], nil
}

(strings + aiapp are already imported in this file.)

[ ] Step 2: Verify + commit

cd /home/efran/remote-development/obscura/go && go build ./... && go vet ./...
cd /home/efran/remote-development/obscura && git add go/internal/semantic
git commit -m "feat(semantic): EmbedQuery for semantic search"

Task 1.3: httpapi — the search endpoint

Files:
- Create: go/internal/httpapi/handlers_semantic_search.go
- Modify: go/internal/httpapi/server.go (route next to POST /semantic/suggest-folder, ~:283)
- Modify: api/openapi.yaml (+ npm run gen:api)

[ ] Step 1: Write the handler (mirror handlers_semantic.go SemanticSuggestFolder: PrincipalFrom, isContentAdmin, writeJSON, writeProblem, nonNilDocs; json decode)

go/internal/httpapi/handlers_semantic_search.go:

package httpapi

import (
    "encoding/json"
    "net/http"

    "github.com/Virtue-Digital-Indonesia/obscura/internal/kernel"
)

// SemanticSearch ranks the caller's readable documents by semantic similarity to a query, using
// the pgvector index. Gated by requireModule("semantic"). Body: {query, classification?,
// expired_only?, k?}. Response: {documents: [...ranked...], scores: {doc_id: 0..1}} — reuses the
// same document serialization keyword search returns so the UI reuses its row mapper.
func (s *Server) SemanticSearch(w http.ResponseWriter, r *http.Request) {
    p, _ := PrincipalFrom(r.Context())
    var body struct {
        Query          string `json:"query"`
        Classification string `json:"classification"`
        ExpiredOnly    bool   `json:"expired_only"`
        K              int    `json:"k"`
    }
    if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
        writeProblem(w, &kernel.Error{Kind: kernel.ErrValidation, Code: "request.invalid_json", Message: "invalid request body"})
        return
    }
    vec, err := s.semantic.EmbedQuery(r.Context(), body.Query)
    if err != nil {
        writeProblem(w, err)
        return
    }
    if len(vec) == 0 {
        writeJSON(w, http.StatusOK, map[string]any{"documents": []any{}, "scores": map[string]float64{}})
        return
    }
    docs, scores, err := s.dms.SearchDocumentsByEmbedding(r.Context(), p.Subjects(), s.isContentAdmin(r.Context(), p), vec, body.Classification, body.ExpiredOnly, body.K)
    if err != nil {
        writeProblem(w, err)
        return
    }
    scoreByID := make(map[string]float64, len(docs))
    for i, d := range docs {
        sc := scores[i]
        if sc < 0 {
            sc = 0
        }
        scoreByID[d.ID] = sc
    }
    writeJSON(w, http.StatusOK, map[string]any{"documents": nonNilDocs(docs), "scores": scoreByID})
}

Confirm the helper names against handlers_semantic.go (PrincipalFrom, writeJSON, writeProblem, s.isContentAdmin, nonNilDocs, s.semantic, s.dms) — they're all used by the suggest handler + the list handlers. Adapt if any differs.

[ ] Step 2: Register the route in server.go, immediately after the suggest-folder route (~:283):

                r.With(s.requireModule("semantic")).Post("/semantic/search", s.SemanticSearch)

[ ] Step 3: OpenAPI + regen. Add POST /api/v1/semantic/search to api/openapi.yaml (match the file's style): requestBody json {query: string, classification?: string, expired_only?: boolean, k?: integer}; response 200 json {documents: {type: array, items: {type: object}}, scores: {type: object, additionalProperties: {type: number}}}; response 403. Then:

cd /home/efran/remote-development/obscura/web && npm run gen:api

Confirm web/src/api/schema.ts contains semantic/search.

[ ] Step 4: Verify + commit

cd /home/efran/remote-development/obscura/go && go build ./... && go vet ./...
cd /home/efran/remote-development/obscura && git add go/internal/httpapi api/openapi.yaml web/src/api/schema.ts
git commit -m "feat(httpapi): POST /semantic/search gated by requireModule(semantic)"

Task 1.4: Deploy backend + e2e

Files: none (verification only)

[ ] Step 1: Deploy + module assertion

cd /home/efran/remote-development/obscura
docker compose -f deploy/docker-compose.yml --env-file deploy/mekari.env up -d --build obscura

Run the post-deploy assertion (expect 5 modules).

[ ] Step 2: e2e — ranking + ACL + filters + gating. Ensure a few demo docs are embedded (they are, from the semantic backfill), then:

TOKEN=$(curl -s -XPOST localhost:38080/api/v1/auth/dev-login -H 'content-type: application/json' -d '{"email":"admin@obscura.local"}' | python3 -c 'import sys,json;print(json.load(sys.stdin)["token"])')
# Rank by meaning — a conceptual query returns the semantically-closest docs, scored:
curl -s -XPOST localhost:38080/api/v1/semantic/search -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' -d '{"query":"kontrak perjanjian vendor","k":5}' | python3 -c 'import sys,json;d=json.load(sys.stdin);[print(round(d["scores"][x["ID"]],3), x["Title"]) for x in d["documents"]]'
# Classification filter narrows results:
curl -s -XPOST localhost:38080/api/v1/semantic/search -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' -d '{"query":"laporan","classification":"none","k":5}' -o /dev/null -w "filtered: %{http_code}\n"
# Empty query → empty (no 500):
curl -s -XPOST localhost:38080/api/v1/semantic/search -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' -d '{"query":"  "}' | python3 -c 'import sys,json;d=json.load(sys.stdin);print("docs",len(d["documents"]))'

Expected: the conceptual query returns ranked docs with sensible scores; the empty query returns 0 docs (HTTP 200).
- ACL check: create a doc readable only by a non-admin position (or reuse an existing restricted doc), then query as a user without read access and assert it does NOT appear.
- Fail-closed: hot-swap to a core-only license (as in prior work) and assert POST /semantic/search → 403, then restore the 5-module license + redeploy + re-assert. (Or reason from code: the route is requireModule("semantic").)

[ ] Step 3: Clean up any test docs created; confirm the demo is intact.

Phase 1 shippable here (backend-only; no UI yet).

PHASE 2 — Frontend

Task 2.1: `score` on the row + `useSemanticSearch`

Files:
- Modify: web/src/api/types.ts (add score? to DocumentRow)
- Modify: web/src/api/search.ts (add useSemanticSearch)

[ ] Step 1: Add an optional score to DocumentRow. In types.ts, find the DocumentRow interface and add:

  // Relevance score (0..1) — only set by semantic search results.
  score?: number

[ ] Step 2: Add useSemanticSearch to search.ts (mirrors runSearch/useSearch but POSTs /semantic/search and attaches the score). Append:

// runSemanticSearch ranks documents by meaning via POST /api/v1/semantic/search (premium
// `semantic` module). Same result shape as keyword search, plus a relevance score per row.
async function runSemanticSearch(c: SearchCriteria): Promise<DocumentRow[]> {
  const body: { query: string; classification?: string; expired_only?: boolean; k?: number } = { query: c.text.trim(), k: 50 }
  if (c.classification) body.classification = c.classification
  if (c.expiredOnly) body.expired_only = true
  const [res, users] = await Promise.all([api.POST('/api/v1/semantic/search', { body }), userMap()])
  const data = ok(res) as unknown as { documents?: ApiDocument[]; scores?: Record<string, number> }
  const scores = data.scores ?? {}
  return (data.documents ?? [])
    .filter((d) => !isDemoGarbage(d.Title))
    .map((d) => ({ ...toRow(d, users), score: scores[d.ID] }))
}

// useSemanticSearch runs meaning-based search for the given criteria. Disabled until a text
// query is set (semantic search needs a query; filters alone don't rank).
export function useSemanticSearch(criteria: SearchCriteria) {
  return useQuery<DocumentRow[]>({
    queryKey: ['semantic-search', criteria],
    queryFn: () => runSemanticSearch(criteria),
    enabled: !!criteria.text.trim(),
  })
}

(ApiDocument, toRow, isDemoGarbage, userMap, api, ok, useQuery, SearchCriteria are all already in this file.)

[ ] Step 3: Verify + commit

cd /home/efran/remote-development/obscura/web && npx tsc --noEmit && npx vite build
cd /home/efran/remote-development/obscura && git add web/src/api
git commit -m "feat(web): useSemanticSearch hook + row score"

Task 2.2: The Keyword | Smart toggle

Files:
- Modify: web/src/features/search/SearchResultsPage.tsx
- Modify: web/src/i18n/locales/en.ts + id.ts

[ ] Step 1: Imports + gating + mode state. In SearchResultsPage.tsx:
Add imports: ContentSwitcher, Switch from @carbon/react; useSearch, useSemanticSearch, type SearchCriteria from @/api/search (extend the existing import); useMe from @/api/me; moduleEnabled from @/lib/nav.
In the component, after the existing state, add:

  const me = useMe()
  const semanticEnabled = moduleEnabled(me.data?.enabledModules, 'semantic')
  const [mode, setMode] = useState<'keyword' | 'smart'>('keyword')

[ ] Step 2: Run the active hook only. Replace the single useSearch call:

  const { data: results = [], isFetching, isError } = useSearch(criteria)

with both hooks (the inactive one gets empty criteria so it never fetches):

  const EMPTY: SearchCriteria = { text: '' }
  const keyword = useSearch(mode === 'keyword' ? criteria : EMPTY)
  const smart = useSemanticSearch(mode === 'smart' ? criteria : EMPTY)
  const { data: results = [], isFetching, isError } = mode === 'smart' ? smart : keyword

[ ] Step 3: Render the toggle — only when licensed. Add just above the <form className="search__bar">:

      {semanticEnabled && (
        <ContentSwitcher
          className="search__mode"
          selectedIndex={mode === 'smart' ? 1 : 0}
          onChange={({ index }) => setMode(index === 1 ? 'smart' : 'keyword')}
        >
          <Switch name="keyword" text={t('search.modeKeyword')} />
          <Switch name="smart" text={t('search.modeSmart')} />
        </ContentSwitcher>
      )}

[ ] Step 4: Match-% badge in Smart mode. In renderCell, the 'name' case, append the badge when the row has a score:

    case 'name':
      return (
        <div className="doc-name">
          <Document size={16} className="doc-name__icon" />
          <span className="doc-name__title">{doc.title}</span>
          <DocFlags flags={doc.flags} />
          {doc.score !== undefined && <span className="foldersuggest__badge">{Math.round(doc.score * 100)}%</span>}
        </div>
      )

(.foldersuggest__badge already exists in app.css from the folder box.)

[ ] Step 5: i18n. Add to en.ts search block: modeKeyword: 'Keyword', modeSmart: 'Smart'; to id.ts search block: modeKeyword: 'Kata kunci', modeSmart: 'Cerdas'.
[ ] Step 6: Verify + commit

cd /home/efran/remote-development/obscura/web && npx tsc --noEmit && npx vite build
cd /home/efran/remote-development/obscura && git add web/src
git commit -m "feat(web): Keyword|Smart search toggle + match-% badge (semantic-gated)"

Task 2.3: Deploy + UI e2e + final review