Graph database for Obscura — Neo4j vs. what we already run

Date: 2026-08-15 · Status: research, no decision taken
TL;DR: Don't add Neo4j as infrastructure. The graph-shaped queries Obscura runs today are
shallow tree walks that Postgres already handles; the one genuinely graph-flavoured
opportunity (a knowledge graph feeding Ask-the-Archive) is better served by Apache AGE —
openCypher inside the Postgres we already ship — which inherits our backups, tenancy and
air-gap story instead of breaking all three.

1. What is actually graph-shaped in Obscura today

Measured against the codebase, not imagined:

The honest observation: every one of these is ≤2 hops or a bounded tree climb. Postgres
recursive CTEs and the existing Go walks handle them. Nothing here waits on a graph engine —
there is no "find shortest path across 6 hops" query anywhere in the product, and none of the
features on the roadmap need one.

2. Where a graph would genuinely add something

Two candidates, one strong:

A. Knowledge graph under Ask-the-Archive (GraphRAG). Today retrieval is pgvector
similarity (mig 00071) — it finds passages that sound like the question. It cannot answer
multi-hop questions: "which agreements with vendor X carry termination clauses and were
signed by someone in Direktorat Y?" That needs entities (people, orgs, dates, clause types)
extracted at enrichment time into a graph, walked at ask time, with the vector index picking
entry points. This is the strongest real use — and it pairs directly with the
contract-intelligence idea (§5).

B. A "related documents" exploration UI — letter chains, disposisi trails, who-signed-what.
Nice, but the data is 1–2 hops and buildable on the existing tables; a graph DB is not the
blocker, a UI is.

3. Why Neo4j specifically is a poor fit for this product

• 🔴 Community Edition is one database, no RBAC, no clustering, no hot backup — those are
  Enterprise (commercial). Cloud/VM2 is schema-per-tenant in one Postgres; on Neo4j CE,
  tenant isolation would be entirely app-side discipline in every Cypher query, which is
  exactly the class of bug the tenant-boundary audit exists to prevent.
• 🔴 It breaks the backup doctrine. Backups are pg_dump + mc mirror, age-encrypted,
  pull-based, with a rehearsed restore. A Neo4j store sits outside all of it; CE has no
  online backup, so it's either downtime dumps or an unprotected data set holding a copy of
  the archive's most sensitive metadata.
• A new always-on JVM service in a compose that already runs 15+ services with 3–4 GiB
  sidecars, on hosts where we've had a dockerd crash under load (VM2) and 87% disk (demo).
  Air-gapped installs gain another image, another licence conversation, another thing doctor
  must probe.
• Dual-write consistency. Postgres stays the source of truth (ACLs, retention, legal
  hold are non-negotiably relational), so every document/link/position change must be
  mirrored into the graph. That sync layer is where these projects actually die.

4. The recommended shape — if we want graph features

Phase 1 — prove the value with zero new infrastructure. Build the entity extraction
(the AI enrichment pipeline already exists — provider config mig 00146, embed sidecar) into
plain tables: entities, entity_mentions, entity_edges. Feed Ask-the-Archive retrieval
from vector hits expanded one hop through those tables. If answer quality doesn't move,
stop — we learned it cheaply.

Phase 2 — Apache AGE if the walks get deep. openCypher as a
Postgres extension: PG 11–18 supported, ASF-licensed, active (Jan 2026 release added
row-level security), shipped as a managed extension on Azure. Same database ⇒ same
pg_dump backup, same schema-per-tenant isolation, same transaction as the source-of-truth
write (no dual-write problem at all), nothing new in compose. The swap-in cost is a
custom Postgres image (pgvector + age), which we already do for other services.

Phase 3 — Neo4j only if a dedicated graph-analytics workload emerges that AGE
measurably can't serve (community-detection at millions of edges, Bloom-style visual
exploration as a sold feature). That's a licence + ops decision to make then, with
numbers.

4b. When to actually revisit AGE — written down so it stops being a judgement call

Reviewed 2026-08-18. Phase 1 shipped in part: entities + entity_aliases (mig 00198)
and a one-hop expansion feeding Ask-the-Archive. entity_edges never shipped, which is why
the product still has effectively one edge type — an entity is party to a contract:

SELECT DISTINCT cp.document_id FROM contract_parties cp
  JOIN documents d ON d.id = cp.document_id AND d.deleted_at IS NULL
 WHERE cp.entity_id = ANY($1)          -- internal/contracts/adapters/entity_pg.go

Adopting AGE on top of that would be a graph engine querying a graph with one kind of edge in
it. The successor design is document_edges, a derived table
(docs/superpowers/specs/2026-08-18-investigation-module-design.md) — and once its rows are
uniform (src_kind, src_id, rel, dst_kind, dst_id), an arbitrary-depth walk over
heterogeneous edges is a single self-join, which is most of what AGE was wanted for.

🔴 The trigger is a query shape, not a row count

AGE buys expressiveness, not speed. It stores its graph in Postgres tables under
ag_catalog — it is not index-free adjacency the way Neo4j is — so for a simple traversal it
is doing broadly what a well-indexed recursive CTE does. Benchmark before believing any
performance claim; do not adopt it expecting one.

Adopt when someone asks for a query in this list, because these are the ones a recursive CTE
answers badly:

1. Shortest path / all paths between two nodes — needs manual cycle detection and path
   accumulation in SQL, and degrades fast.
2. Cycle detection — "find a vendor → signer → vendor loop", the procurement-fraud query.
3. Per-hop predicates on a variable-length pattern — "any-length paths whose intermediate
   edges are all signatures but whose endpoints are entities".
4. Maintenance mass — four or more distinct traversal queries each written as a bespoke
   40-line CTE. This one stands alone even when every individual query performs fine.

The scale axis, with real numbers

Measured on the demo (2026-08-18): 178 documents → ~259 edges from the projection sources,
≈1.5 edges per document. A mature archive runs 3–8, so a 1M-document B2G deployment
projects to 3–8M edges.

Obscura will essentially never reach a scale where AGE is needed for performance. If it is
ever adopted, it will be for query shape.

🔴 The trap: the real enemy is branching factor, and AGE does not fix it

What kills a traversal is a hub, not table size — a ministry that is party to 5,000
contracts, a Direktur Utama who signed 20,000 documents. A depth-3 walk from a hub touches
millions of rows in any engine. The fix is product design: cap hops, cap fan-out per hop,
and say in the UI when the result was truncated. Swapping the storage engine changes nothing
here.

The test to run when a trigger fires

1. Write the query as a recursive CTE anyway.
2. EXPLAIN ANALYZE against the largest real corpus available.
3. Slow → it is hubs; AGE will not save you, fix fan-out instead.
4. Fast but unmaintainable → this is the real AGE case, and it is an honest one.
5. Only then: custom pgvector + age image, and think hard about ag_catalog under
   schema-per-tenant on VM2 before it touches a live multi-tenant database.

5. The "Icertis" pairing (to confirm)

If the second half of the ask is Icertis-style contract intelligence, the pairing makes
sense as one feature, not two technologies: extract parties / dates / values / obligations /
clause types at enrichment → structured fields + graph edges → obligation deadlines feed the
existing scheduler + notification pipeline (renewal reminders), clause library builds on the
templates module, risk flags on the classifications the DLP registry already carries. That
is an Obscura module (licensable, like ai/office), not an integration with Icertis —
which is a competing CLM platform, not a component: it brings its own repository, workflows
and e-sign, so adopting it means paying for what we already ship. It does offer on-premise
and hybrid deployment alongside its Azure SaaS, so the air gap is a cost, not the reason.

Sources: Neo4j open-core licensing FAQ ·
Neo4j CE limitations (community) ·
Neo4j — Wikipedia ·
Apache AGE release notes ·
AGE PG17/18 roadmap discussion ·
AGE on Azure PostgreSQL ·
What is Contract Intelligence — Icertis ·
Icertis platform