Migrating a customer into Obscura — Google Drive and SharePoint/OneDrive

obscura-migrate imports a new customer's existing corpus at onboarding. It is a one-way
onboarding loader, not a sync engine: it runs while a tenant is being stood up, lands their
content with its history intact, and is then done.

Build it with the rest of the repo:

cd go && go build -o obscura-migrate ./cmd/obscura-migrate

1. What it preserves, and why that is the whole point

Loading a corpus through the ordinary POST /documents would work, and would be wrong. Every
document would arrive owned by the migration service account and dated import-day. In a
records system that is not cosmetic:

Retention floors run from issuance. A corpus dated import-day carries a corpus of wrong
disposal dates, and the error is invisible until a disposal review years later.
Every version's audit trail would name the service account instead of whoever wrote it.
Ownership confers Manage, so whoever ran the migration would hold control of everything.

So the tool loads through a separate surface, /api/v1/migrate/*, which accepts what the
normal write path must never accept: a caller-chosen owner, author and timestamp.

That capability is privileged. It sits behind its own permission, documents.migrate
(migration 00186), granted to nobody by default. The intended lifecycle is: grant it to the
migration service account, run the migration, revoke it.

2. Before you start — what to collect from the customer

Google Workspace

A service account with domain-wide delegation, and the Workspace super-admin who can
authorise its client id in the admin console.
Scope: https://www.googleapis.com/auth/drive.readonly
The email of a user to impersonate (--gdrive-subject) — an admin who can see the
content in scope.
Which Shared Drives are in scope (--gdrive-drive).

Per-user OAuth is deliberately not supported: it means chasing every member of the
organisation for a consent screen before their files can be read.

SharePoint / OneDrive

An Entra app registration with admin-consented application permissions:
Files.Read.All, Sites.Read.All, User.Read.All.
Tenant id, client id, client secret.
The drive id of each document library in scope (--graph-drive).

Obscura side

An obsk_ API key for a service account holding documents.migrate,
folder.create/folder.create_root and document.create.

Use an API key, not a session token. API-key principals are exempt from the per-user
rate limit (600 req/min); a session-authenticated run crawls at roughly 200–300 documents
a minute for no reason.
The folder id to import under (--root-folder). A migration lands inside a named
folder rather than at the repository root, so a foreign tree is never mixed irreversibly
into the deployment's own structure.

3. The four phases

Phase	Writes?	What it does
`inventory`	no	Walks the source and counts everything. Produces the report the customer signs off.
`folders`	yes	Recreates the tree under `--root-folder`, parents first.
`grants`	yes	Maps source sharing onto Obscura folder ACLs.
`docs`	yes	Creates each document and replays its versions oldest-first.
`all`	yes	All four in order.

Step 1 — Inventory. Needs no Obscura at all.

obscura-migrate --source gdrive \
  --gdrive-key sa.json --gdrive-subject admin@customer.com --gdrive-drive <driveId> \
  --phase inventory \
  --report inventory.json \
  --write-identity-skeleton identities.csv

This writes nothing anywhere and can be run against a prospect before an Obscura instance
exists. It gives you file counts and sizes, an extension histogram, everything that will be
skipped and why, every link share, and every identity that needs mapping.

Step 2 — Fill in the identity map

--write-identity-skeleton emits a pre-filled CSV of every identity the inventory saw,
ordered by how many documents depend on it, with the kind already guessed:

source,kind,target
dewi@oldcorp.com,user,dewi@oldcorp.com
budi@oldcorp.com,user,budi@oldcorp.com
legal team,group,legal team

kind is user, group, position or drop (drop = import the content, ignore this
identity's grants — an explicit decision, recorded so the report can tell it apart from a gap).

An identity with no row passes through unchanged. When the customer keeps the same email
addresses in Obscura — much the commonest case — an empty mapping file is the correct
configuration. The server still resolve-or-refuses, so a passthrough naming somebody who does
not exist fails loudly rather than becoming a ghost owner.

Step 3 — Rehearse

obscura-migrate --source gdrive ... --identity identities.csv \
  --phase all --dry-run --api https://customer.obscura.id/api/v1

Step 4 — Load

obscura-migrate --source gdrive ... --identity identities.csv \
  --phase all \
  --api https://customer.obscura.id/api/v1 \
  --token-file .apikey \
  --root-folder <folderId> \
  --state customer-migration.json \
  --report customer-migration-report.json

Interrupt it whenever you like (Ctrl-C is clean). Re-run the same command to resume — the
state file makes it a no-op over everything that already landed.

4. Decisions the tool makes, deliberately

Situation	What happens
Google Docs/Sheets/Slides	Exported to DOCX/XLSX/PPTX only. No PDF is generated — users make one in Obscura if they want it.
Google Forms, Sites, Maps, Apps Script	Skipped and reported: they have no file representation.
File types outside Obscura's allowlist (`.zip`, `.psd`, `.mp4`, …)	Skipped and reported, checked locally before download.
"Anyone with the link" / domain shares	Never recreated. Reported for a human decision — recreating them would republish content the customer has not re-consented to.
A source role we do not recognise	Not applied; reported.
`commenter` / `review`	Mapped to read, not read-write. Over-granting is invisible; under-granting self-corrects.
Per-file shares that differ from their folder	Reported, not applied. Obscura's model is folder-inherited, and thousands of per-file exceptions produce a permission structure nobody can reason about.
A file with several Drive parents	Filed under the first; the rest are reported.
An owner that maps to a group	Refused — ownership is held by a person. Falls back to `--default-owner`, or the document is skipped and reported.
Trashed items	Skipped unless `--gdrive-include-trashed`.
OneNote sections	Skipped and reported.
Classification	Left to the target folder's `default_classification` unless `--classification` is given. Setting the default on folders once and letting inheritance work is the recommended arrangement.
Numbering	Off. Imported documents keep whatever identifier the source gave them; routing them through the auto-number counter would invent references and shift later documents.

5. SharePoint cutover with the delta token

Graph's /delta returns the whole drive and a token naming that point in time. After a
bulk pass the tool prints it:

Graph delta token captured. For the cutover catch-up run, replay only what
changed since this pass with --graph-delta:
  https://graph.microsoft.com/v1.0/drives/…/root/delta?token=…

So an onboarding does not need the customer's SharePoint frozen for the duration:

Run the bulk load over however many days it takes.
On go-live day, re-run with --graph-delta <token> and the same --state file. Only
what changed since the bulk pass is walked.

Google Drive has no equivalent; a Drive cutover is a second full pass, which the state file
makes cheap (already-loaded items are skipped).

6. Reading the report

The JSON report is the reconciliation artifact the customer signs off. Its job is to make
gaps visible — a migration that silently drops four thousand files reads exactly like one
that succeeded.

  folders seen      312
  files seen        48,201  (91.4 GiB)
  files eligible    43,889  (74.2 GiB)
  documents loaded  43,889
  versions loaded   61,204  (74.2 GiB)

  SKIPPED 4,312 files (17.2 GiB):
    file type not accepted by Obscura (.zip)      2,904
    file type not accepted by Obscura (.mp4)      1,203
    Google Form has no file representation          205

Skipped items are grouped by reason, not listed one per line — a number the customer can act
on rather than something they scroll past. The full per-file list is in the JSON.

Check three things every time:

files seen − files eligible = the skipped count. Nothing should vanish between them.
documents loaded equals files eligible, or the difference is in failures.
unresolved_identities is empty, or every entry is one you deliberately chose to leave.

7. Testing without credentials

The fixture source is the same interface over a JSON file, so the whole pipeline can be
exercised before a customer has issued anything:

obscura-migrate --source fixture \
  --fixture go/cmd/obscura-migrate/testdata/sample-drive.json \
  --phase all --dry-run --api http://localhost:8080/api/v1

The sample fixture deliberately contains a nested tree, a three-revision document with two
different authors, an unmappable owner, a link share, and three files the allowlist refuses.

go test ./cmd/obscura-migrate/ covers the wire contract (that owner, created_at and
per-version authorship are actually sent), resume behaviour, report deduplication, and a
guard that the tool's copy of the upload allowlist has not drifted from the server's.