Benchmark Harness Design — Three Backends

Status (2026-08-07): three-way, topology-led (peer-to-peer: LuckPerms + LCore; master-slave: NeoCore). One rig, three real systems behind a common interface. Security is a separate analytical metric (see §Attack surface), not measured here. Artifact is NeoCore (12,495 LOC, 13 TODO markers, pure gRPC, no Redis) — replaces Atreus.

Goal: one measurement rig that drives the real synchronization code of all three systems under identical conditions, so the comparison is fair. Only the propagation mechanism differs.

Architecture

The three backends (each wraps the REAL code)

• LuckPermsBackend — real LuckPerms configured with the Redis messenger + shared MySQL storage, running inside small Bukkit test servers. Origin = trigger a permission change via LuckPerms API/command; receiver = observe when LuckPerms on another server reflects it (via its event/API). Heaviest to stand up (LuckPerms is built to live inside a server).
• LCoreBackend — real RedisHandler + Packets publish/subscribe (push-data), extracted headless.
• NeoCoreBackend — real rpc-master + dataloader client stream path (PubSubRPC bidi + PlayerList RPC), extracted headless. NeoCore has only 13 TODO markers and ran in staging, so it should start with far less resurrection work than Atreus would have.

The pluggable interface (Java-ish sketch)

interface PropagationBackend {
  void     start(Config cfg);
  Origin   connectOrigin(String serverId);
  Receiver connectReceiver(String serverId, ApplyListener onApply);
  void     injectFailure(FailureType t);   // KILL_DB / KILL_BROKER / KILL_MASTER
  void     close();
}
interface Origin   { void commit(Mutation m); }   // m has a unique correlationId
interface Receiver { }                             // on apply, invokes onApply(correlationId, System.nanoTime())

Instrumentation

• Origin stamps correlationId + commitNanos per mutation; receiver stamps applyNanos when the change is reflected; latency = applyNanos - commitNanos.
• Single host + System.nanoTime() → no clock skew.
• HdrHistogram for p50/p95/p99; rolling ops counter for throughput.
• Loss = correlationIds committed but never applied within a timeout.

Metrics computed

Attack surface (metric #7 — ANALYTICAL, not run here)

Determined by reading each architecture, not measured:
| | LuckPerms | LCore | NeoCore |
|---|---|---|---|
| Nodes holding store credentials | N | N | 1 (master) |
| Nodes able to write/propagate directly | N | N | 1 |
Honest caveat (one sentence in the paper): NeoCore's transport is currently plaintext/unauthenticated → the advantage is credential distribution; channel hardening (TLS/auth) is future work.

Config knobs (identical across backends)

N receivers · target rate (ops/s) · op-mix · duration · warmup · repeats (k).

Output schema (CSV, one row per op)

run_id, backend, N, target_rate, op_type, correlation_id, commit_ns, apply_ns, latency_ns, applied, db_reads, wire_bytes

Fairness controls

• Same host, JVM/GC flags, warmup, monotonic clock; same MySQL for all three.
• Each backend uses its real serialization/mechanism; measure wire bytes AND DB load, don't assume.
• Compare only the propagation path — acknowledge LuckPerms does far more (contexts, inheritance) so "NeoCore is faster" is not "NeoCore is better."
• Report distributions with variance over k repeats; fixed workload seeds.

LuckPerms backend — how it was actually stood up (2026-08-07)

Two Paper 1.12.2 servers (build 1620, from fill.papermc.io/v3), LuckPerms 5.5.71 bukkit-legacy,
MySQL 8 + Redis 7, sync-minutes: 0. Both in eclipse-temurin:17-jre on --network host.

• Drive it over RCON, not the console. docker attach works exactly once: when the attached
  client disconnects the container's stdin hits EOF and the server stops reading it forever. Enable
  enable-rcon in server.properties and use a small RCON client (harness/luckperms/rcon.py).
• LuckPerms commands are asynchronous. The RCON response returns before the work is done, and is
  usually empty. Verify effects by querying luckperms_user_permissions, not by reading the reply.
• Measure cost with the server counters, not by instrumenting LuckPerms:
  SHOW GLOBAL STATUS LIKE 'Questions' on MySQL and INFO stats on Redis, before and after a known
  number of changes.

Lessons from the first runs (2026-08-07)

• Give every simulated receiver its own gRPC channel. Sixteen streams on a shared channel
  measured faster than eight separate ones, because HTTP/2 multiplexes them; that hides the very
  fan-out cost the sweep exists to measure.
• Receivers must keep pinging. The master registers an observer only after a ClientPubSubPacket
  arrives and drops it after five quiet seconds. A silent receiver stops receiving with no error.
• Read results from the CSV, not from the driver script's stdout. Backgrounded processes get
  killed at command boundaries in some environments; the CSV survives.
• Record nproc and load average with every run. A smoke run on a host at load 13.77/12 cores
  produced a p95/p50 ratio of ~19x for a loopback push, which measures the scheduler.

Threats to validity

• Single-host testbed removes network variance → optional tc/netem condition.
• LuckPerms runs in real (small) servers; LCore/NeoCore paths are extracted headless → disclose the asymmetry, validate once against a full run if feasible.
• NeoCore reached staging but never production → it has been exercised end-to-end by real servers, but is not a production baseline (LuckyCore is). Disclose this asymmetry.
• Stated confound: topology and transport co-vary (both P2P systems use Redis; the master-slave system uses gRPC). The study compares whole architectures; isolating one variable is future work.