Experiment Record · 2026-07-22
MEM-SHARED-3K
One contract, three memory systems: Cortex vs SAM vs SAME
Hypothesis
Three local persistent-memory architectures can only be ranked fairly under a single deterministic contract — one event generator, one answer key, one stale definition, one payload counter.
Setup
A deterministic stream of durable project decisions, explicit revisions, non-durable status noise, and questions issued by a different agent than the writer, replayed into all three adapters at 250 / 1,000 / 3,000 events across seeds 0, 1, and 2. Each system used its native transport: Cortex typed memory records, SAM MTROI-selected reconstruction text, SAME structured runtime patch. Persistence was tested by closing and reopening each isolated store. Zero LLM calls, zero mutations to live production memory, temporary stores only.
Variables
- ◆Memory architecture (Cortex / SAM / SAME)
- ◆Event-stream scale
- ◆Random seed
Results
- →Cortex at 3,000 events: 99.3% current-answer accuracy, 0.7% stale, 0.0% missing, 30.3 normalized payload tokens/query, 2.936ms p95 recall, 100% restart accuracy.
- →SAM at 3,000 events: 45.5% accuracy, 36.2% stale, 18.2% missing, 219.1 tokens/query, 4,213ms p95 recall, 49.3% restart accuracy.
- →SAME at 3,000 events: 28.3% accuracy, 0.0% stale, 71.7% missing, 24.0 tokens/query, 8.134ms p95 recall, 0% restart accuracy.
- →All three passed their native regression suites (SAM: 34 passed, 1 skipped).
Observations
- ·Native benchmarks flattered everyone — SAME 100% accuracy, Cortex 100%, SAM 97.7% — but each measures a different token model and workload, so none of them ranks anything.
- ·SAME never returned a stale value at any scale. It withholds rather than asserting an obsolete fact — a genuine safety property, entirely separate from its recall failure.
- ·SAM's recall p95 degraded from 150ms at 250 events to 4.2 seconds at 3,000: a scaling problem, not a tuning problem.
- ·The shared harness itself cost 2,889s (~48 minutes). Reporting evaluation overhead is part of an honest benchmark.
Conclusion
Cortex is the deployment baseline for shared lab memory: stable across scale, ~30-token payloads, no missing answers, and full restart persistence. SAM and SAME stay research systems — their shared-contract adapters did not reproduce their native performance, and that gap is an open investigation, not a disproof of the underlying ideas.
Next Experiment
Diagnose SAME's post-reload runtime-patch loss and SAM's formation/supersession behavior against the same corpus, then require a measured improvement before either enters the critical memory path.