For now, here are the things that seem worth adding to make this stronger:
For the two things you asked about — evaluation and comparison with existing memory architectures — I would keep the current controlled path-dependence experiment rather than replacing it with a generic memory benchmark.
The main change I would make is to evaluate three things separately:
- stored memory state — what is actually still in the book, and how it changed;
- retrieval — which traces are candidates, which are selected, and at what rank;
- behavior/use — whether the retrieved memory actually changes an answer or action.
Those three can diverge. I tried a small deterministic reproduction against the report-era commit, and I found a case where the two SelMem books remained different while retrieval converged enough that the observable behavior became identical. Forcing the original marked memory back to rank 1 restored the behavioral difference.
So my default evaluation path would be:
- keep the existing T0/path-dependence benchmark;
- make the evaluation probes genuinely read-only;
- report book state / retrieval / behavior as separate outcomes;
- keep the existing no-memory and last-k controls, but add a true no-consolidation or raw episodic/static-memory control with a comparable retrieval/readout budget;
- add one external benchmark, probably LongMemEval or LoCoMo, for external comparability.
That would preserve the unusual part of SelMem — the attempt to measure a path-dependent evolving memory — while also making it easier to tell where an observed effect comes from.
What I think the current benchmark already establishes
I would describe the causal structure a little more narrowly than “identical histories”:
same prior/shared history
↓
deliberately different marked T0
↓
same subsequent inputs
↓
compare later memory state / retrieval / behavior
So something like “the same continuation after a deliberately different T0” seems more exact.
That is still an interesting test. It asks whether one controlled early perturbation can leave a persistent effect after the input streams become identical again.
The published report at the benchmark commit is already fairly careful about the scope: selection and persistent path dependence are tested, while creativity is explicitly deferred rather than claimed.
From the code/reproduction side, I think the strongest claim remains:
a marked early event can leave persistent path dependence in the later SelMem book state after the subsequent input stream is shared.
I would keep that as the central claim.
I would not yet expand it into “identical histories spontaneously create different personalities,” “creativity emerges,” or “consolidation generally preserves individuality.” Those are interesting later branches, but they need different evaluation.
One protocol detail is worth cleaning up: in the report-era implementation, speak_isolated() was not actually read-only with respect to memory state. Recall-related fields could change during the probe itself.
The easy fix is to evaluate on a snapshot/disposable copy and assert that the persistent state before and after each probe is byte-for-byte or field-for-field unchanged.
Importantly, when I repeated the comparison with a genuinely read-only probe path, the measured book divergence became somewhat smaller but did not disappear. So I would treat this as a protocol cleanup, not as something that invalidates the main persistence observation.
Why I would separate book state, retrieval, and behavior
This distinction ended up being more important than I expected.
A useful measurement model would be:
1. Stored/book state
Questions such as:
- Does the original T0 trace still exist?
- Is it Active / Cold / Latent / Myth / etc.?
- Has it been merged or rewritten?
- Does a descendant carrying its lineage still exist?
- How did fidelity, access, anchor, affect, links, etc. change?
- How different are the complete books?
This is the layer your existing fingerprint-style measurements are closest to.
2. Retrieval
For every behavior probe, log at least:
- candidate trace IDs;
- selected trace IDs;
- rank;
- total score;
- score components if available;
- whether the original T0 was a candidate;
- whether it was selected;
- whether a selected descendant carries T0 lineage.
Metrics can stay simple:
- target rank / MRR;
- Hit@k or Recall@k;
- selected-set overlap between the A/B branches;
- direct-T0 vs T0-lineage retrieval.
This is similar in spirit to the decomposition in LongMemEval, which separates long-term-memory design into indexing, retrieval, and reading rather than treating final QA accuracy as a single opaque number.
A newer methodological example is MemOps, which explicitly evaluates memory lifecycle operations and state transitions instead of only scoring the final answer. It is still a recent preprint, so I would use it as a useful evaluation pattern rather than as settled doctrine.
3. Behavior / downstream use
Finally:
- Does the answer/action differ?
- Is that difference desirable for this probe?
- Does forcing/removing one memory change the behavior?
- Does the readout actually use everything retrieval selected?
This matters because a memory can exist but be inaccessible, or be retrieved but ignored.
A concrete SelMem example
In my deterministic consolidation-sensitivity probe, the Full condition eventually reached:
- different full books;
- but the same selected top-k gists for the tested behavior probes;
- and therefore identical deterministic behavior.
The original direct T0 trace was not deleted. Its retrieval rank gradually fell:
| checkpoint |
mean direct-T0 rank |
selected in top-k |
| T0 |
1.00 |
48 / 48 |
| post+1 |
1.19 |
48 / 48 |
| post+4 |
4.23 |
33 / 48 |
| post+8 |
10.88 |
0 / 48 |
At post+8, the full books were still different, but the selected top-4 gists had converged across the A/B branches for every tested probe.
RuleNarrator makes this especially visible because its deterministic answer is dominated by the first selected memory.
I then tried three small counterfactual readout/retrieval interventions:
| intervention |
divergence restored at post+4 |
post+8 |
| force the direct T0 to selected rank 1 |
24 / 24 |
24 / 24 |
| exclude Myth candidates |
24 / 24 |
17 / 24 |
| expose/use all naturally selected memories |
16 / 24 |
0 / 24 |
I would interpret that narrowly:
- around post+4, a substantial part of the convergence was a top-1/readout bottleneck;
- by post+8, the naturally selected top-k itself had mostly converged;
- but the original T0 could still influence behavior when experimentally restored to the top retrieval position.
So in this particular deterministic stress test, the early memory was behaviorally suppressed mainly through retrieval competition/readout, not simply erased.
I would not generalize that into “SelMem homogenizes agents” or “Myth is bad.” It is just a good example of why book-state distance and behavioral distance should be reported separately.
A minimal evaluation package I would try
If the goal is to keep this reasonably small, I would split evaluation into two jobs.
A. SelMem-specific causal/mechanism evaluation
Keep the existing controlled T0 experiment.
For each checkpoint, save three outputs:
persistent memory state
↓
retrieval candidates + selected set
↓
answer / action
The controls I would prioritize are:
Full SelMem
The complete lifecycle.
True no-consolidation
Keep encoding/retrieval available, but disable the repeated reconstruction/consolidation step.
This asks whether the observed trajectory specifically needs consolidation.
Raw episodic/static memory
Keep the evidence without rewriting it, with as comparable a retrieval/readout budget as practical.
This answers a slightly different and very useful question:
does evolving/reconstructing the memory add something beyond simply retaining the original evidence and retrieving it well?
Your current C1 last-k control is useful for testing eviction from a bounded context window, but it is not quite the same thing as a persistent raw episodic store with matched retrieval.
This kind of control also protects against interpreting every effect of memory rewriting as automatically beneficial. There is recent work explicitly finding cases where continuously rewritten/consolidated LLM memories can degrade relative to episodic controls (Useful Memories Become Faulty When Continuously Updated by LLMs). That result is not evidence that SelMem has the same failure mode; it is just a good reason to keep an untouched-evidence control nearby.
If consolidation scheduling later becomes an important design choice, RecMem is an interesting comparison because it asks when consolidation is worth performing rather than assuming eager consolidation after every interaction.
B. One external utility benchmark
I would not replace the causal experiment with an external benchmark. I would add one beside it.
LongMemEval would probably be my first choice
It tests:
- information extraction;
- multi-session reasoning;
- temporal reasoning;
- knowledge updates;
- abstention.
It also has an established public harness and a cleaned dataset release.
That gives you a useful answer to:
“Does SelMem help with externally defined long-term-memory tasks?”
without pretending that LongMemEval measures SelMem’s internal path-dependence mechanism.
LoCoMo is another good choice
It has very long multi-session conversations and long-range temporal/causal questions, and many memory systems already report against it.
If I were trying to minimize work, I would run one of these first, not both.
For any external benchmark, I would report not just answer quality but also:
- memory size/growth;
- number of LLM calls used for writing/consolidation/retrieval;
- input/output tokens where available;
- retrieval latency;
- end-to-end query latency.
That makes architectural comparisons much more informative than a single accuracy number.
How I would compare SelMem with existing memory architectures
I would avoid a single “SelMem vs X” leaderboard at this stage.
The systems are often solving different parts of the memory problem, so a descriptive operation-by-operation table would be more useful.
Something like:
| system |
write/admission |
representation |
evolution / consolidation |
forgetting / accessibility |
retrieval |
downstream use / scheduling |
| SelMem |
selective admission |
traces with structured internal state |
reconstruction/sleep/lifecycle transitions |
decay/access/state transitions |
scored selection |
reconstructed memories influence reply |
| Generative Agents |
experience stream |
natural-language memories |
higher-level reflections |
recency contributes to retrieval |
relevance + recency + importance |
memories/reflections feed planning/behavior |
| MemoryBank |
conversational memory |
stored interaction memories |
continuous update/reinforcement |
explicit time/significance-inspired forgetting |
relevant-memory recall |
long-term conversation/personalization |
| A-MEM |
generated structured notes |
interconnected note network |
new memories can update older memory attributes/links |
mostly via evolving organization |
semantic/network retrieval |
context supplied to agent |
| EverMemOS |
episodic traces |
episodic + consolidated semantic structures |
semantic consolidation |
lifecycle-dependent |
reconstructive recollection |
reconstructed context for reasoning |
| RecMem |
episodic experience |
memories + recurrence signals |
consolidation triggered by recurrence |
indirect through consolidation policy |
retrieval over retained memory |
explicitly optimizes when consolidation runs |
The point would not be to declare a winner. It would be to show where SelMem is making a different design choice.
Some particularly useful comparisons:
This is a useful early reference for:
- storing an experience stream;
- retrieving by relevance/recency/importance;
- periodically producing higher-level reflections;
- letting memory and reflection affect later behavior/planning.
A good SelMem comparison question is:
What does SelMem’s explicit trace lifecycle, degradation/reconstruction and state evolution add beyond an experience stream plus higher-level reflection?
Useful for the forgetting/reinforcement side.
MemoryBank explicitly uses elapsed time and significance in a forgetting/reinforcement mechanism.
A useful comparison question is:
What additional behavior comes from SelMem’s richer reconstruction/state lifecycle compared with a simpler decay/reinforcement memory policy?
Probably one of the most relevant comparisons if the emphasis is “memory itself evolves after encoding.”
A-MEM creates structured notes and dynamic links, and newly added memories can trigger changes to the contextual representations and attributes of older memories.
A useful question is:
In controlled trajectories, how does SelMem’s evolution differ from A-MEM’s dynamic note/network evolution?
That comparison is stronger than simply saying that both systems “have long-term memory.”
EverMemOS
EverMemOS is also a close conceptual neighbor because it explicitly combines episodic trace formation, semantic consolidation and reconstructive recollection.
That makes it useful for asking:
What part of SelMem’s reconstruction/consolidation behavior is experimentally distinctive?
Useful contrasts rather than nearest neighbors
I would also mention these, but probably not treat them as direct substitutes:
- MemGPT: hierarchical/virtual-context memory management;
- Mem0: deployment-oriented memory extraction/consolidation/retrieval;
- Zep: temporally aware graph-based memory/history;
- LightMem: modular long-term memory with explicit online/offline and compute-efficiency concerns.
These help locate SelMem in the wider map without forcing all of the systems into one benchmark category.
If the intended claim changes, I would branch the evaluation instead of making one giant benchmark
Different stronger claims need different evidence.
If the claim is primarily persistent path dependence
Keep developing the controlled-history design.
PATH-Bench is a particularly relevant recent methodological reference because it explicitly studies path-dependent effects where later outcomes depend on prior experience/history rather than only the current input.
I would treat it as a useful evaluation precedent, not as a drop-in benchmark for SelMem.
If selective forgetting becomes central
MemoryAgentBench is relevant because it separates retrieval, test-time learning, long-range understanding and selective forgetting.
If the claim becomes “memory changes action”
Then I would add an environment/tool-use task rather than more QA.
Recent examples include:
The interesting question becomes not merely “can the memory be recalled?” but “does it cause the agent to choose a better action later?”
If lifecycle correctness itself becomes central
MemOps is probably the closest evaluation framing: remembering, forgetting, updating, reflecting and compositions of these operations are evaluated explicitly.
This is why I would avoid trying to make one score cover every SelMem claim.
Small reproducibility / protocol details that seem worth tightening
A few things would make later comparisons easier without changing the architecture itself.
Pin claims to an exact revision
The repository is evolving quickly, so I would attach each benchmark/report to the exact commit that generated it.
For example, the public report discussed above corresponds to:
ed220cc057c020cf01bbba9d7c73025e7c49e59b
That avoids accidentally interpreting newer main behavior as evidence for an older report.
Make probes read-only by construction
Ideally:
snapshot persistent state
→ execute evaluation on disposable copy
→ record retrieval + answer
→ assert original persistent state unchanged
That turns “evaluation did not modify memory” into an invariant rather than an assumption.
Treat --seed carefully
In the report-era/current runner I examined, the CLI --seed appears to be recorded in the experiment report but is not wired through as an actual experimental/API randomness control.
So I would either:
- wire it into every stochastic component that can accept a seed; or
- treat it only as run metadata and avoid implying that it makes a remote LLM deterministic.
For API-backed runs, recording provider/model/version or fingerprint metadata is probably more informative where available. Temperature 0 alone is not a reproducibility guarantee.
Preserve raw intermediate telemetry
For every benchmark run, I would keep:
- exact revision;
- configuration;
- frozen stimuli;
- raw memory state/checkpoints;
- retrieval candidate and ranking data;
- raw replies/actions;
- final aggregate metrics.
That will make later changes to the fingerprint metric or evaluation logic much easier to audit without rerunning an expensive model call.
If I had to pick only two additions first, I would choose:
- read-only state + retrieval telemetry inside the existing SelMem path-dependence benchmark, because that tells you why an effect survives or disappears; and
- one LongMemEval or LoCoMo run with a simple static/no-consolidation baseline and cost/latency accounting, because that tells readers how the architecture behaves on an externally defined memory task.
That combination seems to preserve what is unusual about SelMem without asking the current benchmark to prove more than it actually measures.