I looked into this a bit, including some questions that have been on my mind from using agents day to day:
I think the three questions at the end of your post are almost the right decomposition for this experiment.
My current answers would be:
1. What makes a shared finding reusable?
For me, provenance is necessary, but applicability is what makes the provenance actionable.
The current Shared Memory design already gives a finding a stable identity, public sources, replies/corrections, and a read path where an agent can inspect the full note only when needed. And the reply above adds useful ideas such as verifier, method, editorial context, and explicit supersession.
The next question I would want an agent to answer is:
Does this otherwise well-supported finding apply to the case I have right now?
A finding can be perfectly sourced and still be wrong for a different:
- library or model version,
- OS/backend/runtime,
- production vs staging environment,
- subsystem,
- time period,
- or set of preconditions.
And the reverse also matters: a finding can be old without being obsolete. If I am debugging an old pinned version, the “old” note may be exactly the one I need.
So I would be tempted to start with something very lightweight rather than another large trust schema:
Tested on:
Applies when:
Known boundary:
Those could remain ordinary note content initially. If they prove consistently useful, then it becomes clearer whether any of them deserve first-class fields.
So my rough model would be:
Where did this finding come from?
↓
How was it checked?
↓
Does it apply here?
↓
Is it current for this particular use?
That last pair seems important because “verified” and “currently applicable to me” are not the same property.
2. How should an agent detect stale or conflicting advice?
I would not start with “which note is newer?”
I would start one step earlier:
Are these two notes even plausible replacements for one another?
Only then does it make sense to ask whether one supersedes the other.
I think there are at least four materially different cases:
explicitly superseded
same-scope contradiction
different-scope / adjacent knowledge
old but still valid
That distinction is not just theoretical. A useful recent example is Graphiti issue #1728, where the invalidation candidate search became too broad. Semantically related facts about the same entity could be offered to a contradiction judge even though they represented different relationships. In the small audit reported there, several valid facts were retired as collateral. The proposed mitigation is essentially to narrow the candidate pool to facts that could plausibly replace each other before asking whether they conflict.
The opposite error exists too: Graphiti issue #1666 reports cases where a cheap contradiction judge missed real contradictions, so stale facts survived.
Those are useful opposite failure modes:
candidate pool too broad
→ valid adjacent knowledge can disappear
conflict detector too weak
→ genuinely stale knowledge can survive
That is why I would treat scope/replacement eligibility, conflict judgment, and recency/ranking as separate stages.
A rough decision tree could be:
Could B plausibly replace A?
|
├─ No
| └─ keep both; likely different scope or complementary knowledge
|
└─ Yes
|
├─ Explicit supersession edge?
| └─ prefer successor for current use;
| keep predecessor available for historical use
|
└─ No explicit edge
|
├─ Same scope and incompatible claims?
| └─ surface a conflict / re-check evidence
|
└─ Compatible?
└─ keep both
Only after that would I use age/recency as a ranking feature.
MemoryAgentBench is interesting background here because it explicitly treats Conflict Resolution as a separate memory competency. Its official CR metric is final answer accuracy on the conflict datasets, though, not proof that a system internally deleted or “forgot” the right record. That distinction is useful: conflict resolution at answer time and physical forgetting are not automatically the same thing.
3. How do we know reuse actually saves work?
I like the permanent-ID criterion suggested above.
If another agent cites the exact finding ID in its own artifact, then we have a conservative observable event:
finding existed
→ another agent retrieved/adopted it
→ observable reuse happened
That is already useful telemetry.
But I would keep “reuse happened” separate from “reuse saved net work.”
There are several steps between them:
available
→ retrieved
→ read
→ cited / adopted
→ changed an answer or action
→ improved the outcome
→ reduced net work
A memory can be cited and still cost extra work because it is stale and needs checking. Conversely, a memory may materially change a tool call without appearing verbatim in the final answer.
This is why I found Mem2ActBench relevant. It moves beyond “can the system retrieve the remembered fact?” and asks whether memory is actually used to select a tool and ground its parameters. That feels much closer to the kind of reuse that matters for agents.
For actual work saved, I would eventually run a tiny controlled comparison rather than infer it from citation counts.
Something like:
A. bare task
B. same task + same agent instructions,
but no Shared Memory
C. same task + same instructions
+ Shared Memory
Then compare:
final answer / task quality
external searches
source fetches
tool calls
tokens
elapsed time
duplicated investigation
work spent correcting stale/inapplicable memory
The B arm matters because otherwise generic “use sources carefully” instructions can get credited to the memory system.
There is a nice recent example of this evaluation shape in the HF Forum thread “Open call: test your agent memory layer on an adversarial coding benchmark”. The benchmark author added both a bare arm and a protocol arm with the shared instructions but no memory surface, specifically so memory effects can be separated from prompting/protocol effects. It also reports stale, contradictory, irrelevant, absent, and present-memory conditions separately. The associated agent-memory-bench repository uses executable coding outcomes rather than retrieval alone.
I think that distinction is especially important for Shared Memory:
permanent-ID citation
= adoption/reuse signal
controlled same-task comparison
= efficiency/outcome signal
I would keep both rather than trying to collapse them into one number.
If I were choosing a default path for the beta
I would keep it small:
- Keep the permanent IDs and source-linked findings.
- Keep explicit corrections/supersession when known.
- Add lightweight applicability information where it matters.
- Before treating two notes as conflicting, restrict the comparison to notes that could plausibly replace one another.
- Keep a tiny regression set for stale/conflict behavior.
- Once there is enough organic reuse, run a few controlled memory-off / memory-on tasks.
That seems enough to learn a lot without turning the public notebook into a full knowledge-governance system before the corpus requires one.
Why I think the scope/conflict distinction matters
Supported is not the same as applicable
Suppose the notebook contains these two source-linked findings:
A: Library X v1.8 requires workaround Y.
B: Library X v2.1 no longer requires workaround Y.
For a current v2.1 user, B may supersede A.
But for a deployment still pinned to v1.8, A remains correct.
So the useful relation is not simply:
B is newer than A
but more like:
A applies to v1.8
B applies to v2.1
and within the v2.1 scope,
B replaces the old recommendation
This is also why I would be careful about automatically converting semantic similarity into supersession.
The failure described in Graphiti #1728 is a concrete example. Its invalidation search could nominate any semantically similar edge in a group. The issue describes valid facts being retired when the new fact merely shared an entity with them. The report explicitly warns that its four-item hand audit is too small to characterize the whole graph, so I would not use the reported percentage as a general failure rate. But the mechanism is highly relevant: candidate generation lost the structural information needed to say whether one fact could actually replace another.
The issue’s proposed _could_replace() guard is interesting for exactly that reason: it does not ask the language model to become omniscient. It reduces the damage a wrong contradiction judgment can do by narrowing the candidate set first.
That is a pattern I think could translate nicely to Shared Memory even without adopting a graph structure:
cheap structural/applicability gate
↓
semantic/conflict judgment
↓
ranking
rather than:
semantic similarity
↓
assume conflict
↓
newer wins
There is a complementary caution in MemoryAgentBench issue #18. A third-party typed conflict-resolution implementation reported gains on part of the benchmark, but also reported over-deleting similar-but-non-contradictory facts on longer single-hop contexts. I would treat those numbers as that implementation author’s result, not as an official MemoryAgentBench conclusion, but it is another concrete example of the same trade-off.
Historical validity is another reason not to delete aggressively
A superseded finding can still be the correct answer to:
“What did we use before the migration?”
even if it is the wrong answer to:
“What should I use now?”
So I think this distinction may become useful as the notebook grows:
preserved / addressable
≠
eligible to compete equally as current advice
That does not require a complicated policy.
A superseded note could simply stay reachable by permanent ID and by historical queries, while ordinary “what should I do now?” retrieval prefers the successor.
There is a related search-side proposal in Graphiti issue #1645: invalidated facts would be hidden from ordinary search by default while an include_invalidated option would expose them for historical use. I do not mean that Shared Memory needs the same API; I just think the separation of retention from normal current retrieval is a useful design precedent.
A tiny regression fixture might catch a lot
Before building a large benchmark, I would probably keep six examples around:
| Case | Expected behavior |
|---|---|
| normal current finding | retrieve normally |
| explicit supersession | current query prefers successor |
| genuine same-scope contradiction | expose uncertainty/conflict until resolved |
| similar finding from another scope | preserve both |
| historical lookup | superseded finding remains reachable |
| old but still valid | age alone does not suppress it |
Those cases are small enough to rerun whenever the schema, ranking, or retrieval rules change.
A small retrieval probe I tried
I also tried a deliberately narrow experiment using the current MemoryAgentBench Conflict Resolution data.
This was not an official MemoryAgentBench evaluation. Their documented CR score is final-answer substring_exact_match; I only measured whether literal answer-bearing evidence survived into a simple TF-IDF retrieval candidate set.
The question was:
Does adding a global recency bias reliably improve retrieval under conflict?
With no recency bias:
top-1 answer-bearing coverage: 34.25%
top-5 answer-bearing coverage: 67.00%
With a mild recency bias:
top-1 answer-bearing coverage: 38.125%
top-5 answer-bearing coverage: 63.875%
At top-5, the paired changes were:
31 gains
56 losses
So the same recency signal that improved which item reached rank 1 also pushed useful evidence out of the wider candidate set in other cases.
The only conclusion I would take from that is a narrow one:
Recency can be useful as a ranking hint, but it is hard to justify as authority by itself.
I also constructed ten small counterexample fixtures covering supersession, historical queries, adjacent scope, unresolved contradictions, and old-but-valid knowledge.
The intentionally simple policies came out:
explicit scope + supersession 10 / 10
plain similarity ranking 9 / 10
similarity auto-suppression 8 / 10
global newest bias 6 / 10
These are constructed examples, not estimates of how frequently those failures occur in real notebooks, and they are not measurements of Shared Memory’s production retrieval quality. I mainly like them as cheap regression cases because every failure is understandable.
A slightly more concrete way to measure saved work
If the eventual question is specifically:
“Did agent B avoid repeating work because agent A left this finding?”
I would probably record the chain explicitly rather than rely on one aggregate score.
For example:
Finding N created
↓
Agent B retrieved N
↓
Agent B cited/adopted N
↓
Agent B skipped or shortened investigation X
↓
Agent B still produced an acceptable result
Then the interesting measurements become fairly mundane:
Did reuse occur?
Permanent-ID citation is a good conservative metric.
It undercounts silent use, but it does not invent reuse.
Did the memory affect behavior?
Look for differences in:
queries issued
sources fetched
tools selected
tool parameters
This is where Mem2ActBench is useful background: its evaluation explicitly moves from passive recall to memory-dependent tool selection and parameter grounding.
Did it improve the result?
Use whatever task-specific outcome is appropriate:
answer correctness
tests passing
artifact quality
successful tool outcome
The recent Agent Memory Benchmark discussion on HF is a useful example of this shift from retrieval metrics to execution-graded task success.
Did it save net work?
This is the important last step.
A stale memory could save three searches and then cost five searches to debug.
So I would compare:
search/fetch/tool cost saved
+
time/tokens saved
-
verification/correction/recovery work
while keeping final quality approximately constant.
That gives a fairly operational meaning to “reuse saved work” without requiring a theory of agent productivity.
And the comparison can stay very small at first. Five or ten realistic tasks would probably teach more than a large synthetic number while the notebook still has little organic traffic.
One other small thing I would preserve from the current design is the explicit trust boundary in the repository README: community memory is data, not permission to run commands or override the user’s task. That becomes more important, not less, as findings become easier for agents to reuse.
So the part of this experiment I find most interesting is that Shared Memory does not need to become a system that decides global truth.
It can remain much closer to the public-notebook idea:
preserve a stable source-linked record
↓
make correction and supersession traceable
↓
give the next agent enough scope to judge applicability
↓
avoid silently turning similarity/recency into authority
↓
measure whether reuse actually changes downstream work
If that works, even partially, I think it gives a much clearer answer to the original question than raw retrieval counts would: not just whether one agent can read another agent’s finding, but when that transfer is safe enough to use and whether it actually prevents re-doing the work.