Edition 36: Seven memory systems averaged 27.9% when facts kept changing

An assistant can remember the newest fact and still fail the question.

Suppose a research lead changed one policy three times:

Ask for the current policy and the answer is “peer-reviewed only.” Ask what applied after the second update and the answer is “peer-reviewed plus preprints.” Ask how many distinct policies existed and the answer is two, not three.

The useful finished artifact is a revision ledger that can produce all three answers with dates attached. A single “current policy” field cannot reconstruct the February state. A bag of three semantic memories can return the wrong one without showing which version governed when.

This is the reader task for this edition: give a research or project assistant enough temporal structure to answer current-state, historical and change-count questions without pretending that a small script is a complete memory system. A plain current-state record is sufficient when only the latest value matters. The ledger earns its extra work when audit, comparison or lookback questions matter.

The research question is: What should an application owner preserve when an assistant must reason about facts that change, rather than merely recall one similar passage?

Seven systems averaged 27.9 percent

LongMINT, released May 18 by Hyunji Lee and five coauthors, tests memory under repeated updates and interference. Interference means that an older fact can disrupt recall of a newer one, or a newer fact can obscure an older state that a historical question still needs.

The benchmark contains 15,531 question-answer pairs across state tracking, multi-turn dialogue, Wikipedia revisions and Git commit histories. Contexts average 138,800 tokens and 86 ordered updates, with the largest reaching about 1.8 million tokens. Its five question types cover current facts, historical lookback, ordering, counting and multistep reasoning.

The authors evaluated full-context input, standard retrieval, graph retrieval and trained memory managers with specified answering and embedding models. Across seven evaluated systems, average exact-match accuracy was 27.9%. The best overall system, MemAgent, reached 33.4%.

Those figures are paper results, not production failure rates. Exact match is a strict metric, the domains and prompts shape the outcome, and some question generation used Gemini 3.1 Pro. Six annotators checked 20% of generated Wiki and Git samples, with 95.6% judged valid. The benchmark authors also publish the LongMINT code and prompts, which makes the construction inspectable.

The most useful split is not the leaderboard. Simple recall averaged 47.5%, while historical lookback averaged 21.0% and multi-target aggregation averaged 26.5%. Memory-management systems gained an average 9.9 percentage points over non-memory baselines on the short, symbolic bAbI domain, yet lost an average 3.0 points on the longer domains with evolving revisions.

That result changes the design question. “Did the assistant store something relevant?” is too weak. Ask whether it can identify the governing version, recover a prior version and combine several dated changes without treating recency as truth.

Public bug reports show the same reader problem at smaller scale

Two dated reports establish practical demand for this question without claiming search volume or incident frequency.

On April 24, a user filed a Mem0 issue about ADD-only extraction. The report describes a mutable fact such as an employer changing from Company A to Company B. Both facts remain, and retrieval can rank the older one first because the scoring path described in the issue does not incorporate recency.

On June 29, a contributor reported that LangGraph's InMemoryStore retained stale vectors after an update made with index=False. The stored value changed, but a search for the old text could still return the item with its old similarity score.

These are public issue reports, not verified customer incidents. They concern different implementations and failure paths. Together with LongMINT, they show why updating a value, updating its index and preserving usable history are separate operations.

A bounded commissioning scan on November 9 used five query variants: “AI agent memory stale facts,” “agent memory revised information benchmark,” “long-term memory outdated facts LLM,” “how to track changing facts AI agent” and “LongMINT memory interference.” The first results included vendor guides, benchmark directories, academic papers and implementation advice. That sample helped avoid writing another general memory overview. It does not establish rankings, reader traffic or market demand. Search-volume and keyword metrics are NOT_AVAILABLE.

Store events before asking retrieval to choose a truth

For a changing field, preserve four things:

1. The stable key. “source_policy” identifies the fact that changes.
2. The value. Store the exact state that applied at that point.
3. The effective date or sequence. Without order, “current” and “before” have no reliable meaning.
4. The source. A production ledger should point to the message, document, commit or system record that introduced the change.

Do not erase history if the application must answer historical questions. Do not leave contradictory entries as an unordered pile if the application must answer current-state questions. Preserve the events, then derive a current view from their order.

This does not require a vector database. For one project with a handful of explicit policy changes, a dated table or versioned document can be clearer. Retrieval becomes useful when the history is too large to inspect manually, but similarity should locate candidate evidence rather than silently decide which version is authoritative.

A procedure for one changing field

Use this seven-step route before adding model memory:

1. Choose one mutable field. Start with a policy, owner, limit or preferred output format whose changes can be dated.
2. Write one current and one historical question. For example, “What is the current source policy?” and “What applied after the February update?”
3. Collect every change event. Record the effective date, exact value and source pointer. Keep unknown dates visibly unknown instead of guessing order.
4. Validate the event shape. Reject missing keys, values or dates before they enter the ledger.
5. Sort and group by stable key. The ordered group is the history. Its final valid event supplies the current view.
6. Answer from the required slice. Current questions use the latest event. Lookback questions select the named position or date. Counting questions operate across the whole ordered history.
7. Show the evidence with the answer. Return the value and governing date. For consequential decisions, retain the source pointer for human review.

A research lead can inspect the small table directly. An assistant becomes useful when many fields and sources must be combined, but the human still decides whether two statements are true revisions of the same field or separate facts.

Optional clean-folder revision check

This credentials-free exercise implements the ledger rule on four synthetic events. It does not run a language model, install a memory product or reproduce LongMINT.

Prerequisites: Python 3.10 or later and an empty folder. Save this complete file as revision_ledger.py.

from datetime import date

events = [
    {"date": "2026-01-10", "key": "source_policy", "value": "peer_reviewed_only"},
    {"date": "2026-02-14", "key": "source_policy", "value": "peer_reviewed_plus_preprints"},
    {"date": "2026-04-02", "key": "source_policy", "value": "peer_reviewed_only"},
    {"date": "2026-05-07", "key": "output_format", "value": "two_page_brief"},
]

bad_event = {"date": "", "key": "source_policy", "value": "web_sources_allowed"}


def validate(event):
    missing = [name for name in ("date", "key", "value") if not event.get(name)]
    if missing:
        raise ValueError("missing:" + ",".join(missing))
    date.fromisoformat(event["date"])


for event in events:
    validate(event)

history = {}
for event in sorted(events, key=lambda item: item["date"]):
    history.setdefault(event["key"], []).append(event)

policy = history["source_policy"]
print(f"CURRENT source_policy={policy[-1]['value']} as_of={policy[-1]['date']}")
print(f"AFTER_UPDATE_2 source_policy={policy[1]['value']} as_of={policy[1]['date']}")
print("ORDER source_policy=" + " -> ".join(item["value"] for item in policy))
print(f"DISTINCT source_policy={len({item['value'] for item in policy})}")

try:
    validate(bad_event)
except ValueError as error:
    print(f"REJECTED bad_event reason={error}")

print(f"CHECK events={len(events)} keys={len(history)} invalid=1")

Run it from the clean folder:

python3 -I revision_ledger.py

The complete independently executed output is:

CURRENT source_policy=peer_reviewed_only as_of=2026-04-02
AFTER_UPDATE_2 source_policy=peer_reviewed_plus_preprints as_of=2026-02-14
ORDER source_policy=peer_reviewed_only -> peer_reviewed_plus_preprints -> peer_reviewed_only
DISTINCT source_policy=2
REJECTED bad_event reason=missing:date
CHECK events=4 keys=2 invalid=1

The repeated value is intentional. Three updates produced only two distinct policies. The negative case rejects a plausible value because its missing date makes temporal questions unsafe.

If Python reports an invalid ISO date, use YYYY-MM-DD. If two changes share a date, add a sequence number or full timestamp rather than relying on input order. If a query asks about a period before the first recorded event, answer unknown. If two sources disagree about the same effective time, preserve the conflict and ask for a ruling instead of selecting the most similar sentence.

To adapt the example, replace the synthetic keys and values with sanitized project events. Add a source field and require it in validate. Store live operational facts, such as permissions or account status, in their authoritative system and fetch them at decision time rather than trusting an old memory.

What completion looks like

The paper establishes that evolving, interference-heavy histories are difficult for the evaluated systems, especially for lookback and aggregation. The two issues establish concrete stale-state bugs in named open-source projects. The local script establishes only that one explicit ledger produces the displayed deterministic answers and rejects an undated event. It says nothing about model accuracy, deployed reliability, costs or funded outcomes.

A completed reader implementation can answer a current question, a historical question and a distinct-change question from the same ordered evidence. It rejects an event that cannot be placed in time and shows the date beside each answer.

The next experiment should compare three matched routes on a small set of real, sanitized project changes: latest-value overwrite, unordered semantic retrieval and the revision ledger. Record which evidence each route returns for current, historical, ordering and counting questions. Then add a model only at the answering stage. That separation will show whether a failure came from storage, retrieval or reasoning, which is the distinction LongMINT says current systems often blur.