Issue 23: The Forgetting Engine: What a 2,647-Page Wiki Taught Me About Memory
In the back of a good repair shop there is a wall of index cards, one per fault that ever came in and got fixed: the symptom, the cause, the thing that worked. A new mechanic reads the wall before touching anything, because it is the shop's memory and faster than learning the same lesson twice.
The wall has a failure mode, and it is not running out of space. It is the word SOLVED. Someone fixes a fault, writes SOLVED across the card, and nobody ever takes it down or checks whether the fault came back the next week under another name. After a few years the wall is thousands of cards deep, a quarter of them stamped by a mechanic who has since left, and the new hire cannot tell a card still true from one true once. The memory has not filled up; it has stopped being trustworthy, which is worse.
This is the shape of every knowledge base an agent writes and reads back. This fortnight one widely shared post named the last layer of a five-layer memory stack the forgetting engine, the one almost nobody builds, and another declared the whole LLM-wiki pattern dead for accumulating pages nobody retires. Those are one complaint: a wiki with no forgetting engine rots; a forgetting engine has nothing to forget without a wiki.
The studio runs a 2,647-page wiki of exactly this kind, with a forgetting engine on top, and numbers almost nobody publishes. This week's free tool reports every page stamped SOLVED with nothing behind the stamp; the paid flagship gives that check a memory and asks the harder question: did the fault actually stop.
Three from the wave this week, each read through one question: when a page says a thing is handled, what makes that claim worth believing?
- A widely shared post lays out agent memory in five layers and names the last one the forgetting engine, "the layer almost nobody builds, and it decides whether the other four stay worth trusting" (@AnnatarXBT, 25 Oct 2026, x.com/AnnatarXBT/status/2114319310005150101, quoting @Mr_Buzzoni). Its efficiency numbers check out against Mem0's own LoCoMo figures; the forgetting layer is the one line with no number attached, because nobody has shown one running.
- A reel declares the pattern over: "can we stop talking about LLM wikis now and just accept karpathy wasn't right. he moved on, why haven't you?" (@kem_glitch, 29 Oct 2026, instagram.com/reel/DfFSZDvN6oV). The studio dossier is blunter than the reel: Karpathy's April 2026 llm-wiki gist is real and stands, no retraction exists, and every live critique is the same one: an LLM wiki accumulates pages nobody retires and readers cannot tell current from historical.
- Anthropic's Claude Duet (21 Oct 2026), a proactive agent that works while you are away, for the adjacency: the more an agent does unattended, the faster its wiki grows and the fewer pages anyone reads back, exactly when the forgetting engine stops being optional.
Both tools follow The Read. retirewithproof.py, free and runnable below, reports every closed page whose closure it cannot verify. Lesson Recurrence Memory, the paid flagship below the cut, gives that check a memory, grading whether each closed page's fault actually stopped.
The Read: a closed flag is cheap talk until it costs something
The oldest result that fits is from 1982. Vincent Crawford and Joel Sobel asked what a message is worth when it costs the sender nothing and their interests diverge from the reader's. The answer, now textbook, is nothing: a costless message carries no information in equilibrium, because the sender says whatever serves them and the reader discounts it. Economists call it cheap talk, and SOLVED on a card is cheap talk exactly: costless to write, every reason to write it, discounted by anyone burned once.
Michael Spence, nine years earlier, had the other half: a signal becomes worth reading the moment it costs something only the honest type can pay cheaply. So attach to a SOLVED flag a cost a genuine closure can pay and a lazy one cannot. Two clauses do it: name the proof, the mechanism that now refuses the fault, in one checkable line, and name the replacement, the file the lesson now lives in, which must exist. A genuine closure produces both for free; a page that merely stopped being touched cannot, not without lying checkably. That asymmetry is the whole mechanism.
This is the deliberate opposite of where this newsletter started. Issue 2 argued "Stop Retrieving. Start Accumulating.", that keeping what you learn beats re-deriving it. This issue is the bill: accumulation with no closure-with-proof is how you get a 2,647-page wall nobody can read. The forgetting engine is not the enemy of accumulation, it is what keeps it worth doing. A closure that passed only because nobody asked for proof is a cousin of this newsletter's running theme, the check that passes without doing its job.
So the rule the free tool enforces is small and total: a page is closed only if its frontmatter says so, it names a non-empty proof field, and that proof's replacement file resolves on disk. Fail any clause and the page is not retired, only untouched, which is the more honest word.
The twenty-minute version
Point the tool at your wiki folder, require of every closed page a one-line proof field and a replacement path that resolves to a real file, and wire it into whatever closes your pages. The twenty minutes is reading how many pages you thought were handled were only quiet.
A closed page is a claim. Proof is the cost that makes the claim worth reading.
The Tool: retirewithproof.py
The free, from-scratch tool this week. Give it a directory of markdown pages with YAML frontmatter; it reports every page marked closed that fails to prove the closure, with no proof field or a replacement not on disk. Exit 1 if any, 0 if none, 2 if unreadable. Standard library only.
#!/usr/bin/env python3
"""retire_with_proof.py - find closed pages in an agent wiki that were retired
without proof, so a 'forgetting engine' cannot quietly rot the pages it keeps.
An LLM-maintained wiki (Karpathy's April 2026 llm-wiki pattern, and the fifth
'forgetting' layer named but not shipped in the Mem0-centric five-layer memory
stack now circulating as production practice) grows confidently-written pages
faster than anyone retires them. The failure that follows is not that pages are
deleted; it is that pages are marked CLOSED with nothing behind the flag. A reader
cannot then tell a page that was genuinely superseded from one that merely stopped
being touched, and the wiki's errors compound.
This tool is the fifth layer as a standalone check. It reads a directory of
markdown pages with YAML frontmatter and reports every page that is marked closed
(stage or status in a closed vocabulary) but fails to PROVE the closure. A closure
is proven only when it carries at least one non-empty proof field (default
retired_because or lesson_carried_by) AND, where that field names a replacement
file, the file actually exists. A closure with no proof field, or one pointing at
a replacement that is not on disk, is reported.
The design is a direct application of signalling theory. A bare 'closed' flag is
cheap talk (Crawford and Sobel, Strategic Information Transmission, Econometrica
1982): costless to emit, and therefore uninformative once a reader learns it costs
nothing. Requiring the closure to name its proof and its replacement file converts
the flag into a costly signal (Spence, Job Market Signaling, Quarterly Journal of
Economics 1973): only a genuinely closed page can produce a verifiable pointer
cheaply, so the flag regains meaning. This check is the cost.
Input: a directory of *.md pages, each opening with a YAML frontmatter block
fenced by lines of exactly '---'. Only simple 'key: value' frontmatter is read;
this is deliberately stdlib-only and does not depend on a YAML parser.
Exit 0 when every closed page proves its closure. Exit 1 when at least one closed
page is unproven. Exit 2 when the directory cannot be read. Standard library only.
"""
from __future__ import annotations
import argparse
import os
import re
import sys
CLOSED_VOCAB = ("closed", "retired", "superseded", "done", "resolved")
DEFAULT_PROOF_FIELDS = ("retired_because", "lesson_carried_by")
STATE_FIELDS = ("stage", "status")
# a token that looks like a file path: a word with a slash or a known extension.
_PATH_TOKEN = re.compile(r"[\w./-]*\.(?:md|py|sh|txt|json|jsonl|ya?ml)\b")
def parse_frontmatter(text):
"""Return a dict of the simple 'key: value' pairs in the leading YAML
frontmatter block, or None if the page has no frontmatter. A value is taken
verbatim up to end of line, with surrounding quotes and whitespace stripped.
Nested or multi-line YAML values are read only as their first line, which is
all this check needs (it looks at scalar state and proof fields)."""
lines = text.splitlines()
if not lines or lines[0].strip() != "---":
return None
fm = {}
for line in lines[1:]:
if line.strip() == "---":
return fm
m = re.match(r"^([A-Za-z0-9_.-]+):\s?(.*)$", line)
if m:
key = m.group(1)
val = m.group(2).strip().strip('"').strip("'").strip()
# only record the first occurrence of a top-level key
if key not in fm:
fm[key] = val
# no closing fence: not a valid frontmatter block
return None
def closed_state(fm):
"""Return the closing word if the page is marked closed, else None."""
for field in STATE_FIELDS:
raw = fm.get(field, "")
word = raw.strip().lower().split()[0] if raw.strip() else ""
if word in CLOSED_VOCAB:
return word
return None
def replacement_exists(value, root):
"""A proof value may name a replacement file. Return True if it names no file
path at all (nothing to check), or if every file path it names exists relative
to root or to the current directory. Return False only when it names a path
that cannot be found."""
tokens = _PATH_TOKEN.findall(value)
if not tokens:
return True
for tok in tokens:
candidates = (tok, os.path.join(root, tok), os.path.join(root, os.path.basename(tok)))
if not any(os.path.exists(c) for c in candidates):
return False
return True
def inspect_page(path, proof_fields, root):
"""Return (closed_word, problem) for one page. problem is None when the page
is not closed or proves its closure; otherwise it is a short reason string."""
try:
with open(path, encoding="utf-8") as fh:
text = fh.read()
except OSError as exc:
return (None, "unreadable: %s" % exc)
fm = parse_frontmatter(text)
if fm is None:
return (None, None)
word = closed_state(fm)
if word is None:
return (None, None)
present = [f for f in proof_fields if fm.get(f, "").strip()]
if not present:
return (word, "closed as '%s' with no proof field (%s)"
% (word, " or ".join(proof_fields)))
for field in present:
if not replacement_exists(fm[field], root):
return (word, "closed as '%s' but %s names a replacement that is not on disk: %s"
% (word, field, fm[field]))
return (word, None)
def scan(directory, proof_fields, root):
"""Walk directory for *.md pages. Returns (findings, closed_count,
scanned_count). Raises OSError if the directory cannot be listed."""
findings = []
closed_count = 0
scanned = 0
for name in sorted(os.listdir(directory)):
if not name.endswith(".md"):
continue
path = os.path.join(directory, name)
if not os.path.isfile(path):
continue
scanned += 1
word, problem = inspect_page(path, proof_fields, root)
if word is not None:
closed_count += 1
if problem is not None:
findings.append((name, problem))
return findings, closed_count, scanned
def render(findings, closed_count, scanned, directory):
lines = ["RETIRE WITH PROOF: %s" % directory, ""]
lines.append(" markdown pages scanned : %d" % scanned)
lines.append(" pages marked closed : %d" % closed_count)
lines.append(" closures without proof : %d" % len(findings))
lines.append("")
if findings:
for name, problem in findings:
lines.append(" UNPROVEN %s" % name)
lines.append(" %s" % problem)
lines.append("")
lines.append("VERDICT: UNPROVEN CLOSURES. %d of %d closed pages carry a 'closed'"
% (len(findings), closed_count))
lines.append(" flag with nothing behind it. A bare flag is cheap talk")
lines.append(" (Crawford and Sobel 1982): costless to emit, so worthless")
lines.append(" to read. Name the proof and the replacement file, or the")
lines.append(" page is not retired, only untouched.")
else:
lines.append("VERDICT: PROVEN. Every closed page names its proof and a replacement")
lines.append(" that exists. The forgetting layer is honest here.")
return "\n".join(lines)
def run(directory, proof_fields, root):
try:
findings, closed_count, scanned = scan(directory, proof_fields, root)
except OSError as exc:
print("cannot read directory %s: %s" % (directory, exc), file=sys.stderr)
return 2
print(render(findings, closed_count, scanned, directory))
return 1 if findings else 0
# (frontmatter text, expected closed?, expected problem?). Each is a fact the
# inspector must get right. The replacement check is exercised in the on-disk
# worked example, where a real missing file can be posed.
SELF_TESTS = [
# a proven retirement: closed, with proof, no path to check
("---\nstage: retired\nretired_because: superseded by the hourly check\n---\n",
True, False),
# closed with no proof field at all
("---\nstage: retired\n---\nbody\n", True, True),
# status (not stage) carries the closing word
("---\nstatus: resolved\nlesson_carried_by: guard.py\n---\n", True, False),
("---\nstatus: done\n---\n", True, True),
# an open page is never a finding, proof or not
("---\nstage: in-force\n---\n", False, False),
# a page with no frontmatter is ignored
("# just a heading\nno frontmatter here\n", False, False),
# 'closed' as the first word of a longer value still counts
("---\nstage: closed 2026-10-29\nretired_because: proven by fixture\n---\n",
True, False),
# a proof field that is present but empty does not count as proof
("---\nstage: superseded\nretired_because:\n---\n", True, True),
]
def self_test():
import tempfile
passed = 0
for i, (text, exp_closed, exp_problem) in enumerate(SELF_TESTS):
fm = parse_frontmatter(text)
if fm is None:
got_closed, got_problem = False, False
else:
word = closed_state(fm)
got_closed = word is not None
if not got_closed:
got_problem = False
else:
present = [f for f in DEFAULT_PROOF_FIELDS if fm.get(f, "").strip()]
got_problem = not present # path check is covered on disk below
ok = (got_closed == exp_closed and got_problem == exp_problem)
passed += ok
print(" %-4s case %-2d closed=%s problem=%s (expected closed=%s problem=%s)"
% ("PASS" if ok else "FAIL", i, got_closed, got_problem,
exp_closed, exp_problem))
# on-disk: a closure pointing at a replacement that does not exist is a finding;
# the same closure pointing at a file that does exist is not.
with tempfile.TemporaryDirectory() as d:
present = os.path.join(d, "replacement.md")
with open(present, "w", encoding="utf-8") as fh:
fh.write("# the page that holds the lesson now\n")
good = os.path.join(d, "good.md")
with open(good, "w", encoding="utf-8") as fh:
fh.write("---\nstage: retired\nlesson_carried_by: replacement.md\n---\n")
bad = os.path.join(d, "bad.md")
with open(bad, "w", encoding="utf-8") as fh:
fh.write("---\nstage: retired\nlesson_carried_by: deleted-page.md\n---\n")
_, good_problem = inspect_page(good, DEFAULT_PROOF_FIELDS, d)
_, bad_problem = inspect_page(bad, DEFAULT_PROOF_FIELDS, d)
for label, got, want in (("existing replacement", good_problem is not None, False),
("missing replacement", bad_problem is not None, True)):
ok = got == want
passed += ok
print(" %-4s on-disk %-22s finding=%s (expected %s)"
% ("PASS" if ok else "FAIL", label, got, want))
total = len(SELF_TESTS) + 2
print()
print("SELF-TEST: %d/%d checks passed" % (passed, total))
return 0 if passed == total else 1
def main():
ap = argparse.ArgumentParser(
description="Find closed wiki pages retired without proof.")
ap.add_argument("directory", nargs="?", help="directory of *.md pages to scan")
ap.add_argument("--proof-field", action="append", dest="proof_fields",
help="a frontmatter field that counts as proof (repeatable; "
"default: retired_because, lesson_carried_by)")
ap.add_argument("--root", help="root to resolve replacement file paths against "
"(default: the scanned directory)")
ap.add_argument("--self-test", action="store_true",
help="run built-in checks and exit")
args = ap.parse_args()
if args.self_test:
return self_test()
if not args.directory:
ap.error("a directory is required unless --self-test is given")
proof_fields = tuple(args.proof_fields) if args.proof_fields else DEFAULT_PROOF_FIELDS
root = args.root if args.root else args.directory
return run(args.directory, proof_fields, root)
if __name__ == "__main__":
sys.exit(main())
Feed it a three-page wiki: one retired honestly, one stamped closed with no proof, one pointing at a deleted replacement.
RETIRE WITH PROOF: samples/wiki-3page
markdown pages scanned : 3
pages marked closed : 3
closures without proof : 2
UNPROVEN lp-0002-closed-no-proof.md
closed as 'retired' with no proof field (retired_because or lesson_carried_by)
UNPROVEN lp-0003-points-at-a-ghost.md
closed as 'superseded' but lesson_carried_by names a replacement that is not on disk: lp-0009-the-page-that-was-deleted.md
VERDICT: UNPROVEN CLOSURES. 2 of 3 closed pages carry a 'closed'
flag with nothing behind it. A bare flag is cheap talk
(Crawford and Sobel 1982): costless to emit, so worthless
to read. Name the proof and the replacement file, or the
page is not retired, only untouched.
Exit one. The honest page passes silently; the second is the purest cheap talk, stamped with nothing behind it; the third names a replacement that is gone, so the lesson lives nowhere. A clean wiki returns VERDICT PROVEN at exit zero, and --self-test runs ten planted cases, two on real files, all passing.
Founder offer
If an agent writes pages you later read back, run this against your wiki folder today: it counts how many of your closed pages are closed in name only, the first number most memory systems have never looked at.
Pro is $15 a month or $250 a year: the Brief, the paid flagship tool, the paid Feed payload, and the archive. Founder is $300 a year, capped at one hundred seats, with a founders-only MCP server built once the hundredth seat sells. Terms: nonxerosum.studio/mcp.
No pitch beyond that. The proof is the argument.
