The Dreaming Loop

I was using Claude Code at the time, with its MEMORY.md file carrying context between sessions. I remember watching an Anthropic video about a “dreaming loop”; the one I believe prompted this work is Memory and dreaming for self learning agents, on Claude’s YouTube channel. It gave me an idea to try in Rootweaver: could the system do something useful with the memories it had accumulated, rather than just keep adding to the file?

The index is only the entrance to the archive. Conceptual illustration.

The talk describes a separate process that reviews sessions and refines memories outside the active agent loop. Its dreaming section starts at 10:57. It is about Claude Managed Agents. I was adapting the idea to the local AUTO-memory setup I was using, with its own files, maintenance scripts and review step.

There was earlier research behind this. A comparison note dated 7 June already examined Anthropic’s memory and dreaming ideas alongside Rootweaver’s memory systems. Some of that analysis concerned the typed facts store, a different surface from the Claude Code files. This episode follows the local AUTO-memory implementation on 13–14 June, with that earlier research as context.

The previous episode followed outside material coming into Rootweaver. This was another example of that process: something I watched gave me an idea to try in the system. The work became a practical memory-maintenance routine over the same weekend. I was directing and reviewing agent-assisted implementation, working out which parts could run automatically and which changes still needed review.

By the weekend of 13–14 June, I had a memory problem that was easy to miss. The notes were still there. The index that helped the agent find them had grown beyond the limit the maintenance tooling expected, and the handover recorded that its tail was being truncated when loaded. I could keep adding useful context and still make less of it available to the next session.

The starting point above is my recollection of what prompted the work. The implementation that follows is reconstructed from the June handover, retained evidence ledger and the code committed that weekend. The measurements below are historical reports, with their limits preserved. They do not describe the current installed memory system.

The file existed. The whole index did not fit.

Rootweaver had several things called memory. The loop here targeted Claude Code’s AUTO-memory: a MEMORY.md index pointing to topic files in its project memory directory. It was separate from the typed facts in Postgres and their enrichment or consolidation paths. Sharing a name did not make those systems interchangeable.

The period handover recorded an index of 25,953 bytes against a 24,400-byte limit used by this tooling. That put it 1,553 bytes over. I am keeping those as the historical figures; they are not a statement of a universal limit across current agent versions. The failure was especially awkward because the files remained available on disk. Nothing about a successful file write guaranteed that the next session received the full index.

The maintenance effort began under VW-723 — Memory consolidation routine ("dreaming") — scheduled curation of auto-memory index + files. The objective was to recover useful context under a fixed loading budget. Shorter pointers could help. Throwing away the detail behind them would be a different change, with a different risk.

That distinction shaped the approach. The retained June ledger records an earlier candidate and a recall delta of +0.037, described as within noise. It also records the subsequent live index-size fix separately. The raw recall result is not retained in this review package, so I can report the historical assessment without treating it as a freshly verified improvement.

The engineering objective was to keep enough description for the agent to choose the right topic, preserve the topic files, and check retrieval after editing. Optimising only the byte count can produce a beautifully small index that no longer helps anyone find anything.

Freeze the before-state before trying to improve it

On 13 June, the first job was to preserve something we could compare against. VW-787 — Dreaming loop Phase 0 — baseline measurement harness (auto-memory) [VW-723] recorded an immutable snapshot of the existing topic files and index, with per-file hashes. It also froze a 40-prompt golden replay set: the same questions could be asked against the original memory and a proposed replacement.

That made the maintenance question testable. Once the live index changed, an unrecorded “before” would be gone. A snapshot kept the original available; fixed prompts gave the comparison a stable set of tasks. The baseline report planned repeated, order-swapped judging rather than treating a single model score as precise. It also acknowledged that the prompts skewed towards recent sessions, which limited how broadly the result could be interpreted.

The size measurement at this snapshot was 25,921 bytes. The later pre-apply figure was 25,953 bytes. Those are separate captures, not an unexplained before-and-after result. Both exceeded the recorded 24,400-byte limit.

The hygiene findings changed what a useful repair looked like. Deduplicating entries would not solve an index with no detected duplicate pairs at that threshold. Shortening a broken path would not repair its destination. The work needed to preserve useful pointers, make them fit and flag stale references, then use replay to check what the proposed changes did to recall.

Fix the index, then explain what the fix proves

The early-Sunday handover records the live index moving from 25,953 to 22,664 bytes. Seventy-six pointer lines were shortened, with zero memories dropped and zero content drift reported by that session. Those last two statements are the historical report’s assessment; this draft has not independently replayed its entire comparison.

Shorten the route to a record while keeping the record. A separate close-up composition for the consolidation beat.

The ledger also retains an earlier candidate at 23,718 bytes, with 682 bytes of headroom. That is a separate revision. Mixing its recall result, the later 22,664-byte index and an assumed improvement in task efficiency would manufacture a single successful experiment out of different observations.

Getting under the limit fixed a concrete maintenance problem. It did not prove that future sessions would use fewer tokens, make fewer tool calls or need fewer corrections. Those needed their own before-and-after measurement. The handover explicitly left that work open.

I find that separation useful. There was a result worth keeping: the reported live index fitted again. There was also a much larger question still waiting for data: did the maintenance make subsequent work better?

Recovering what earlier sessions had missed

There was another part to the loop: VW-793 — Dreaming loop Phase 2 — enrich auto-memory from JSONL archive (Claude/batched) [VW-723]. Compression worked on the index already present. Enrichment asked whether older sessions contained durable information that had never reached the topic files.

The historical scripts/dream_consolidate/mine_archive.py describes four stages. First, it formed bounded digests from typed user prompts and assistant prose, leaving tool-call and tool-result envelopes out of that digest. A small-model filter then shortlisted potentially useful material. A deterministic pooling pass collapsed near-duplicates and weighted candidates by recency and cross-session frequency. A separate batched review compared the shortlist with existing memory, rejected stale or superseded facts and prepared proposals. The script itself did not write those proposals into live memory.

The stale-fact check mattered because an archive is a record of what was said at the time. An old path, configuration choice or operating rule could have been correct then and wrong by the time it was reconsidered. Recovering more information was only useful if the review distinguished a durable lesson from an obsolete instruction.

The code also kept the evaluation window out of the enrichment input. It excluded 2–13 June, the dates used to form the golden replay set. Otherwise a later recall score could look better because the enrichment had harvested material from the very sessions supplying the test questions. Keeping those inputs apart made the intended comparison more meaningful; it did not by itself prove that the recovered facts improved later work.

That added a second question to memory maintenance. Had the existing index become too large, and had useful context been missed in the first place? The retained code establishes the proposed enrichment mechanism. I am not treating it as proof that every archived session was processed or that every recovered candidate was worth keeping.

A use count has to measure something selective

By early Sunday, the routine had acquired three more pieces under VW-806 — Dreaming loop Phase 3 — reinforce-on-recall + launchd schedule + advisory gate [VW-723]. The handover records four merged changes, including the earlier tooling, at commits d663e633, 77510150, f22fc756 and 4e6297be. The excerpts here are read from the last of those commits, rather than inferred from today’s working tree.

The first piece was reinforcement. Counting a memory every time the index loaded would tell me very little: every entry in the loaded index would receive credit whether it helped the task or not. The implementation instead looked for selective citation of a topic slug in the agent’s session output.

It also recomputed counts from the available corpus. That is more precise than describing the routine as incrementing counters. Re-running the same corpus should not keep adding credit for the same sessions. The count meant sessions in the current corpus that cited the slug, rather than an endlessly accumulating number of maintenance runs.

    for path in iter_session_files(source):
        n_sessions += 1
        cited, date = session_cited_slugs(path, slug_re)
        for slug in cited:
            use_count[slug] += 1
            if date and (slug not in last_used or date > last_used[slug]):
                last_used[slug] = date
    return use_count, last_used, n_sessions

The returned citation set matters. A session that mentions a slug repeatedly still contributes one session of use. last_used takes the latest matching date. The resulting metadata can help identify which proposed pointer edits deserve attention, but it only describes what this parser can see.

That limitation surfaced immediately. The handover gives the example of a critical rule that received a use count of one because the agent could follow its meaning without mentioning the filename. Slug matching was evidence of explicit reference. It was an incomplete measurement of practical importance.

Low counts therefore could not safely justify demotion. A quiet memory might be irrelevant, or it might be a foundational rule that rarely needed naming. Those cases look similar to a string matcher and have very different consequences if removed.

The advisory gate drew attention to the risky edits

The advisory gate used the positive evidence. If a proposed shortening affected a frequently cited memory, it appeared in a review list. The implementation removed the .md suffix to align the proposal’s filename with the counter’s key, compared the count with a threshold, and sorted the flagged items by use.

    flagged = []
    for p in proposals:
        stem = p["slug"][:-3] if p["slug"].endswith(".md") else p["slug"]
        uc = use_counts.get(stem, 0)
        if uc >= threshold:
            flagged.append({**p, "use_count": uc})
    flagged.sort(key=lambda p: -p["use_count"])
    return flagged

This function does not reject a proposal. It returns annotations for the report. It does not infer that everything below the threshold is safe to compress or delete. The code and its comments support that narrower description; calling it an automatic safety barrier would give it too much credit.

That was enough to make review more useful. A human could see where a small-looking pointer edit affected something cited often, while retaining responsibility for the meaning of the change. The automation gathered evidence and prepared the work. Approval remained a separate step.

Making it run outside the terminal

The next piece was a weekly Mac schedule. The handover records a Sunday launchd job, but getting the command to work interactively was only part of the job. The wrapper had to find a usable interpreter in the environment that would exist later, without an interactive shell session.

The recorded failure was mundane: the obvious environment interpreter resolved to a system Python path that errored in the non-interactive setup. The repair was more precise than simply hardcoding a new path. The historical wrapper tried candidates in order and accepted the first one that actually ran a small Python command.

PY=""
for _cand in /opt/homebrew/bin/python3 "$REPO/python-enhancement-env/bin/python" python3; do
    if "$_cand" -c 'import sys' >/dev/null 2>&1; then PY="$_cand"; break; fi
done
if [ -z "$PY" ]; then
    log "FATAL: no working python3 found — aborting"
    exit 1
fi
log "interpreter: $PY"

The first candidate was Homebrew Python, followed by an environment interpreter and then python3 on the path. If none worked, the wrapper logged a fatal error and stopped. The maintenance scripts did not require activating the platform’s full package environment, which kept this check small.

This is the sort of failure I want a scheduling test to catch before I rely on next week’s run. A file can exist and be executable without being the interpreter this job needs. The probe tested the relevant behaviour: could this candidate start Python and import its standard system module?

Five passes, with different consequences

The weekly wrapper ran a budget check, reinforcement, hygiene, proposal generation and notification. Reinforcement wrote citation metadata. Hygiene checked matters such as broken paths and index size. Consolidation wrote a proposal file. Applying that proposal was a separate manual action.

Direction shows the historical wrapper's intended sequence, not live activity. The separate apply step requires review; the advisory gate adds warnings to the proposal.

Two details in the code stop this from becoming an overly neat success story.

First, the budget check guarded the cost of generating review work when the weekly allowance was nearly exhausted. The wrapper’s comments explicitly distinguish that from its own computation: reinforcement and hygiene were Python work, while the consolidation pass could use a separate OpenAI call. This was not one meter accounting for every resource used by the job.

Second, missing or unreadable budget state became zero in the historical wrapper. That let processing continue. A missing measurement therefore did not mean the budget had actually been verified. This is a limit visible in the retained source, not a claim that it caused a particular incident that weekend.

The wrapper also continued after reinforcement or hygiene failed, and logged the failure. Even proposal-generation failure led to a final end-of-run message. The sequence provided useful logs and outputs, but successful process completion alone could not establish that every pass succeeded. A useful operational check had to inspect those pass results and the proposal itself.

A prepared proposal still needs someone to read it. Conceptual illustration of the review step.

These are small distinctions, but they change what I can honestly say shipped. The historical handover describes the routine as operational and records the index repair. The inspected wrapper supports the sequence and manual apply separation. Neither establishes indefinite weekly reliability, complete citation tracking or improved task efficiency.

A later baseline was still a starting point

The June 14 handover captured a later before-baseline over twenty trailing sessions. This was a separate capture from the June 13 snapshot and replay setup; the two changing session samples are not an improvement measurement. It recorded 984,406 fresh input tokens, 156.4 tool calls and 6.5 redundant retrieval calls in its session aggregates; a 14.5% redundant retrieval rate; a correction rate of 0.191; and memory utilisation of 0.314, or 50 of 159 slugs cited.

Those are the source’s historical measures. Without the retained raw baseline and an independently checked matched after-sample in this package, they cannot substantiate a savings percentage. The handover explicitly said the efficiency claim was instrumented and waiting for later data. That is the useful part to preserve.

It also recorded a trap in the measurement pipeline: the log sink appended rows without deduplicating them. There were 463 rows for 307 sessions. Counting rows as sessions would distort the comparison. A later analysis needed session identity for deduplication or the saved JSON snapshot aggregates, and it needed the correct time window because the rows carried session timestamps.

A weekly job can look healthy because it runs on schedule and writes something. This baseline was an attempt to ask a better question: after changing the index, did comparable work need less repeated searching or fewer corrections? The episode-period evidence leaves that answer open.

What I would keep from this approach

I would keep the separation between the index and the underlying notes. It gave the maintenance job a narrow repair to propose and gave review a concrete preservation question to answer. I would also keep the replay step, while being stricter about recording which revision each result belonged to. A candidate’s recall result and a later live index size should remain individually attributable.

I would improve the use signal before using it for anything more consequential. Explicit citation is useful positive evidence. It misses rules followed implicitly and concepts recalled without filenames. That uncertainty belongs in the decision, particularly when a proposed action would remove context.

I would make pass-level outcomes part of the routine’s visible result. The historical wrapper’s continuation behaviour was understandable for maintenance, but it put more responsibility on log inspection. A schedule, a final message and a proposal path each establish different things. They should not collapse into one green status.

The remaining problem was truth over time. Shortening a pointer cannot decide whether a belief is stale or contradicted by newer evidence. It can make an obsolete claim easier to load. Conflict detection and supersession need their own evidence and handling; this maintenance loop did not supply them.

By early Sunday, the recorded result was a working maintenance sequence, a smaller live index and a measurement baseline. That was useful progress. The stronger claim—that Rootweaver could improve its memory autonomously and measurably—still needed more than a successful tidy-up.

The next episode moves from memory maintenance to a different kind of restructure: how the platform’s services were organised in Kubernetes. The same question follows it there: after changing the structure, what evidence shows the system still does what I need?