The Gate Was Measuring the Wrong Question

Episode 4 ended with a deliberate pause. A graph repair could stand on its own without justifying another full backfill. Finding the earlier decision that helped explain that pause raised a different question: why was the platform not consistently consulting the engineering history it already had?

I wanted agents working on Rootweaver to look for relevant decisions before repeating an investigation. A memory tool could be available, indexed and useful in a direct test while an agent still reached first for a text search. Making the tool available had not established that it was being used at the right time.

The scale can work perfectly while counting the wrong thing. Conceptual illustration.

This episode covers July 6, 2026, continuing the memory-tool thread recorded alongside the graph work. It is a curated reconstruction from dated handovers, an implementation report and retained hook/report source. I owned direction and review; agents assisted with implementation and checking. Historical live checks remain historical reports. The new examples use synthetic messages, and the isolated checks do not read personal transcripts, emit telemetry or reproduce the original session.

A reminder needed a measurable contract

The proposed behaviour was narrow: when an engineering question asked about history or an earlier decision, consult memory before a broad search. A guard could notice the first search action, determine whether the current question was historical, and check whether a memory call had already happened. A once-per-turn block could prompt the agent to change its approach.

This was a workflow intervention, not a new retrieval algorithm. It did not establish that memory contained the answer, or that searching source was unnecessary. After retrieving a prior decision, the agent still needed to inspect the relevant source and consider whether circumstances had changed. The practical objective was to stop paying for avoidable rediscovery while preserving verification.

The records also contained a correction to the baseline. A figure quoted during the earlier session had described a narrow session-only share. The carried-forward three-month reconstruction put memory-tool use at roughly 4.8% of information-finding activity. That is a historical usage estimate with its own classification rules, not a quality score or a claim about current behaviour.

The guard recorded three outcomes: blocked, when it caught a history question starting with search; complied, when memory had been used first; and bypass, when search was repeated after a block. Each sounds simple until the unit is specified. A tool event is not a whole user turn, and several events can belong to one question. Counting them interchangeably can produce convincing but misleading percentages.

Unit	Useful question	What it cannot answer alone
Tool event	What action did the harness observe?	How many distinct human questions were involved
First decision in a turn	Which behaviour defined the comparison arm?	Whether the answer was useful
Whole turn	What search work happened around the question?	What would have happened under the alternative workflow
Reviewed task result	Did the work meet its acceptance criteria?	Whether the same effect repeats across tasks

The important next step was to instrument the intervention rather than assume it helped. That instrumentation exposed a more basic problem: the guard was not reliably identifying the question it was supposed to govern.

The last user-role message was not always mine

The first targeting rule looked for the latest transcript entry marked as a user message. In an interactive agent system, that label can carry more than a person typing. The harness may inject skill instructions, dispatch work to another agent context, or add a metadata caveat using the same broad message role.

Those injected bodies can contain the very words a history detector recognises. A skill might discuss earlier decisions as an example. A dispatch prompt might name an architecture decision record. If the guard treated that content as my current request, an ordinary task could suddenly look like an instruction to reconstruct history.

The July report says five of seven initial blocks belonged to injected or delegated text rather than my questions. That is a tiny and contaminated sample, but it was enough to invalidate an optimistic reading of the first dashboard. A low compliance figure might describe a targeting bug rather than an agent refusing to use memory.

Listening to the system's own echo does not establish who asked the question. Conceptual illustration bound to the prompt-identity problem.

The repair used structural fields instead of maintaining a list of skill-title prefixes. The retained hook skips entries marked isSidechain, entries with sourceToolUseID, and entries marked isMeta. In that historical transcript format, these fields identified the machine-injected cases the investigation had found.

guard.py · _last_user_prompt_and_offset · claude-config 0b12448

if obj.get("isSidechain") or obj.get("sourceToolUseID") or obj.get("isMeta"):
    continue

Explain this code line by line

• obj is one decoded transcript record, not an independently authenticated person.
• The condition checks three fields that identified injected content in the observed harness format.
• If any field is truthy, continue skips this record and the backward scan keeps looking for an earlier prompt.
• Other checks in the full function still require a user-role message with nonempty text. The two lines do not classify the question as historical by themselves.

The complete function still checks message role, extracts text from a string or text blocks, and returns the position of the chosen entry. That position matters: the later memory-use check looks only after the selected prompt. Selecting the wrong boundary can change both the classification of the question and the evidence considered to have happened within its turn.

This is an implementation contract for that harness format, not a universal identity test. Different clients can encode messages differently. Even within one client, the absence of an injection flag is evidence about the record, not cryptographic proof that a human wrote it. The useful correction is to identify the observed structural distinction and test it, rather than infer authorship from the word user alone.

Follow one harmless request through the guard

Consider two synthetic transcript entries. The first is an ordinary request. The second represents a skill body injected after it. The example uses compact JSON because the inspected hook has a fast substring check that depends on that representation.

{"type":"user","message":{"role":"user","content":"Please inspect the image configuration."}}
{"type":"user","sourceToolUseID":"example-skill-call","isMeta":true,"message":{"role":"user","content":"Example: why did we choose this database?"}}

The second line is newer and contains a history-shaped question. A latest-user-entry rule would choose it. The repaired selector sees the injection fields, skips that line, and returns the earlier request. The guard can then evaluate the actual request instead of the skill’s example.

Now change the earlier request to “Why did we choose this database?” and leave the injected entry unchanged. The selector should still return the earlier line, but the history condition should now be meaningful. Avoiding a false block must not disable the legitimate memory-first reminder.

Follow the mechanism

Pause flowExpand diagram

Close expanded diagram

The diagram separates prompt selection from the later behaviour decision. It does not show a running security boundary or live telemetry. Its central point is that the classification cannot be better than the record selected as its input.

The retained tests reflect both sides. They check that skill bodies, sidechain dispatches and metadata caveats are skipped, while a genuine historical question still reaches the blocking path. The historical handover reports six new tests and 34 passing hook tests overall, plus a before/after live check of the same harmless search. The fresh checks for this draft isolate the selector and use supplied records; they do not repeat the original harness run.

A small test-fixture detail mattered. Python’s normal JSON encoder inserts spaces after separators. The hook’s quick check searched for the compact token "type":"user". A fixture written as "type": "user" could be skipped before the meaningful branch was exercised. A test that only expected “no prompt selected” might then pass for the wrong reason. Matching the actual wire shape was part of testing the mechanism, not cosmetic formatting.

The metric was late, not necessarily broken

Prompt targeting was not the only misleading signal. The telemetry-hardening report began by investigating why VictoriaLogs contained decisions while VictoriaMetrics appeared to have no corresponding counter. Its reconstruction says the metrics-push code had only been added shortly before the query. The query and code-change window overlapped before the first real sample.

That distinction changes the investigation. An event recorded before a sink existed cannot acquire a historical twin simply because a dashboard later expects one. The report identifies five early log events without metrics counterparts and says later correspondence was checked separately. I have not recovered the complete raw sink bundle here, so those timings and counts remain reported observations.

Two streams need a common observation window before their counts can be compared. Conceptual artwork; the clocks are not measurements of the historical run.

The hook deliberately swallowed telemetry exceptions so a monitoring failure would not stop ordinary tool use. That protected one failure boundary but reduced observability at another. Silence could mean no event, a failed push, or a sink that had not started receiving data. The historical diagnosis used a controlled push with captured HTTP status and a query-back check to separate those cases.

The alerting plan also changed after checking the installed stack. The earlier suggestion named vmalert; the retained report says that component was absent. The implemented path used Prometheus rules and existing hook metrics through a node-exporter textfile collector. That is a dated topology finding. It does not authorise installing or repairing anything in today’s cluster.

The informational compliance rule used a seven-day ratio below 25%, with a floor of at least ten decisions. The volume floor avoided treating one or two events as a stable pattern. It did not repair misclassified inputs. A perfectly evaluated rule over polluted events would still produce the wrong operational interpretation.

Fair windows before a cost comparison

The reporting change measured search activity over the whole user turn rather than starting at the guard’s decision timestamp. That choice was necessary because a compliant memory call could happen before the decision was logged. Starting the accounting clock at the decision would omit part of the memory-first arm and make it look artificially cheap.

retrieval-report.py · _episode_cost · claude-config 0b12448

start = max((t for t in stamps if t <= dec_ts), default=None)
if start is None:
    return None
end = min((t for t in stamps if t > dec_ts), default=start + EPISODE_CAP_S)
end = min(end, start + EPISODE_CAP_S)

Explain this code line by line

• The first expression finds the latest collected prompt timestamp at or before the guard decision.
• If no start exists, the function reports no usable cost measurement. It does not invent a zero.
• The next expression chooses the next prompt timestamp, or a capped end if none follows.
• The final line ensures that the window never extends beyond the 900-second cap from its start.
• The source collects these timestamps with its own transcript filter. That limitation is separate from the repaired guard selector.

The inspected report caps the turn window at 900 seconds. It counts search calls and approximates result context using raw result-line lengths. That is useful instrumentation, but it is not a tokenizer, billing meter or causal experiment. JSON wrappers and encoding can affect line length, and a byte estimate does not establish how much useful information reached the final answer.

There is another limitation in the historical source. The report’s turn-boundary pass uses its own fast checks for user-shaped entries; it is not simply the repaired prompt selector reused everywhere. Fixing the guard therefore does not independently prove that every downstream cost window was immune to injected records. That would need its own targeted test and review. The chapter keeps that boundary visible rather than promoting the whole telemetry chain because one helper improved.

The day-one report gave roughly 134 KB and six search calls per grep-first turn, with no complied turns available for comparison. Those figures were also flagged as polluted by the targeting bug. There is no defensible percentage saving to compute from a treatment arm with no observations. Nor does a later increase in memory use, by itself, establish better engineering outcomes.

What I would carry forward

I would define the unit before choosing a metric. “The agent used memory” can refer to an event, a turn or a completed task. Each is useful for a different question. A dashboard should not quietly move between them.

I would test the origin of the input as carefully as the final decision. The same text can be an instruction, a quoted example or an injected skill body. Matching words without inspecting their structural role can produce precise enforcement of the wrong request.

I would give comparisons symmetric boundaries. If one approach spends effort before the event that labels it, that work still belongs in the accounting window. Raw tool-result size can help locate waste, but cannot stand in for tokens, cost or quality without further measurement.

I would also keep the follow-up question open. The historical record booked later clean-data and task-quality reviews. It did not establish a sustained improvement that evening. The repair made subsequent measurement more credible; it was not the measurement’s final answer.

Next in the sequence

The next July thread asks a related question about the instructions themselves. A reusable skill can be internally consistent and still contain dated advice, or be well sourced without improving a real task. The July 7 records follow a small skills pilot through those separate checks. That is the next episode’s scope, not a promise that a universal productivity effect was proved.

Glossary — plain-English definitions

User role

A message category; it does not alone prove a human typed the content. Episode evidence and limits (retained historical evidence).

Injection metadata

Structural fields indicating harness-supplied content in the historical format. Episode evidence and limits (retained historical evidence).

First-touch decision

One initial behaviour label for a turn rather than every tool event. Episode evidence and limits (retained historical evidence).

Observation window

The interval over which related events are compared. Episode evidence and limits (retained historical evidence).

Counter

A cumulative measurement whose start and reset behaviour matter. Episode evidence and limits (retained historical evidence).

Conversion

Here, a later memory call after a block within the stated window. Episode evidence and limits (retained historical evidence).

Causal effect

A difference attributable to the intervention; an observational count alone does not establish it. Episode evidence and limits (retained historical evidence).

Sources and limits

The source and claim ledger (retained in review ledger) retains the original handovers and telemetry report unchanged, together with the historical hook, tests and reporting script. Private prompt ledgers and native conversation bodies are excluded. Historical sink correspondence, live gate behaviour and the original usage baseline have not been independently replayed. This selection covers the memory-first intervention, not every activity on July 6.