The Self-Healing Turn

The June autonomy decision described an SRE service that collected alerts, gathered diagnostic context and produced a suggested explanation. The harder question was what that explanation was allowed to become.

I wanted help investigating incidents in a lab I treated as production. That did not mean giving a model permission to turn a plausible sentence into a production change. The previous chapter made alerts easier to route and act on. The season closes by making the boundary around that action more explicit.

The route can open incrementally while the human retains the lever that permits action. Conceptual artwork, not a historical photograph.

I owned the direction, acceptance and operational decisions. Agents assisted with implementation and review. This reconstruction follows the June 28 autonomy decision and the June 29 hardening review, distinguishing the design from the code paths I can inspect and the deployment outcomes reported at the time.

A ladder is a plan, not a capability badge

ADR-088 describes an incremental autonomy ladder. Its prose calls it six-rung, but its table labels R0 through R6. There are seven labelled levels, including the safety foundation. More importantly, those levels do not all describe shipped capability.

Level	Intended capability	Historical boundary
R0	Redaction, input limits, rate limiting and liveness	Safety work with separate tasks and checks; avoid treating every item as one completed block
R1	Typed action catalogue	A schema and action vocabulary, not an executor
R2	Model-assisted triage	Described as the current operating level; hypothesis and recommendation
R3	Read-only hypothesis verification	Planned safe next step with restricted reads
R4	Human-approved remediation	Future execution boundary
R5	Scoped supervised remediation	Future blast-radius, verification and rollback requirements
R6	Learning from outcomes	Future labelled evidence and calibration work

A free-text recommendation is useful for a person. It is a weak interface for an executor. If the model says to restart something, a separate component still needs to know which action is permitted, what object it targets, what evidence justifies it and who approved it. The action catalogue was intended to make that question typed before any execution path existed.

This was informed by other people’s research, not invented in isolation. Yinfang Chen and colleagues’ STRATUS paper formalises Transactional No-Regression as a safety specification for agentic reliability work. That is a useful design reference for checking whether mitigation makes a previously healthy condition worse; it is not a certification that Rootweaver implements that property. Saurabh Jha and colleagues’ ITBench also makes the need for measured incident tasks concrete. Neither paper’s results are Rootweaver’s benchmark score.

The safety work existed before the status caught up

The June 28 handover is a useful starting point because it does not claim a deployment. It records the autonomy decision, a security specification and work still waiting to run. An unrelated dirty checkout stood in the way of a clean implementation base. At that point, a written plan was the output.

By the June 29 reconciliation, the retained report says the first boundary-hardening work had already merged the previous day. Its tickets still appeared in Backlog and its implementation record was missing. Those observations describe different systems: code history said something had changed; the work tracker had not caught up. Neither alone described the operating result.

That mismatch led to a fresh review. The merged work’s description mentioned a third security finding, but its detailed verifier transcript had been lost from temporary storage. The useful response was to inspect the paths again, rather than reconstruct a reassuring story from a missing transcript.

The review also initially inspected the wrong checkout. The handover reports that the local main tree was more than five hundred commits behind the relevant remote history. Searching it appeared to show no redaction calls. That was evidence about an old source tree, not evidence that the merged fix did nothing. Selecting the correct historical source was part of the investigation.

Record or observation	What it could establish	What it could not establish
A gate specification	Intended checks and scope	That the code had run or merged
A merged change	Accepted source at a particular revision	Every live caller used the changed path
A Backlog ticket	The tracker’s recorded state	That no implementation existed
A missing verifier transcript	An evidence-retention gap	The contents or completeness of the lost review
A source search	What was present in the selected tree	The state of a different revision

I owned the acceptance and operational decisions. The historical records attribute implementation and review work to coding agents, including Codex work on the initial boundary and later collector/prompt changes. The corrective value came from checking the actual paths and reconciling the records, not treating an agent’s completion sentence as the final observation.

The useful boundary is an actual path

The design placed redaction at the end of context collection, before evidence reached the model. It also called for protected output before results were dispatched. That sounds complete when drawn as one arrow in and one arrow out.

The review found more than one route out.

The June 29 report describes an HTTP triage endpoint that could publish reasoning and result fields to Kafka without the same protection as the normal consumer path. An earlier review had not exhausted those routes. The follow-up then caught a further distinction: the engine redacted the successful model result, while the rule-based fallback did not arrive in the same protected state.

The fallback matters because it is precisely the route that runs when the preferred path is unavailable. A guard attached only to model success cannot protect output produced somewhere else. The correct question becomes which boundary every producer must cross before publishing.

One engine, two publication paths

Pause flowExpand diagram

Close expanded diagram

Consider a fictional incident in which a model call fails and a rule-based fallback produces a result. The strings below are invented test data with no credential value. This is a partial result shape for explanation, not a captured incident, complete API response or authorisation message.

Synthetic result before the HTTP handler's Kafka-publication boundary. Keys match the text fields copied by that handler.

{
  "root_cause_hypothesis": "Demo worker rejected password=EXAMPLE_ONLY",
  "recommended_action": "Inspect demo-worker logs",
  "reasoning": "Request contained password=EXAMPLE_ONLY",
  "affected_services": ["demo-worker"]
}

The engine’s normal model-success branch called its result redactor. In the retained engine source, the fallback branch instead called rule_based_triage, recorded metrics and returned the result. Protecting the successful model branch therefore did not establish that the fallback result was protected too.

That is why the caller mattered. The ordinary consumer and the HTTP endpoint could reach the same engine, then leave through different code. The first HTTP correction covered the retained reasoning trace. The follow-up review caught the result fields as another object requiring protection, especially when generated by the fallback.

The inspected historical HTTP handler copies the result with redacted text fields before handing it to the publisher. It also separately redacts the retained reasoning trace in that publish call. This is useful evidence of the fix’s structure. It does not prove that every possible secret format is recognised.

Exact fragment, dedented: sre-brain/sre_brain/health.py, triage HTTP handler, historical commit 0dd9e73b. The following publisher call also redacts the retained reasoning trace.

redacted_result = result.model_copy(
    update={
        "root_cause_hypothesis": redact(result.root_cause_hypothesis),
        "recommended_action": redact(result.recommended_action),
        "reasoning": redact(result.reasoning),
        "affected_services": [
            redact(service) for service in result.affected_services
        ],
    }
)

Explain this code line by line

1. model_copy makes a modified result rather than editing the original object in place.
2. update names the fields replaced in the outgoing copy.
3. The hypothesis passes through the redactor even if it came from a fallback.
4. The recommended action receives the same treatment; it is still a recommendation.
5. The explanation is protected as another output field, rather than assuming the recommendation contains all sensitive material.
6. Every affected-service string is also passed through the redactor.
7. The copy closes here. The surrounding publisher receives this copy and a separately redacted reasoning trace.

For the synthetic input, the corresponding outgoing text fields should look like this. I checked these substitutions against the retained pure redactor with no network or model call.

Synthetic expected Kafka-result copy after the historical regex redactor. This demonstrates one recognised password pattern, not all possible sensitive content.

{
  "root_cause_hypothesis": "Demo worker rejected password=[REDACTED]",
  "recommended_action": "Inspect demo-worker logs",
  "reasoning": "Request contained password=[REDACTED]",
  "affected_services": ["demo-worker"]
}

The source also preserves a narrower boundary than “the HTTP endpoint is safe.” The corrected publisher receives redacted_result and a separately redacted retained reasoning string. The handler later returns result.model_dump(), using the original result object. The new copy does not replace that original. This inspected change establishes protection for the named Kafka publication fields; it does not establish the same policy for every HTTP response or persistence path. I have not reproduced a live disclosure, and I would not infer that one occurred from this source distinction alone.

The publisher call is also conditional on a configured producer, and its failure is caught as non-blocking. A successful HTTP response therefore cannot, by itself, prove that the Kafka event was published. The mechanism has at least three outcomes to keep separate: an assessment was returned, a protected event was assembled, and a downstream delivery succeeded.

Context included more than the visible log lines

The collector was the input boundary: it assembled diagnostic information before the model prompt was constructed. The retained source applies redaction to ordinary logs, previous-container logs, extended logs and selected event fields. Those categories do not all have the same destination. Crash logs can also reach evidence storage even when they do not become part of the model prompt.

The follow-up found that recent changes and code-impact context needed attention too. A change author or a nested string may look harmless because the application calls it metadata. Once it is serialized into a prompt or retained evidence, it becomes another content path.

The historical collector change handles recent-change metadata selectively. It leaves commit identifiers and file lists visible and redacts the other direct string values. It uses a recursive helper for code-impact values. Those are different policies; the code does not demonstrate a universal walk over every field in every stored object.

Exact function: sre-brain/sre_brain/context/collector.py, historical commit 195b9c42. Any is a typing annotation; the function walks values, not dictionary keys.

def _redact_string_leaves(value: Any) -> Any:
    if isinstance(value, str):
        return redact(value)
    if isinstance(value, list):
        return [_redact_string_leaves(item) for item in value]
    if isinstance(value, dict):
        return {key: _redact_string_leaves(item) for key, item in value.items()}
    return value

Explain this code line by line

1. The helper accepts a value whose shape may be text, a list, a dictionary or something else.
2. The first condition identifies text at the current position.
3. That string passes through the existing redactor.
4. A list needs each item examined because its contents may contain more structures.
5. Calling the helper again preserves the list while processing those nested values.
6. A dictionary needs the same treatment for its values.
7. The keys remain unchanged; each value is processed recursively.
8. Numbers, booleans and other unsupported shapes pass through unchanged. A nested email-shaped string also remains unless the called redactor is configured to remove it.

For example, a fictional code-impact value can contain a nested note with password=EXAMPLE_ONLY and a numeric count of two. The helper replaces the recognised password value while retaining the count. That is useful because the diagnostic structure survives. It is also bounded: dictionary keys are not scrubbed, and the caller must actually apply the helper to that field.

The underlying regex redactor recognises several token/password patterns. Email redaction is explicitly opt-in. The collector’s default redact call does not enable that option. It would therefore be inaccurate to describe the implementation as removing all personal information or every possible secret. The tests and examples need to state which pattern they exercise.

A delimiter in metadata could cross the prompt boundary

Input size limits and evidence tags served a different purpose from secret redaction. The engine wrapped the collected context in an evidence block and told the model to treat evidence as data. The historical source includes limits on formatted context and individual log lines. Those limits constrain input volume; they are not proof that the remaining text cannot influence the model.

The review found that sanitising only the context blob left separate template fields exposed. Event type, severity, pod and namespace were supplied independently to the skill prompt. An unexpected value in one of those strings could contain the same closing evidence marker that the context protection was meant to neutralise.

The fix applied the same delimiter replacement to those metadata fields before prompt assembly. A synthetic label containing a closing evidence marker and an instruction-shaped sentence makes the issue easier to see: the marker is untrusted input too, even if it arrives under a field called pod.

Exact function: sre-brain/sre_brain/triage/engine.py, historical commit 195b9c42. Literal replacements protect the chosen delimiter syntax.

def _neutralize_evidence_delimiters(value: str) -> str:
    return value.replace("[/evidence]", "[ /evidence ]").replace(
        "[evidence]", "[ evidence ]"
    )

Explain this code line by line

1. The function takes one string before it is inserted into the prompt template.
2. A closing evidence tag is changed into spaced text, so it no longer matches that literal delimiter.
3. The opening tag receives the corresponding replacement.
4. The result is returned to the caller. The instruction-shaped words remain text; this is not a general detector or proof against all prompt injection.

A schema that rejects unexpected output fields is another useful constraint, but it does not independently verify the truth or safety of an allowed string. Input caps, delimiter handling, output shape and publication redaction address different failure mechanisms. Passing one check does not make the others redundant.

The review changed what complete meant

The sequence of reviews changed the meaning of complete. The first protected path was useful, but an alternate caller, a fallback producer and separate prompt fields each needed their own examination.

The June 28 handover initially recorded a specification waiting for implementation. The June 29 record then reconciled the merged security work and describes the additional findings and fixes. Historical Git source supports the particular output-copy and collector changes inspected here. The report’s merged and shipped statements remain historical reports; this review does not establish today’s cluster state or reconstruct every original adversarial test.

The earlier report records fifty-two affected tests passing for the HTTP-publication fix. The later handover describes two adversarial review passes and the additional fixes as merged. These are historical reported checks; the fresh examples in this review run only the retained pure text helpers. They do not rerun the original integration suite or reproduce its deployment.

There were residuals too. The handover left an embed-version assertion mismatch and optional follow-ups around raw pod-detail persistence and telemetry. The earlier report was written while changes were still open; the later handover records their merged state. I keep that progression rather than silently reading the earlier report as a final deployment receipt. A broad sentence that all safety work was finished would hide that record. I would rather describe the actual inspected paths and leave the remaining questions visible.

The system was gaining a better place to form a hypothesis, not a general licence to act on it. Protecting what leaves the investigation is part of that boundary.

Protect alternate output routes at the boundary rather than assuming one successful path covers them all. Conceptual artwork, not a historical photograph.

A season can close with work still open

This episode follows the work from June 28–29, then closes with June 30 to explain what was still unfinished and would carry into July. Reaching the end of the month did not mean every task was complete.

This is the right stopping point for June because the direction has changed. Earlier episodes were often about making a component run or discovering why a signal was misleading. Here the platform is beginning to say which kinds of help an agent can offer and which decisions remain separate.

The June 30 graph-ingestion handover describes fixes merged into source while the intended live backfill evidence was still pending. The first attempted backfill had failed against the deployed image, and the follow-up source fix had not yet reached that image. A successful merge and a graph populated by the corrected job were still different observations. That is a useful boundary into July, rather than a reason to describe June as a fully completed autonomy project. The next season can pick up that unfinished result without disguising it as a new problem that appeared overnight.

What I learned	What I would do next time
A safety diagram can hide alternate callers	Enumerate HTTP, consumer, fallback and publication paths
A typed recommendation is not execution	Keep catalogue, verifier and executor permissions separate
More review can find a different class of fault	Revisit the sink, not just the previously named finding
A signed or merged record is bounded evidence	Check the mechanism and the operational observation separately

Across this season, the repeated problem was reading a success signal too broadly. A build could pass before a route worked. An alert could arrive before its receiver or first action was useful. A trust check could run without stopping a release. Here, one redacted result could coexist with another output path still needing attention.

The season closes with a more practical test for agent assistance: name the evidence, follow the actual caller, state which boundary was checked and keep the decision to change production explicit. The historical autonomy ladder supplies a direction for later work. It does not let the finale claim that diagnosis has become verified cause, or that a recommendation has earned execution authority.

The human still holds the decision to change production. More useful assistance, more explicit boundaries, and work left to earn the next level is a concrete ending for June.

Next season’s proposed opener, The Job Was Green. The Graph Was Not., follows that ingestion thread into July, where a successful process exit was much less informative than it first looked. The July title and ordering remain editorial proposals, supported by the retained follow-up rather than a recovered Season 5 master map.

Terms and references

Glossary — plain-English definitions

Triage

An initial assessment of a problem, distinct from confirmed cause or repair.

Redaction

Removing or replacing recognised sensitive text; coverage depends on the patterns and paths inspected.

Fallback

An alternate response path used when the preferred method is unavailable.

Typed action

A recommendation expressed in a constrained vocabulary, not arbitrary shell text.

Transactional No-Regression

STRATUS safety specification for ensuring agentic reliability work does not regress previously healthy conditions.

Kafka publication boundary

The point where an event is handed to the Kafka publisher; a returned assessment does not prove downstream delivery.

String leaf

A text value at the end of a nested structure, such as a note inside a dictionary inside a list.

Prompt injection

Untrusted content attempting to influence an assistant beyond its intended role; delimiter replacement addresses one particular syntax.

Schema

Rules for an output shape and field types; they do not establish that the values are true.

No-regression check

A check that a proposed change has not broken an explicitly observed previously healthy condition.

Execution authority

Permission for a component to change a target system, separate from the ability to describe a possible action.

Primary reference: STRATUS by Yinfang Chen and colleagues.

How this episode was reconstructed

This episode focuses on the safety work from June 28–29 and closes with June 30 to show what carries into July. Generated daily journal summaries were checked for coverage but do not substantiate a complete account of this arc; the relevant dated session handovers, reports and inspected historical source provide the detail. Broader native-hook, roadmap and commercial comparisons in the old draft are omitted without their own source audit. Remaining rate/liveness tasks are not all asserted complete. No full June29–30 session replay or exhaustive month inventory; research references are design context, not Rootweaver benchmark scores. The linked primary sources beside the relevant passages provide upstream attribution. The private review ledger and retained evidence distinguish historical reports, inspected code and new examples; they are not public site assets.