Skip to content

The Routing Fix Did Not Rescue the Experiment

S5·E13 ·Rootweaver · Experimentation · Retrieval · SoftwareEngineering

The previous comparison changed the interpretation of an attractive retrieval probe. Another experiment then needed a more basic inspection. Its routed and default requests were supposed to support a decision about domain-aware retrieval, but the routed endpoint was using a different implementation path.

I needed the experiment to exercise the behaviour its labels described. The July work repaired that path, checked how filters reached the agents and reran the reported comparison. The implementation improved. The remaining evidence still supported retiring the current hard-slice rollout, with a much narrower conclusion than declaring all domain-aware retrieval ineffective.

The engineer compares a narrow fenced route with the whole orchard, noticing needed trees outside the fence.

A selected part of a corpus can exclude evidence needed by the task. Conceptual illustration.

This chapter covers July 15, 2026. It is a curated reconstruction from the signed retirement handover and historical endpoint, agent and evaluation source at f4ef4410. I owned direction and review, with agent-assisted implementation and analysis. Scores, confidence intervals and deployment checks remain attributed historical results. New inspection follows the retained code. It does not rerun the held-out evaluation, invoke a model or independently re-score the answers.

The endpoint had bypassed the common pipeline

The domain endpoint was intended to select a route and corpus filter. The handover says the earlier implementation called the retriever directly, bypassing the shared agent path used by ordinary search. A comparison could therefore mix the routing idea with unrelated differences in how the two paths assembled results.

The repair dispatched routed requests through the common agent-search function. It carried both the selected filter and the domain persona into that call.

domain_endpoints.py · routed single-filter dispatch · platform f4ef4410 · exact historical excerpt

agent_response = await agent_search(
    request,
    trace_id,
    start_time,
    filters=plan.filters[0],
    domain_persona=plan.persona,
)
Explain this code line by line
  • The endpoint awaits the shared agent-search path.
  • The request, trace identifier and start time remain available to that path.
  • plan.filters[0] supplies the selected corpus constraint for this branch.
  • plan.persona supplies a separate answer-generation instruction.
  • The call establishes what this boundary sends. Each receiving agent still has to implement the relevant argument.

A union branch handled several selected filters, collected their result lists and merged them. The source shows that this branch also passed filter and persona arguments. It combined particular planned slices. It did not automatically unite a domain slice with the complete unfiltered corpus.

That difference affected the proposed successor experiment. A fallback formed from two narrow slices still excludes documents outside both. A future whole-corpus-union ablation would require a genuinely different eligible set and another explicit acceptance plan.

A keyword can arrive and still lose its meaning

The historical account records a defect in the Oracle agent. It could accept extra keywords while failing to apply the supplied filter. A routed response could therefore report routing activity without establishing that the intended corpus restriction had reached retrieval.

The inspected repair merges the explicit request filter with the agent’s ordinary source constraints, then passes the merged value onward.

oracle_agent.py · search_async filter merge · platform f4ef4410 · exact historical excerpt

source_filters = _merge_payload_filters(
    build_source_filter(
        source=source, sources=sources, meta_status=meta_status
    ),
    filters,
)
Explain this code line by line
  • The agent constructs its usual source and status constraints.
  • The separately supplied filters argument enters the merge.
  • The resulting value is stored as source_filters.
  • The subsequent inspected _async_search call receives filters=source_filters.
  • The merge preserves both sets of intended constraints according to the helper’s contract.

Following the receiving code mattered more than finding the word filters in an endpoint signature. A permissive **kwargs can make a call syntactically acceptable while discarding its intended semantics.

The engineer follows two hoses through a shared greenhouse filter box to their receiving beds.

A request constraint needs to reach and affect the implementation that uses it.

The Code agent still had a different contract

The retained Code agent source adds an important qualification to the handover’s broad repair summary. Its entry point accepts a filter argument, but the shown branch warns that it cannot apply that constraint.

code_agent.py · search_async filter warning · platform f4ef4410 · exact historical excerpt

if filters:
    logger.warning(
        "CodeAgent received payload filters but cannot apply them "
        "(no-op for code generation): filters=%s",
        filters,
    )
Explain this code line by line
  • A supplied filter enters the warning branch.
  • The message explicitly describes a no-op for this agent’s code-generation path.
  • Logging the value makes the unsupported request visible to an operator.
  • This excerpt supplies no filtering implementation.
  • A uniform parameter name across agents therefore does not establish a uniform corpus-selection guarantee.

The agent’s shown vault-search call also omits a filter argument. That is a source-level limit on claiming every routed branch was hard-sliced. The observed route flag identifies control flow. It cannot, by itself, certify the eligible document set used by every receiving implementation.

This qualification does not remove the useful endpoint and Oracle repairs. It keeps their scope inspectable. The next experiment would need a per-agent contract that either demonstrates the intended filter semantics or clearly identifies an unsupported branch.

The two arms still changed several things

The evaluation source makes another boundary explicit. The default answer used the base persona. The routed answer used the persona returned by the routed response, with the base persona as fallback.

eval_live_ab.py · routed answer instruction selection · platform f4ef4410 · exact historical excerpt

routed_system = (
    routed_resp.get("persona_system_prompt") or BASE_PERSONA.system_prompt
)
Explain this code line by line
  • The harness first asks the routed response for its persona instruction.
  • If that value is absent or empty, it uses the base persona.
  • The inspected default-answer call always uses the base persona.
  • Retrieval inputs and answer-generation instructions can therefore differ between the arms.

The comparison still measured a bundle of routing, corpus selection and persona behaviour. Repairing the pipeline removed an implementation difference, but it did not turn the experiment into a single-factor test of slicing.

FactorDefault armRouted arm
RouteDefault search endpointDomain endpoint and selected agent
Corpus constraintDefault eligible corpusPlanned filter, subject to receiving-agent support
Answer instructionBase personaReturned domain persona or base fallback
Judge contextThe question’s source truthThe same question’s source truth

The shared judge context helped make the comparison meaningful. It did not isolate which of the changed treatment factors caused a difference. A factorial or otherwise controlled follow-up would be needed for that attribution. NIST experimental-design reference

Follow the mechanism

RequestChoose routed pathPipelineShared agent flowFiltersReach each callerExperimentIdentify changed factorsDecisionApply rollout criterion
RequestChoose routed pathPipelineShared agent flowFiltersReach each callerExperimentIdentify changed factorsDecisionApply rollout criterion
Simplified editorial model of the inspected mechanism. Dots show direction.

Repeated judging did not create more questions

The historical handover reports forty held-out questions, ten per domain, with three judge passes over the same generated answer pairs. Repeated judgments helped inspect judge variation. They did not create 120 independent retrieval questions.

The saved harness also made its acceptance calculation inspectable.

eval_live_ab.py · verdict calculation · platform f4ef4410 · exact historical excerpt

verdict = (
    all(v["routed_ge_default"] for v in aggregate.values()) and max_std <= 0.05
)
Explain this code line by line
  • Each domain aggregate contains a point-estimate comparison between routed and default means.
  • all requires every stored comparison to satisfy that condition.
  • max_std is the largest recorded run-to-run standard deviation across judge passes.
  • The final Boolean also requires that value to be at most 0.05.
  • The expression does not calculate a confidence interval for the treatment effect.

The source and the handover therefore support different observations about uncertainty. A stable judge can repeatedly score an unrepresentative set of questions. A failed rollout condition can justify withholding a feature even when the sample is too small to prove a general performance difference.

The report’s statistical correction makes that limit concrete.

Domain exampleReported routed-minus-default differenceReported 95% interval
Architecture+0.042−0.272 to +0.356
SRE−0.163−0.583 to +0.257

Both intervals span zero and a broad range of plausible effects under the stated analysis. The source record supplied these values. I have not reconstructed the original answer pairs or rerun its statistical procedure.

The useful decision was about readiness of this rollout. The stronger conclusion that all domain-aware retrieval had been disproved would require evidence the experiment did not supply.

The slice boundary excluded relevant evidence

The handover records another finding that changed the interpretation. Five of the ten SRE questions had ground-truth documents outside the configured SRE paths. A hard filter based on those paths could remove material the question actually needed.

That is an inspectable design mismatch, even though the complete raw benchmark is not retained in this reading package. It belongs to this particular slice definition. A different definition, a soft ranking feature or a full-corpus union would be another treatment.

There is also a subtle top-k point worth preserving. Filtering can remove a high-ranked out-of-slice document and promote an in-slice document that previously fell below the cutoff. It therefore changes both eligibility and the visible ranking. A general claim that filtering can only reduce recall would skip that selection effect.

Consider a synthetic top-three list. An outside-slice document ranks first, two inside-slice documents rank second and third, and another inside-slice document ranks fourth. Applying the slice before taking three results can promote that fourth document. Whether this helps depends on which documents the question needs.

{
  "unfiltered_top_three": ["outside-A", "inside-B", "inside-C"],
  "next_unfiltered_candidate": "inside-D",
  "filtered_top_three": ["inside-B", "inside-C", "inside-D"],
  "quality_outcome": "requires the question's evidence needs"
}

The example explains a ranking property. It does not reproduce a held-out question or establish that this happened in every historical run.

The engineer separates three unmarked treatment cards while comparing two baskets in an orchard packing shelter.

A comparison needs an account of each factor that changed before a result can be attributed to one of them.

A transient field did not become a durable report

The source also clarifies the handover’s concern about missing routed-domain detail. The harness temporarily stored routed_domain in its in-memory answered rows. That looked promising at first inspection.

The later loop passed the expected domain, question, answer pair and truth context to the judge. Its output was appended to a comparisons list. The final returned report contained those comparisons and aggregate routing counters, while omitting the original answered rows.

The transient field therefore did not establish a saved per-question actual-route record. The aggregate count of matching domains could not reconstruct which judged question had been routed correctly. That prevented a reliable correctly-routed subgroup analysis from being recovered from the saved summary alone.

This is a data-lineage failure at a small boundary. Collecting a value in memory and persisting the value needed by the next analysis are different operations. A useful experiment record has to carry the treatment actually applied to the same unit whose outcome is being judged.

Retire the treatment, retain the useful repair

The closeout records the live comparison failing and the routing flag rolled back. It also records a recommendation to retire the current hard-sliced experiment. Removing the duplicate bypass path and making filter handling inspectable remained useful implementation work.

A follow-up proposal retained the full-corpus baseline, optionally added a high-confidence domain branch, deduplicated the union and applied one shared reranker. It also proposed testing persona effects separately. Those were conditions for reopening the investigation, rather than a deployed successor demonstrated by this episode.

The report’s statistical and implementation corrections narrow the conclusion further. The experiment changed several factors. A receiving agent still had an explicit filter no-op. The sample was small, some slice boundaries excluded required evidence, and per-question route persistence was incomplete. Those limits belong beside the operational decision.

What I would carry into the next build

Follow the real endpoint and every receiving argument before interpreting an experiment label. An accepted keyword, a route flag and a warning message establish different things. The implementation contract needs to state what each agent can actually enforce.

Record the treatment at the same granularity as the judged outcome. Per-question actual route, effective filter, selected persona and retrieval identifiers are more useful for later attribution than aggregate route counts alone.

Separate implementation repair from experimental success. A better shared pipeline can be worth keeping even when the feature it was built to evaluate does not satisfy its rollout gate.

Preserve uncertainty in the decision record. A conservative rollout decision can be justified without turning a small comparison into a broad scientific verdict. The next records move to a scoring job whose manual debug run and scheduled-controller state answered different operational questions. That returns the same discipline to an alert and its actual source of truth.

Glossary and primary references

Corpus

The body of documents eligible for retrieval.

Primary reference

Top-k

The first k ranked candidates returned from an eligible set.

Primary reference

Ablation

A controlled comparison that changes a defined part of a system.

Primary reference

Confounding

A comparison in which multiple changes make attribution ambiguous.

Primary reference

Persona

An answer-generation instruction associated with a role or domain.

Historical source excerpt in this episode

Rollout gate

A prespecified acceptance rule for deciding whether to deploy a change.

Primary reference