The Route Had to Reach the Model

Episode 2 ended with a useful mismatch. The smaller local model had a larger configured context window, but the application still carried routing assumptions from the previous deployment. More room on the graphics card did not mean the requests that needed it would arrive there.

I wanted the platform to use the capacity I had already made available. That meant following a request beyond the model endpoint: which agent would receive it, whether another branch could override that choice, what happened when retrieval failed, and what the next identical request would see. Changing a threshold was only one part of that work.

The route has to connect to the capacity it promises. Conceptual illustration.

This chapter covers July 2–3, 2026. I owned the direction, requirements, review and acceptance decisions; coding agents assisted with diagnosis, implementation and checking. The dated handovers describe the original runtime observations. For this reconstruction I separately inspected the historical router, Oracle, cache, gateway metrics and deployment configuration. The new checks use isolated code and synthetic inputs. They do not recreate the original traffic or repeat the model benchmark.

The model was one stop along the request

A request to my search gateway did not go straight to a language model. In automatic mode, a router first estimated the query’s complexity. It selected an agent, and that agent performed its own retrieval and processing. Oracle was the synthesis lane: it gathered evidence and asked a model to produce an answer from it. Deep followed a different retrieval path. A fan-out branch could invoke parallel searches for medium-complexity requests.

These are application roles, not separate human team members. The same underlying model service could participate in more than one role. A working generation endpoint therefore proved much less than a working search request. The gateway still needed a valid model name, the agent needed its dependencies, and the router needed to send the right work to that agent.

The overnight handover reports that Oracle had been effectively unavailable behind an old gateway image, dead model defaults and an inactive Ollama path. It then records a newer image, a working local model route and an observed synthesis. That is a historical account of the repair. The retained source lets me inspect the contracts involved, but cannot independently reproduce the old deployment state.

There was a second visibility problem. A broken classifier had fallen back to keyword heuristics that rarely assigned the scores which selected Oracle. A lane could be present and useful in direct tests while receiving very little ordinary traffic. The handover’s traffic reconstruction put fast search at roughly 69% of about 134 searches since mid-June, with Oracle’s small share largely associated with testing that day. That small, historical sample is context for the investigation, not a general demand forecast.

I needed to distinguish three questions: could the model answer, could Oracle assemble a useful answer, and did automatic routing actually choose it? Each needs evidence from a different boundary.

Two thresholds, doing different jobs

The previous episode’s context problem concerned the overflow threshold inside Oracle. Once retrieval had assembled a prompt, the agent estimated its token count. Inputs above a configured limit could be sent to a hosted model, provided the required credential was available. This is separate from the complexity threshold which chooses Oracle in the first place.

The inspected July 3 deployment configured these settings. The table names their different jobs so that changing one does not imply the others changed with it.

Setting	Historical value	What it controlled
ORACLE_OVERFLOW_THRESHOLD_TOKENS	100,000	Estimated full-prompt input above which the hosted route could be selected
ORACLE_COMPLEXITY_THRESHOLD	5	Complexity score at which normal selection entered Oracle
ENABLE_FAN_OUT	false	Whether the async medium-band override could run
EMBED_TIMEOUT	300 seconds	How long the embedding client could wait for the slower retrieval work

The source for the first decision is OracleAgent in oracle_agent.py at historical platform revision 9dbd25a1. This is an exact excerpt from its synthesis path:

route_to_haiku = (
    self._overflow_enabled
    and estimated_input_tokens > self._overflow_threshold_tokens
)
selected_model = (
    self._haiku_model if route_to_haiku else self._resolve_local_model()
)

Explain this code line by line

• route_to_haiku is the decision the agent will use for this synthesis request.
• _overflow_enabled must be true. In this implementation that depends on a configured API key; a threshold alone does not enable the route.
• The input estimate must be strictly greater than the configured threshold. An estimate equal to 100,000 does not take this branch.
• selected_model records the hosted model when that condition holds. Otherwise it resolves the local model name.
• The code uses an estimate of the full assembled prompt, not simply the number of tokens in the user’s question. Later code makes the selected call and records its outcome.

Oracle constructed that prompt from the query, retrieved context and its instructions before counting it. The local branch requested at most 1,024 output tokens in the inspected source. A 100,000-token estimated input threshold and the earlier 114,688-token serving ceiling therefore had different meanings. The gap left room for output and estimation error; it did not establish that every long prompt would be accurate or fit under every tokenizer.

The overnight record says the missing hosted-route credential was added to the gateway namespace, and its final configuration includes the 100,000-token threshold. That supports a reported configuration change and route availability. It does not supply a new, long-input quality benchmark. Nor does it demonstrate that a hosted request was made for every overflow case. The code and configuration explain when it could happen.

The distinction matters practically: increasing local capacity can reduce unnecessary fallback, but only after the application policy changes. Widening that policy still needs task tests. The previous episode’s planted-detail probes were useful evidence about those probes, not permission to treat all large requests as solved.

The selection rule had another rule after it

The complexity threshold lived in UnifiedAgentRouter. Its default remained seven, preserving the old selection behaviour unless deployment configuration overrode it. The helper parsed the environment value, fell back to the default for invalid text and clamped valid integers between four and eight.

That clamp protected the surrounding bands. Scores below four belonged to fast search; nine and ten had earlier specialist branches. Allowing an Oracle threshold outside that range would collide with those assumptions or leave Oracle unreachable through ordinary scoring. This was a bounded configuration mechanism, not a learned optimisation policy.

The important part of _select_agent is below. This is an exact excerpt at 9dbd25a1; a preceding branch handles graph-specific sentinel queries separately.

if complexity >= 10:
    return AgentType.WORK_UNTIL_COMPLETE, "Autonomous multi-step task"
elif complexity == 9:
    return AgentType.CODE, "Code generation/debugging task"
elif complexity >= _oracle_complexity_threshold():
    return AgentType.ORACLE, f"Deep research (complexity={complexity})"
elif complexity >= 4:
    return AgentType.DEEP, f"Semantic analysis (complexity={complexity})"
else:
    return AgentType.FAST, f"Simple lookup (complexity={complexity})"

Explain this code line by line

• The first branch reserves scores ten and above for the multi-step agent. Its name describes a route, not proof that a task completes.
• The next branch reserves score nine for the code agent.
• The helper supplies the current Oracle threshold. At five, scores five through eight reach this branch before the lower bands are considered.
• Score four still reaches Deep because it is below five but at least four.
• Lower ordinary scores select fast search.
• The returned pair contains the selected agent and a readable reason. The Oracle reason still says “Deep research”; the text alone is an unsafe way to identify the actual selected agent.

Reading this function alone would suggest that setting the threshold to five was enough. The async method did something else immediately after selection. When fan-out was enabled and the score was between four and six, it invoked the fan-out path and returned that result before the ordinary agent lookup.

This exact excerpt comes from search_async in the same historical file:

fan_out_enabled = os.getenv("ENABLE_FAN_OUT", "true").lower() == "true"
if fan_out_enabled and 4 <= complexity <= 6:
    logger.info(
        f"Fan-out search for complexity={complexity}: {query[:80]}"
    )
    result = await self.fan_out_search_async(
        query, n_results=n_results, verbose=verbose, **kwargs
    )
    routing_time = time.time() - start_time
    self._update_stats(agent_type, complexity, routing_time)
    return result

Explain this code line by line

• The environment value enables fan-out only when its lowercased text is true; the default is enabled.
• The score must also be within the inclusive four-to-six band.
• The logger records entry into this branch. That is evidence of selection, not of a useful answer.
• await waits for the fan-out operation and captures its result.
• The method records elapsed time and updates statistics using the previously selected agent variable.
• return result exits here. The normal lookup of the selected agent below this block does not execute.

That early return explains why the repair needed both settings. The threshold selected Oracle for five and six; disabling fan-out allowed that selection to reach Oracle’s ordinary call path. A configuration review limited to the threshold would have missed the competing control.

Ordinary score	Default threshold 7, fan-out enabled	Threshold 5, fan-out disabled
0–3	Fast	Fast
4	Fan-out overrides Deep	Deep
5–6	Fan-out overrides Deep	Oracle
7–8	Oracle	Oracle
9	Code	Code
10	Multi-step selection	Multi-step selection

This table deliberately corrects the early ADR’s broader wording about moving the whole four-to-six band to Oracle. The actual threshold of five leaves four on Deep, and the retained threshold tests explicitly assert that. A historical design explanation can be useful while still containing an error. Preserving it as a source does not require repeating its mistake.

Follow one request through the decision

Here is a synthetic decision fixture. It describes the inputs to the relevant branches, not an actual API payload and not a captured July request. I supply the complexity score to isolate routing; this example does not test whether the classifier assigns that score correctly.

{
  "query": "Compare two designs using the retrieved engineering notes",
  "classifier_score": 5,
  "oracle_complexity_threshold": 5,
  "fan_out_enabled": false,
  "estimated_full_prompt_tokens": 60000,
  "overflow_threshold_tokens": 100000,
  "hosted_route_enabled": true
}

A score of five passes the configured Oracle threshold. Fan-out is disabled, so the async override does not intercept it. Oracle retrieves its evidence and assembles a prompt. With an estimated 60,000 input tokens, the hosted-route condition is false and synthesis stays local. This explains how the extra capacity from Episode 2 could become reachable through the application.

Change only fan_out_enabled to true and the earlier override takes the request, even though _select_agent chose Oracle. Change only the score to four, keeping fan-out disabled, and the request selects Deep. Change only the estimated prompt to 100,001 tokens, with the hosted route enabled, and Oracle selects the hosted model. If the credential is absent, that condition stays false; the caller does not acquire an unlimited local window instead.

A score-five request, followed through

Pause flowExpand diagram

Close expanded diagram

This is a simplified route through the inspected branches. It omits graph-specialist selection, explicit search modes, retrieval details and later escalation. Those omissions are deliberate: the diagram answers why two configuration switches were needed and where the separate token decision occurs. It is not a complete architecture map or a claim that a real request completed.

A failed first attempt could become tomorrow’s answer

Routing to a working lane addressed the immediate experience, but the deep path had another failure mechanism. The overnight record describes multi-variant query embedding taking roughly thirty seconds when idle and exceeding the old 120-second client timeout under concurrent load. The request could return zero documents after around 121 seconds.

A timeout is already a failed user experience. The cache made it persist. The historical report says the per-pod cache retained empty results for its default 24-hour lifetime. An identical retry on the affected replica could then return “No results” in roughly a tenth of a second. The shorter response was not recovery. It was reuse of the earlier empty outcome without attempting the underlying work again.

An empty failed attempt should not circulate as a reusable answer. Conceptual illustration.

The source confirms a narrower mechanism than “never cache failures.” QueryCache.set inspects dictionaries containing a documents key. It skips the write if none of the outer list’s entries are truthy. Other dictionary shapes pass through, and the check does not establish that nonempty text is correct, relevant or successfully generated.

The guard below is an exact excerpt from query_cache.py at 9dbd25a1. It occurs before either the optional Redis write or the in-memory write:

if isinstance(result, dict) and "documents" in result:
    if not any(result["documents"]):
        logger.debug(
            "Skipping cache write for empty result set (query=%r)", query
        )
        return

Explain this code line by line

• The outer condition limits this check to dictionaries with a documents field. Other cached shapes retain their existing behaviour.
• any tests whether at least one outer entry is truthy. Both [] and [[]] fail this test, matching the retained empty-result cases.
• The debug message explains the skipped write without changing the returned search response.
• return exits before creating a cache entry. A later request can try the search again.
• This does not retry the current request, clear old entries or validate the meaning of a populated document. Those are separate responsibilities.

The retained test follows the important recovery sequence: attempt to cache {"documents": [[]]}, confirm the lookup misses, then cache {"documents": [["recovered"]]} and confirm the next lookup returns that content. The distinction is between avoiding a bad reusable state and fixing retrieval itself. A rejected cache write still leaves the current caller with the original empty result.

There is a useful edge to the implementation. A nested list containing an empty string, such as [[""]], has a nonempty inner list and therefore passes this outer truthiness check. That is a property of the inspected Python expression, not a claim that this payload occurred in July. It shows why the guard should be described as rejecting the known empty-container shapes rather than validating every possible failure representation.

Python’s own documentation for any and truth-value testing makes the behaviour inspectable without asking the reader to take the comment on trust. For a future cache contract, I would want an explicit outcome classification alongside result shape. That is a lesson from this reconstruction, not a change claimed as shipped in this episode.

More waiting time was a separate repair

The configuration also raised the embedding timeout to 300 seconds. That made the client more tolerant of a slow CPU retrieval path. It did not make embeddings faster, remove load contention or guarantee the whole gateway request had a compatible deadline. The cache guard and the timeout addressed different parts of the failure.

There was also a logging correction. The handover reports replacing ordinary exception stringification with repr(e) for the async embedding failure because the timeout’s ordinary message was blank in the observed case. An empty message had hidden the kind of failure. That is a report about those exceptions in that environment, not a rule that every HTTPX timeout always has an empty string. The HTTPX exception hierarchy distinguishes read, connect, write and pool timeouts; that distinction is useful diagnostic context.

The historical measurements explain why the wider Oracle band was kept even after the deep-path fixes. The overnight record reports Oracle around 26–39 seconds for the tested medium-band queries, with roughly seven seconds inside vLLM on average. Deep after the fix was reported at 48–103 seconds for raw chunks. These were different outputs and a limited set of requests, not a controlled benchmark proving one strategy superior for every workload.

They did change the next engineering question. If the model spent about seven seconds generating while the request took tens of seconds end to end, optimising decoding alone could not remove the whole wait. Retrieval, query embedding and other request work remained in the critical path. The records attribute substantial delay to CPU embedding; they do not contain a complete per-request trace in this review package from which I can independently apportion every second.

Observation in the historical records	What it supports	What it does not establish
Oracle returned a real synthesis in a 27.3-second smoke request	That recorded request exercised a working path	A latency percentile or all-request reliability
About seven seconds inside vLLM on average in the recorded checks	Generation was only part of the observed wait	Exact overhead for every request
Deep returned chunks after the timeout repair	The tested path could complete after waiting longer	Equivalent synthesis quality or a universal speed comparison
Empty results were rejected by the cache guard	Those shapes stopped becoming new cache entries	Removal of already cached entries or correctness of nonempty results

I would keep request latency and result usefulness beside each other. A fast empty response can be worse than a slower answer; a slow response can still be unacceptable even when it contains evidence. Neither the model’s token rate nor the gateway’s status code is enough on its own.

The missing metric already existed

The next day’s observability work began with a mistake in the handover. It said there was no routing counter by agent type and proposed adding one. On inspection, the counter already existed as rag_search_requests_total{agent_type,status}, along with dashboard panels for agent distribution. The earlier search had looked for a different name.

I want that reversal in the story because it changed the scope of the implementation. The real gap was the complexity dimension: which scores were reaching which agent after the threshold change? Adding another total by agent would have duplicated an existing view while leaving that question unanswered.

The resulting metric was rag_agent_routing_total{agent_type,complexity}. Its emit site used the resolved result’s agent type. That matters because the router’s readable reason could still say “Deep research” when Oracle had actually run. A display label is not a trustworthy substitute for a typed execution result.

Find the measurement attached to the actual path before adding another instrument. Conceptual illustration.

This exact excerpt comes from the gateway’s rag_endpoints.py at 9dbd25a1, after request processing:

_complexity_label = (
    "unknown"
    if _query_complexity == 99
    else str(min(max(_query_complexity, 0), 10))
)
rag_agent_routing.labels(
    agent_type=agent_result.agent_type, complexity=_complexity_label
).inc()

Explain this code line by line

• The gateway uses 99 internally when a complexity score was not available. That sentinel becomes the explicit label unknown.
• Other parsed values are clamped to the inclusive zero-to-ten range and converted to text for the metric label.
• The label therefore has a small bounded set of score values rather than an unlimited range of arbitrary integers.
• agent_result.agent_type describes the resolved result rather than guessing from the reason’s wording.
• inc() adds one to that labelled counter. Its position after processing means it is not a complete record of every request that started or failed before reaching this point.

The score was still extracted from routing-reason text earlier in the gateway. The added clamp made the label set bounded; it did not make the classifier calibrated or turn free text into a fully typed routing contract. The separate sentinel also means a genuine parsed value of 99 would be treated as unknown. Those are implementation boundaries worth keeping visible.

The dashboard grouped increases over the selected time range by agent and complexity. Prometheus’s counter guidance and increase documentation explain why a counter’s rise over time is more useful here than comparing raw per-process totals. Counters reset when the process restarts; labelled series may only appear on the replica that served that combination.

The historical report records requests producing Oracle series at scores five, six and eight, fast search at zero, and later checks with no out-of-range complexity labels after the clamp. That is evidence of the recorded emit path and its labels. It does not give me a week’s routing distribution, a quality improvement percentage or proof that all attempted requests were counted.

The same investigation corrected another overstatement. An empty secondary log index initially looked like a missing Elasticsearch sink, but pod logs were reaching a different primary index. A separate Grafana datasource failure remained real. I am not reconstructing that adjacent repair here; the relevant lesson is that a search in the wrong metric or index can produce a convincing absence that says little about the actual write path.

What changed, and what remained open

At the end of the overnight record, the gateway had a reported working Oracle route, the wider threshold, fan-out disabled, the larger overflow threshold and the longer embedding timeout. The cache write guard and better exception representation were reported deployed. The next day’s record adds the complexity metric, its bounded labels and dashboard view.

The plan also changed over time. The early ADR proposed removing the temporary routing settings once Deep was repaired. The later handover explicitly kept them based on that evening’s results and asked for another review after ordinary traffic accumulated. Treating the early revert plan as the final operational decision would miss that change of direction.

Practical lesson	What I would carry into the next change
Capacity and reachability are different	Follow a request from entry point to the actual model and back to useful content
A selector can be bypassed	Inspect early returns and overrides after the first decision
A cache preserves semantics as well as bytes	Test failed-then-successful recovery, not just repeated successful hits
A mitigation can remain useful after a defect is fixed	Retain the reason and re-evaluation condition rather than removing it automatically
Names are weak evidence of absence	Find the defined metric, scraped series and write index before declaring them missing
A new counter has an observation boundary	State where it increments and which failed requests may never reach it

For this draft, isolated checks exercise the retained selection helper, cache tests and the worked decision cases. The original GPU workload, hosted fallback and July traffic were not replayed. The inventory and claim ledger preserve that distinction, along with the missing raw traces and the deliberately excluded adjacent work.

The next July thread moves back toward the data entering the graph. A retained July 5 handover describes extraction output failing its expected schema and a diagnosis-first handoff. That is a grounded direction for the next chapter, not yet its final scope or a promise that the whole ingestion problem was solved. Here, the progress was more specific: the route could reach the capacity, failures became less reusable, and the measurement question became narrow enough to answer.

Terms and references

Glossary — plain-English definitions

Agent lane

An application path for a kind of search or synthesis; it is not a human team. Official reference.

Complexity score

The router’s estimated classification used for selection, not a proven measure of difficulty. Official reference.

Fan-out

Invoking several search branches and collecting their results; its presence does not guarantee better results. Official reference.

Early return

Exiting a function before later code runs; here the override bypasses normal agent lookup. Official reference.

Synthesis

Producing an answer from retrieved evidence; its usefulness still needs review. Official reference.

Token estimate

An approximation used to choose a model route; it is not a quality measurement. Official reference.

Overflow route

A configured alternate model path for estimated prompts above the threshold, when enabled. Official reference.

KV cache

Generation working memory whose budget competes with model weights and overhead. Official reference.

Query cache

A store of prior search results keyed for reuse; a hit can repeat a bad result. Official reference.

TTL

Time to live: how long an entry is retained as eligible for reuse. Official reference.

Truthiness

Python’s rule for treating a value as true or false; a nonempty list is true even if it contains an empty string. Official reference.

Timeout

A limit on waiting for one operation; raising it does not make the operation faster. Official reference.

Counter

A cumulative measurement that increases and can reset when its process restarts. Official reference.

Label cardinality

The number of metric series created by label combinations; bounding values helps limit it. Official reference.

Sentinel

A special value used to represent an unavailable or exceptional state; this gateway used 99 for missing complexity. Official reference.

Replica

One running copy of a service; per-process state may differ between copies. Official reference.

How this episode was reconstructed

This is a curated reconstruction of the July 2–3 routing arc. Its private review ledger links dated handovers, the routing decision record, the July 3 implementation report, historical platform source at 9dbd25a1 and historical GitOps revisions. Original operational reports remain distinct from newly inspected source and isolated tests. There is no claim of exhaustive period coverage or complete native-session replay. No pre-crafted Episode 3 or authoritative Season 5 continuation map was recovered in the scoped canonical searches; the title is an editorial proposal following the published Episode 2 preview. Habi, employer material, private finance, unrelated music and hackathon work are excluded. Original traffic traces, raw latency samples and a fresh long-input evaluation are unavailable in this package. Upstream references explain mechanisms; they do not verify Rootweaver’s historical deployment.