Skip to content

The Flag Was On. The Dense Route Was Still Wrong.

S5·E11 ·Rootweaver · Retrieval · Observability · Reliability

A second thread was running alongside the graph backfill. On July 13, Rootweaver was switching retrieval from the earlier vault collection to a new collection with named vectors. The new data could exist, the flag could be enabled and a search could still return plausible text while the intended retrieval path remained incomplete.

I needed to know which collection the process doing the search had selected, and whether dense retrieval contributed to its answers. That required following the request beyond the component where the flag had first been wired. The gateway delegated its actual search to a retriever with its own configuration, network rules and vector-store call.

The engineer inspects an open control wheel above two aqueduct channels, with the intended channel still dry.

A control setting needs a working path to the component carrying the request. Conceptual illustration.

This chapter returns to the parallel July 13–14, 2026 cutover. It is a curated reconstruction from historical handovers, GitOps revisions and retriever source at f8c34be9. I owned direction and review, with agents assisting implementation and verification. One legacy cutover handover fails its signature check. Its runtime account remains attributed and is identified in the source ledger. Historical code and GitOps files independently establish the inspected implementation changes, while the later signed closeout supplies the subsequent serving and rollback account.

The flag belonged to an effective environment

The first correction concerned where the flag lived. The initial ticket premise treated registration as a GitOps change. The cutover account says the actual flags were managed through Unleash’s administrative API and stored as service state. A configuration repository could document integration without becoming the authoritative flag registry.

The next detail was easy to misread. The live gateway’s client used the environment named development. Enabling a flag under an environment named production would therefore leave this particular client reading its existing value. The environment’s label described the feature-flag namespace selected by the credential, independently of how seriously I operated the lab service.

Here is a synthetic configuration summary explaining that relationship. It contains no real credential or operational instruction.

{
  "service_role": "live retrieval gateway",
  "flag": "rootweaver.serving.v2.enabled",
  "environment_selected_by_client": "development",
  "selected_environment_enabled": true,
  "other_environment_state": "does not identify this client's value"
}

The useful question was which value that client could obtain. The flag’s name and its appearance in an administration screen did not answer that alone. Registration, selected environment and the component reading the setting were distinct pieces of the request path.

The account also records that I explicitly chose a direct cutover while deferring the proposed shadow-validation step. That is a historical decision about this change. It should not be rewritten as a fully completed shadow rollout merely because that would make the story more orderly.

The gateway was not the entire serving path

The gateway delegated retrieval. The retriever executed the vector-store request. Wiring the gateway’s flag client left another process to inspect.

The July handover describes a retriever with neither the required Unleash configuration nor the network-policy egress needed to reach the flag service. Its resolver took a fail-safe path to the old collection. That behaviour kept a search surface available while concealing the intended cutover from a check that only asked whether results existed.

Two historical GitOps snapshots corroborate the repair’s implementation boundaries. One revision added the retriever’s Unleash environment integration. Another added its egress path to the control service. Those changes are separately inspectable in the retained source files. They establish what the manifests expressed. Deployment observations remain in the dated accounts.

BoundaryQuestion that matteredEvidence required
Flag serviceWhich environment did the client read?Effective client scope and selected value
GatewayDid it delegate the request?Actual retrieval route
Retriever configurationCould this process resolve the profile?Its own environment and startup selection
Retriever network pathCould it reach the flag service?Relevant egress rule and an observed control request
Vector-store callWhich collection and vector space were selected?The arguments reaching the adapter

This is a common distributed-system trap because each component can look reasonable in isolation. A gateway can know the desired collection while the delegated reader remains configured differently. A default can preserve availability while making a configuration failure resemble a successful decision to retain the earlier behaviour.

Initialization made timing part of the change

The historical retriever selected its profile during construction.

retriever.py · constructor profile selection · platform f8c34be9 · exact historical statements

_profile = resolve_serving_collection()
self.collection_name = _profile.collection_name
Explain this code line by line
  • The first statement asks the resolver for the active serving profile.
  • The second stores the profile’s collection name on this retriever instance.
  • A later flag change must be considered alongside the lifetime of that instance.
  • The excerpt establishes assignment at construction. It does not demonstrate a live configuration refresh.

The implementation therefore made process lifetime relevant. The cutover notes describe waiting for flag propagation and then restarting until the process pinned the intended collection. Restarting immediately after changing the flag could read a cached older value and construct another instance with the old selection.

The handover contains shorthand about a roughly fifteen-second poll and rollback. The constructor gives that shorthand a necessary boundary. A control-client poll and an already constructed retriever’s stored collection are different states. A rollback check needed the process’s actual serving selection, rather than a timing assumption based only on the flag service.

The engineer checks a cable entering a coastal relay station through its permitted wall opening.

The service doing the work needs its own configuration and permitted control path. The cable is a conceptual representation.

A named collection still needed a named vector

Resolving the new collection exposed another boundary. The new collection stored its dense representation under the name dense. The earlier collection used an unnamed default vector. The retriever initially discarded the profile’s vector-space name when constructing the query.

Qdrant supports multiple named vector spaces within a collection. A query has to select the intended space. That current reference explains why a collection name alone is an incomplete description of a request. It does not establish which request the July service made. Qdrant vector reference

The historical fix stored the name from the profile on the retriever.

retriever.py · constructor vector-space assignment · platform f8c34be9 · exact historical statement

self.dense_vector_name = _profile.dense_vector_name
Explain this code line by line
  • The right-hand value comes from the selected serving profile.
  • The left-hand attribute keeps that value on the retriever instance.
  • The inspected search call later passes this attribute through its using argument.
  • The attribute can preserve the earlier unnamed behaviour when the profile supplies no vector name.

The last point mattered for compatibility. A migration could need a named query for the new collection and an unnamed query for its retained rollback collection. Hardcoding the new name into every call would have changed the older path’s contract. Carrying the profile’s choice preserved the distinction.

The following is a simplified explanatory request, based on the inspected adapter call. It is not an exact historical wire payload.

{
  "collection": "rootweaver_vault_v2",
  "using": "dense",
  "query_vector": "illustrative embedding omitted",
  "limit": 10
}

The collection chooses the body of stored points. The using field selects a vector space inside that body. The query representation must also be compatible with the selected space. Choosing a collection successfully says little about the other two requirements.

The empty-result boundary hid useful evidence

An unnamed query against the new collection could fail at the vector-store boundary. The old account says the resulting bad request was converted into an empty dense result by a broad exception path. Keyword retrieval could still supply plausible results to a hybrid response.

The retained implementation shows that general exception handling returned an empty list, while the specific embedding-backend error propagated.

retriever.py · vector-search exception boundary · platform f8c34be9 · exact historical excerpt

except EmbeddingBackendError:
    raise
except Exception as e:
    logger.error(f"Vector search failed: {e}", exc_info=True)
    return []
Explain this code line by line
  • A recognized embedding-backend error is raised to the caller.
  • Other exceptions enter the general handler.
  • The handler records the error and traceback in logs.
  • The method then supplies an empty list to its caller.
  • The caller can therefore receive an apparently ordinary empty leg even though logs contain a failure.

The failure was not completely invisible. It crossed a boundary where its meaning changed. An operator reading the logs could find it, while a check that counted returned documents could miss the loss of dense retrieval.

The structural signal in the cutover account was the number of returned results with dense contribution. It reports that contribution moving from zero to an average of 7.40 in the described ten-result checks, close to the older path’s reported 7.20. Those counts helped establish that the dense leg had become active in that observation window. They were not an answer-quality score.

The engineer matches a shaped key to a specific destination slot in a sorting hall.

Selecting the correct destination requires carrying its identity through the final hand-off. Conceptual illustration of a named vector space.

Follow the mechanism

FlagEffective environmentRetrieverServing processControl pathConfig and egressVector nameSelected spaceEvidenceInspect contributions
FlagEffective environmentRetrieverServing processControl pathConfig and egressVector nameSelected spaceEvidenceInspect contributions
Simplified editorial model of the inspected mechanism. Dots show direction.

A fast response could test a different path

The same records contain very different latency observations. Cached gateway responses were fast. Uncached direct full-hybrid retrieval took much longer, and a reranking path could time out before returning results. Those observations belong to different request paths and cache states.

A cache hit can avoid the retriever operation under investigation. That is useful for service responsiveness, but it makes a poor standalone test of a changed vector-store call. The cutover needed a request that exercised the actual dense leg and exposed its contribution.

Reported pathApproximate historical observationScope of the observation
Gateway fast path, cached15 millisecondsResponse from the cache-oriented path
Gateway fast path, cache miss90 millisecondsThe stated fast-mode probe
Direct uncached full-hybrid retrieval11.4 seconds medianThe expensive retrieval path described in the handover
Reranking pathAround 40 seconds followed by no resultsA separately recorded problem, disabled on the serving path

These figures cannot be composed into a single migration speed-up. Their useful role is to show why request shape, cache state and component boundary belong beside a latency number.

The cutover report also identifies a whole-corpus keyword-index construction path as a separate performance opportunity. That remained distinct from getting the dense-vector name through the adapter. A request could be correctly routed and still have an inefficient keyword leg.

Keep the rollback and inspect the comparison

The later signed closeout describes the new collection serving, with the old collection retained as a warm rollback and a snapshot kept for recovery. Physical deletion was scheduled for a later date. It was not part of this chapter’s observed cutover checkpoint.

The earlier quality comparison still needed repair. The legacy handover reports that only five of thirty-two queries were scoreable in both arms because embedding failures could collapse the complete hybrid response. That made the apparent average a poor basis for deleting the rollback collection.

The implementation had reached a useful state without answering every quality question. The retriever could read the intended flag environment, reach its control service, select the intended collection and carry the vector name into the search. Dense contribution supplied one structural observation. A fair paired quality comparison required another set of checks.

What I would carry into the next build

Trace the component that executes the request. A flag in the gateway does not automatically configure its delegated reader. Configuration, network access and selected data representation each have a concrete receiving boundary.

Test the intended route with an observation that can distinguish its internal legs. A successful response and a nonempty result list can coexist with a missing dense component. Keep logs and returned metadata aligned enough to expose that situation.

Observe configuration changes through process lifetime. A polling client can update while a constructed instance retains an older assignment. The serving process’s effective selection is the acceptance observation.

Preserve a rollback while evaluating the new path. The next chapter follows the work needed to make that comparison usable, including a scoped keyword fallback, matched query pairs and a later result that changed the interpretation of the first promising probe.

Glossary and primary references

Named vector

A vector space selected by name within a Qdrant point or query.

Primary reference

Serving profile

The selected collection and vector-space settings used by a reader.

Primary reference

Control plane

The configuration and coordination path that directs a service.

Primary reference

Data path

The component sequence that processes the actual search request.

Primary reference

Feature flag

A runtime setting used to choose between behaviours.

Primary reference

Rollback

Returning to a retained serving configuration when the change is unacceptable.

Primary reference