A Clean Comparison Changed the Decision
The serving cutover had reached the intended collection and restored dense retrieval. The next question was whether its results justified retiring the older path. A response could contain useful documents while the comparison used to judge it remained compromised by dependency failures or an unrepresentative probe.
I needed a usable comparison before choosing another retrieval change. The work first repaired how hybrid search handled an embedding failure, then examined paired queries and fusion methods. A later check changed both the preferred operational decision and the explanation of an apparent unstable result.

A few attractive examples need to be checked against the rest of the paired set. Conceptual illustration.
This chapter covers the July 14, 2026 retrieval comparison and closeout. It is reconstructed from the dated handover, signed ranking decision and historical retriever source at f8b3f97c. I owned direction and review, with agent-assisted implementation and analysis. The numerical results remain historical reports. The original complete query artifacts and scoring environment are not recovered here, so the new checks are limited to source structure and explanatory arithmetic.
A failed dependency could erase the whole response
Hybrid retrieval combined a dense leg with a keyword leg. The dense leg depended on a query embedding. The historical notes describe an upstream overload error encoded inside a provider response whose HTTP status could still be 200. A transport-success counter therefore did not identify every failed embedding operation.
The earlier implementation propagated the recognized embedding-backend exception. That could cause the complete hybrid request to fail even while keyword retrieval had useful results. In the earlier cutover account, only five of thirty-two queries were scoreable in both comparison phases.
The repair preserved a narrower fallback. An embedding-backend failure could leave the dense list empty and retain keyword results, with a degradation reason carried in the retriever’s response metadata. Other vector exceptions and keyword failures still had explicit failure paths.
retriever.py · hybrid_search result handling · platform f8b3f97c · exact historical excerpt
if isinstance(vector_results, EmbeddingBackendError):
logger.warning(
"hybrid dense leg unavailable; degrading to BM25-only",
exc_info=vector_results,
)
vector_results = []
degraded = "embedding_backend_unavailable"
elif isinstance(vector_results, BaseException):
raise vector_results
if isinstance(bm25_results, BaseException):
raise bm25_results
Explain this code line by line
- The first condition recognizes the specific embedding dependency failure.
- The warning preserves evidence of the reduced-capability path.
- The dense result becomes an empty list and a degradation reason is recorded.
- Other vector exceptions are raised instead of being silently converted into this fallback.
- A keyword-leg exception also propagates, since that leg is needed for the remaining search.
- The surrounding method later includes the degradation reason in its metadata.
That scope mattered. Catching every exception and returning whatever remained would have hidden programming errors and keyword failures behind the same reassuring response. The retained code makes the accepted dependency failure distinguishable from those other cases.
The handover reports the retriever image deployed and an actual overload absorbed by the fallback. It also records a later test gap on the keyword-leg propagation path. That test was added in the closeout. The code inspection and the reported live failure observation are separate sources of support.

A scoped fallback can preserve useful work while keeping its reduced capability visible.
Quality and availability needed separate records
A quality comparison needs corresponding observations for the same question. If one arm produces an answer and the other fails at a dependency, silently dropping the question changes the population being scored. Treating the failure as an ordinary low-quality answer changes the meaning in another way.
The useful record preserves both the request outcome and the conditional quality score. Here is a synthetic example explaining that separation.
{
"question_id": "illustrative-pair-2",
"first_arm": {"status": "ok", "quality_score": 0.7},
"second_arm": {"status": "dependency_failure", "quality_score": null},
"paired_quality_delta": null,
"availability_failure_recorded": true
}
The missing quality difference does not erase the availability failure. It keeps two questions available for inspection. One concerns whether a service completed the request. The other concerns the relative quality of two completed answers.
NIST’s paired-comparison reference describes matching each observation under one condition with its counterpart under the other. It provides the statistical vocabulary used here. The July numbers come from the retained reports. NIST paired-comparison reference
The later handover reports a clean run with twenty-two scoreable query pairs, no empty responses and no degraded responses. I keep that roster separate from the earlier thirty-two-query account because the complete underlying bundles are not retained here. A clean twenty-two-pair comparison does not prove that every earlier question was recovered unchanged.
Two comparisons answered different questions
The first comparison asked about the old and new serving collections. It involved changing the effective serving selection and checking which collection the retriever had pinned. The second comparison kept the new collection and changed a per-request fusion method.
| Comparison | Deliberate change | Other requirement |
|---|---|---|
| Earlier collection versus new collection | Serving selection | Flag propagation, correct process initialization and paired requests |
| RRF versus weighted fusion on the new collection | Per-request fusion method | Same query set and relevant request settings |
Mixing those comparisons would make a method result look like a corpus result. Their request boundaries and rollback implications were different.
The collection comparison’s reported mean scores were 0.5779 and 0.5654, with a difference of −0.0125 and an interval spanning −0.0643 to +0.0262. The interval crossed zero and extended beyond a narrow ±0.02 tolerance discussed by the handover.
That did not establish formal equivalence or non-inferiority within that tolerance. It did explain why a simple pass/fail display was a poor substitute for reading the uncertainty and the individual cases. The later operational acceptance was a decision made with those limits visible.
Follow the mechanism
The small fusion probe missed expensive losses
The first probe suggested that weighted fusion improved some exact-fact cases while harming an analytical case. It was useful for forming a question. The full twenty-two-query comparison was needed before changing the serving default.
The final decision report records mean RRF score 0.5705 and weighted score 0.5514. The weighted-minus-RRF point estimate was −0.0191, with a reported 95% bootstrap interval from −0.0851 to +0.0379. That interval left substantial uncertainty about the average effect.
The per-query account explained why the promising probe had been incomplete. Weighted fusion improved one factual case by 0.25, while a code-intelligence case fell from 0.422 to zero and a cross-domain case fell from 0.679 to 0.329. The earlier probe had not sampled those losses.
| Historical fusion result | Reported value | Reading for the decision |
|---|---|---|
| RRF mean | 0.5705 | Reference point in this paired run |
| Weighted mean | 0.5514 | Lower point estimate in this run |
| Weighted minus RRF | −0.0191 | Estimated average difference |
| Reported bootstrap interval | −0.0851 to +0.0379 | Direction and size remained uncertain |
| Query outcomes | Weighted better 7, RRF better 6, tied 9 | Aggregate means hid different local trade-offs |
A count of wins is not a sum of effect sizes. Seven small gains can coexist with six larger losses. That is why the complete per-query set mattered alongside the mean and interval.
Qdrant’s current hybrid-search reference describes reciprocal rank fusion as combining positions within ranked lists. The July application also had its own weighted-fusion implementation. The historical source, rather than today’s server feature set, defines what that application comparison tested. Qdrant hybrid-search reference
The weight knob needed its effective state
The handover says three tested weight values produced the same observed rankings. It described the fusion method as the important lever in that probe. The retained implementation supports a narrower interpretation than saying the weight parameter was universally irrelevant.
The weighted-fusion code multiplies normalized leg scores by their weights. Earlier in the same path, an enabled dynamic-alpha branch can replace the supplied values.
retriever.py · conditional dynamic-alpha assignment · platform f8b3f97c · exact historical excerpt
dat_alpha = compute_alpha(v_eff, b_eff)
vector_weight = dat_alpha
keyword_weight = 1.0 - dat_alpha
Explain this code line by line
- The first statement derives a weight from the evaluated leg-effectiveness values.
- The dense leg receives that derived value.
- The keyword leg receives its complement.
- The surrounding conditional requires dynamic tuning to be enabled and both result lists to be present.
- The excerpt does not establish that this branch ran in the historical probe.
The raw effective request and configuration record is incomplete here. I therefore treat identical rankings at those three tested weights as a local observation. The source shows why a future investigation should retain the effective weights and dynamic-tuning state before deciding a knob is disconnected.
This is the same practical discipline as following the named vector into the final adapter call. A supplied setting and the value used by the active implementation can differ. The evidence record needs the receiving boundary.
Repetition changed the jitter diagnosis
Another early explanation concerned a factual query whose score had drifted between runs. The handover attributed that to embedding non-determinism. The later signed decision explicitly overturned the explanation.
It reports ten back-to-back identical queries with identical scores, membership and ordering. Three rephrasings used distinct cache keys and all retrieved the answer chunk, while their scores differed. Across the thirteen calls, the answer chunk appeared every time.
Those observations supported a more specific account. The check was sensitive to wording and the earlier runs differed in their probe or process context. The controlled repetition did not reproduce the proposed instability. It also did not prove that every embedding request would always be deterministic.

Repeated observations need stable conditions before a changing result can be attributed to the component under investigation.
The distinction affected the next action. A claim of model jitter suggests a different repair from a deterministic system whose scoring rule reacts to wording. Repeating the relevant observation under controlled conditions was cheaper and more informative than changing the model route on the basis of the first explanation.
The decision was to keep the existing default
The closeout kept RRF. The fuller comparison supplied no persuasive operational reason to switch to the weighted method, and the localized gains came with losses elsewhere. This was an acceptance decision with uncertainty retained, rather than a claim that RRF had been proved superior for every future request.
The same closeout described the new collection serving and the older collection retained for rollback. A snapshot and a later deletion date separated the serving decision from irreversible cleanup. A deadline in a runbook was not evidence that deletion had happened.
There was another concrete recovery detail. The runbook needed a snapshot copied off the collection’s storage before deleting that collection, because an in-tree snapshot could disappear with it. The comparison decision and the recovery artifact had separate acceptance checks.
What I would carry into the next build
Record dependency failures separately from conditional quality. A successful fallback should be visible enough that a comparison can distinguish a full hybrid response from a keyword-only one.
Pair the same questions under clearly stated conditions. Preserve the roster, request parameters, effective configuration, returned metadata and per-query outcomes. A headline average cannot reconstruct those boundaries later.
Read the complete trade-off before changing a default. A small probe can reveal an interesting case and still miss the queries that determine the net result. Preserve both the point estimate and its uncertainty.
Finally, correct a causal story when a controlled check contradicts it. The later evidence changed the jitter diagnosis. The next chapter follows another experiment where repairing the implementation was necessary, but the remaining comparison still changed several factors at once.
Glossary and primary references
Paired comparison
A comparison in which each question has a corresponding observation under each condition.
RRF
Reciprocal rank fusion combines ranked lists using candidate positions.
Confidence interval
An uncertainty interval constructed by a stated statistical procedure.
Point estimate
A single estimated value, such as the observed mean difference.
Degraded retrieval
A disclosed reduced-capability search path after a dependency fails.
Embedding
A numerical representation used for similarity search.