Skip to content

More Workers Were the Wrong Fix

S5·E10 ·Rootweaver · KnowledgeGraphs · Reliability · Performance

Episode 9 ended with the graph backfill paused after another search timeout appeared under load. The empty-candidate repair had removed one unnecessary operation. Non-empty searches still reached the database, where some operations remained expensive enough to stop documents completing.

I wanted the backlog to drain reliably. The next investigation tried several plausible capacity changes, including more model-serving slots and overlapping document ingestion. The later evidence changed the direction. Parallel episodes competed for shared graph state, while the database query took a much broader route through that state than its supplied relationship identifier required.

The Rootweaver engineer traces a working rope through a crowded canal-lock mechanism.

Additional pulling capacity still depends on the mechanism carrying the effort to the required place. Conceptual illustration.

This chapter covers July 13–14, 2026. It is a curated reconstruction from the concurrency decision, throughput and query-fix handovers, the following deployment checkpoint and historical source at 3a0ff03e. I owned direction and review, with agent-assisted investigation and implementation. The measurements remain attributed historical reports. Retained code and new arithmetic checks make the mechanism inspectable, while the original database profiles and full runtime bundles remain unrecovered.

The plausible capacity changes

The work moved through generation, embedding and graph operations. Increasing capacity at one stage could help only while that stage constrained the job. A model server with spare capacity could still wait for a document whose graph search was scanning too much data.

The earlier July 13 handover describes a model-serving concurrency sweep and an increase in available sequence slots. It also records ingestion concurrency set to four. At that checkpoint, a short smoke had found no duplicate entities. The handover described the change as safe while leaving its throughput benefit unproved.

The observation window was incomplete. A later thirty-document verification run produced a different picture. The durable concurrency decision reports eighty-four search timeouts across six batches, roughly 43% document failure and newly duplicated entities. It also reports no throughput advantage from overlapping whole episodes.

The sequence matters. The first smoke supplied a limited checkpoint. The later run provided enough evidence to withdraw its reassuring safety interpretation. Preserving that correction explains why the decision changed, instead of presenting the final answer as something that had been obvious throughout.

Historical observationUnitWhat it helped establish
84 timeouts across six batchesSearch-call failuresRepeated database operations failed under the verification load
About 43% failedDocuments in the reported runA substantial part of the work failed to complete
Duplicate query changed from 0/0 to 12/14Duplicated names and extra entity nodesOverlapping episodes introduced a consistency problem
No reported throughput benefitWhole ingestion pathMore concurrent episodes had not delivered the intended improvement

Those denominators describe different properties. One document can perform several searches. The duplicate query counts names with multiple nodes and the extra nodes beyond one per name. A count of failed calls cannot be substituted for a document failure count or a consistency measure.

Two defects shared one noisy symptom

Overlapping episodes increased pressure on the database. They also raced entity resolution. Two documents could independently decide that an entity needed creation before either observed the other’s completed write. The decision record associates new duplicate timestamps with the concurrency window.

A query-plan repair could reduce the cost of finding existing relationships. It would still leave the safety of concurrent entity creation to establish. Reducing concurrency could stop the observed race while leaving an individual query inefficient. Each defect needed its own acceptance question.

The operational decision returned whole-episode ingestion concurrency from four to one. The record identifies the GitOps change and running image separately. The rollback addressed the reported consistency problem before the later query repair was deployed.

It also left a design question open. Graphiti coordinated work inside an episode. Increasing concurrency within an implementation that understands its deduplication boundaries differs from wrapping several complete add_episode calls in an outer semaphore. The records proposed studying that distinction later. They do not establish that a safe concurrent replacement was completed during this chapter.

The plan exposed unnecessary work

Profiling the relationship search supplied the next useful observation. FalkorDB’s GRAPH.PROFILE executes a query and returns its operator tree with execution measurements. The current reference also explains that profiling retains the modifying behaviour of a modifying query. Here it explains the inspection method. The July result remains in the historical report. FalkorDB profiling reference

The old source already had a relationship UUID from its full-text candidate result. Its expansion included node-label predicates on both ends of that relationship.

client-before.py · relationship expansion · parent of platform 3a0ff03e · exact historical line

MATCH (n:Entity)-[e:RELATES_TO {uuid: rel.uuid}]->(m:Entity)
Explain this code line by line
  • n and m name the endpoint nodes.
  • :Entity asks for a label on each endpoint.
  • e:RELATES_TO names and types the relationship between them.
  • {uuid: rel.uuid} selects the relationship identified by the current candidate.
  • The line describes the requested match. The planner chooses how to find it.

The report says the planner began with a node-label scan, expanding substantial intermediate work before reaching the relationship selection. The query contained a selective identifier, but its physical access path still had to be inspected.

The corrected source removed the endpoint-label predicates from this expansion.

client.py · relationship expansion · platform 3a0ff03e · exact historical line

MATCH (n)-[e:RELATES_TO {uuid: rel.uuid}]->(m)
Explain this code line by line
  • The endpoint variables remain available for constructing results.
  • The relationship type and UUID constraint remain present.
  • The endpoint-label requirements are absent from this match.
  • The reported profile then selected a relationship-index path.
  • Applicability depends on the data and query contract. Removing predicates elsewhere requires its own correctness review.

That boundary prevents the example becoming a universal tuning recipe. Removing predicates changes the logical condition. The July account reports result parity for the inspected workload and an improved access path. A different graph, different endpoint types or another relationship model would need another review.

The engineer follows a short chain directly to a selected gear in a canal maintenance pit.

A selective route to the required relationship can avoid examining unrelated starting points. The gears illustrate an access path rather than a measured database trace.

The candidate budget had a correctness job

The repair retained a bound before graph expansion. Its size also accounted for a caller that had already supplied a longer candidate list.

client.py · candidate budget · platform 3a0ff03e · exact historical excerpt

candidate_limit = max(
    limit,
    search_utils.RELEVANT_SCHEMA_LIMIT,
    len(search_filter.edge_uuids or []),
)
Explain this code line by line
  • candidate_limit is an intermediate search budget.
  • limit is the requested final result count.
  • The library’s schema limit supplies another lower bound.
  • A supplied UUID list can require a larger budget so its candidates survive this stage.
  • max selects the largest requirement. The later result limit still controls the returned count.

Consider a synthetic request for ten results, with a library floor of twenty and thirty-one supplied candidate identifiers. The intermediate budget becomes thirty-one. Selecting the final ten results remains a separate step. Reducing the intermediate budget to ten would discard some of the caller’s already bounded search opportunity.

{
  "requested_results": 10,
  "illustrative_schema_floor": 20,
  "supplied_candidate_count": 31,
  "intermediate_candidate_limit": 31,
  "final_result_limit": 10
}

This newly constructed example explains the fields. It is not an original request or a recovered profile. Its fresh check evaluates the arithmetic and verifies the retained source shapes. It does not execute the database planner.

Follow the mechanism

FailuresCount failed recordsTrace workFind the constraintProfileInspect access pathBound writesProtect graph stateCompleteCheck marked records
FailuresCount failed recordsTrace workFind the constraintProfileInspect access pathBound writesProtect graph stateCompleteCheck marked records
Simplified editorial model of the inspected mechanism. Dots show direction.

The diagram follows the investigation from an operational failure count to a physical query path and back to completion. Its connectors describe reasoning and data flow. They are not a trace of the historical job.

The deadline was another boundary

The concurrency decision exposed a completion problem that an efficient query would not automatically solve. The scheduled job had a two-hour active deadline. The reported steady rate implied a substantially longer backlog drain.

Using the decision record’s illustrative rate of about three and a half minutes per batch, 223 batches require roughly 780 minutes, or thirteen hours, before allowing for failed work and other overhead. A two-hour limit permits only about thirty-four such batches. These are derived planning figures, with the rate held constant to explain the mismatch.

Successful records were marked for future runs to skip. Failed records remained eligible. Repeatedly reaching a deadline before enough successful marks accumulated could keep the backlog substantial even while graph-node counts increased. The useful operational observation was completed eligible documents across runs.

A marker records one stage’s success. It is not automatically a transaction spanning model generation, graph writes and the marker store. A crash between effects can make retries relevant, and a graph count can rise through partial work. That is another reason to inspect the actual processed and failed counters alongside graph growth.

The following day’s handover records a dedicated drain with a twenty-four-hour deadline. It reports the query-fix image deployed, ingestion reconciliation resumed and a checkpoint around batches fifteen to sixteen of 202. At that point it recorded five processed and zero failed per completed batch, with no database timeouts over 109 minutes.

That was useful progress evidence. The handover still described the drain as running. Its estimated seventeen to nineteen hours for the job was an estimate. The checkpoint does not establish that the entire backlog completed.

The engineer moves a single handcart across a narrow lock bridge, with completed carts visible on the far bank.

Reliable completion marks let later work advance through a backlog. The waiting carts keep unfinished work visible.

Compare the observation windows carefully

The records contain several tempting numbers, including isolated profile timings and timings under load. They describe different measurement contexts. I have kept a headline speed multiplier out of this reconstruction because the retained summary does not supply one matched end-to-end comparison behind every quoted ratio.

The old and new observations also span different images and settings. Whole-episode concurrency changed, the query changed, and the drain deadline changed. Together they support the recorded operational direction. They do not isolate a universal effect attributable to one knob.

The distinction leaves the engineering result concrete. The inspected source shows a changed relationship match and a preserved candidate budget. The decision records a deliberate concurrency rollback. The deployment handover reports the intended image running and successful batches at a stated checkpoint. The end of the complete drain and cleanup of existing duplicates remain separate acceptance items.

What I would carry into the next build

Profile the operation doing the work before increasing its callers. A short smoke can identify an immediate defect, while a longer run can reveal a race the smoke did not encounter. Preserve that change in evidence when the decision changes.

Treat consistency and latency as separate properties. The query fix addressed an inefficient access path. Sequential episode ingestion addressed the observed duplicate-creation window. Existing duplicate nodes still needed their own cleanup.

Give long work a realistic execution budget and an inspectable resume contract. A healthy service can spend many hours on legitimate extraction. A deadline must be understood alongside the batch rate and the way success is recorded.

Finally, follow the deployed image and completed records. A merged source change is an authoring event. A running image and a successful batch are observations from later boundaries. Keeping them connected makes the account useful to the next operator.

Another thread was unfolding alongside the backfill. The retrieval cutover had a serving flag, a new collection and responses that looked successful. The next chapter returns to that parallel July 13–14 sequence to ask which component was actually serving the query and whether its dense-retrieval path had survived the switch.

Glossary and primary references

Query plan

The operations the database chooses to execute a query.

Primary reference

Relationship index

A structure that can locate a graph relationship by an indexed property.

Primary reference

Concurrency

Operations allowed to overlap. Here the setting wrapped whole ingestion episodes.

Primary reference

Entity resolution

Deciding whether extracted references represent an existing graph entity.

Primary reference

Active deadline

An execution-time limit for a Kubernetes Job.

Primary reference

Checkpoint

A bounded observation of progress, with unfinished work still visible.

Primary reference