The Job Was Green. The Graph Was Not.

The job finished green. The graph inside it was not.

By Wednesday 1 July, the work had crossed a familiar boundary: code had been changed, an image had been built, and a job had run. Yet individual documents were still failing on their way into the knowledge graph. A successful process exit was concealing a much less successful piece of work.

Making a hidden failure visible changes what can be diagnosed; it does not itself complete the work. Conceptual artwork; provisional hero for review.

This is a proposed Season 5 opener, though the season map itself is provisional — the structure will shift as the publishing pipeline sees fit. The setup began on June 30; the core diagnosis is July 1, with a declared July 2 follow-up because it changes what the fixes actually established. I set the engineering direction and acceptance; agents assisted with diagnosis, code and review. The historical reports and directly inspected source have different roles in this account.

But the debugging is only half the story. The other half is intent: why I wanted a knowledge graph at all, and what four months of design decisions had committed the platform to before this job ever ran.

What this season is for

Season 4 ended with a question about boundaries: what an agent’s output should be allowed to do, and how to check it. This season starts one layer below that question. Before an agent’s output matters, the data product underneath it has to be real. A retrieval path that answers from an empty or half-populated graph is not a trustworthy answer no matter how well the agent behaves above it.

The Graph’s Charter: Four Months of Intent

This wasn’t a sprint. It was a four-month campaign to build something structurally different from what the platform already had. The story of the intent arc is as important as the debugging that followed it.

February 7: The Initial Pitch

The engineering brief, “AI Ecosystem Research Briefing,” framed the problem on February 7. The proposal wasn’t just to add a new tool — it was to change the fundamental way the system retrieved and understood data:

“Implementation: Design knowledge graph schema, build entity extraction pipeline (could leverage NER tagging skill), populate FalkorDB, add graph retrieval to hybrid search fusion.”

The proposal identified a key insight: the existing search was capable, but structurally blind. It could find text, but it couldn’t reason about relationships.

February 15: The Blueprint (ADR-015)

A week later, ADR-015 (“Graphiti + FalkorDB Knowledge Graph Integration”) turned the proposal into a decision. The document identified the specific blind spots we were solving for:

1. No relationship awareness — Vector search can’t traverse entity relationships (e.g., “what depends on rag-gateway?” or “which docs reference Kafka?”).
2. No impact analysis — No way to calculate the blast radius of a change across the knowledge base.
3. Disconnected code graph — FalkorDB already hosted code intelligence, but it was isolated from document search.
4. No temporal entity tracking — The system couldn’t track how entities evolved over time.

The architecture decided upon was a hybrid stack. Instead of replacing the existing vector search, we would fuse it:

“Links between document entities and code entities enable ‘which docs describe this module?’ without breaking the existing search.”

This was the pivotal moment: the goal wasn’t just search, it was understanding. We weren’t just adding features; we were changing the fundamental nature of what the system “knew.”

April 18: The Model Allocation Decision

By April, the scoping became more specific. A major technical constraint arose: the main vector database indexing was critical for the platform’s performance. Any experiment or heavy lifting for the knowledge graph could not starve the main pipeline.

The decision was made to isolate the graph work from the main vector indexing infrastructure. This was the moment the “hybrid” architecture became a technical requirement, not just a nice-to-have.

June 30: The Validation Checkpoint

By the end of June, all the pieces were in place — the schema, the pipeline, the infrastructure, the security policies (the path-scoping requirement from the charter). The system was deployed.

The only question left was: does it actually work?

That question was the pivot point for this episode.

A count was too small an acceptance test

The goal was to populate the document knowledge graph from architecture documents. Graphiti, developed by Zep, supplied the graph-building machinery; Rootweaver supplied its integration, a self-hosted vLLM inference server for the extraction model, and the embedding path. Those are multiple opportunities for a document to fail even when the surrounding job remains healthy enough to continue.

The acceptance check written for this work was empirical: the graph should contain Entity nodes produced by a live backfill, not by a probe. The June 30 handover reports the count at zero even after that day’s code fixes were merged to main. Four nodes had appeared by July 1, up from zero.

Four is progress through the path. It is not a corpus. It did not show that the selected documents had been ingested, that each one succeeded, or that recurring operation was ready. It also could not distinguish between a pipeline that was slowly filling and one that was filling almost everything by exception.

The same July 1 report says the ingestion job completed with exit code zero while catching per-document failures and continuing. That behaviour can be intentional: one bad item should not always stop a batch, because a single malformed document should not block a thousand good ones. But it means the scheduler’s success state and the data product’s completion state need separate definitions. The scheduler says the job ran. Only the item-level record can say what the job actually did.

I would want to know how many items were selected, attempted, completed and failed, alongside the markers that determine whether a later run retries them. A graph count is one signal in that account, not a substitute for it.

The first useful fix made failure visible

The June 30 session had already shipped two sets of fixes to the platform’s main branch, and the first live backfill against that image failed. The failure was hard to read. The July 1 diagnosis identified the observability layer first, because without it nothing else could be named.

The sequence on June 30 is worth retelling because it shows how a fix can be correct and still fail in production. Two merge requests shipped to the platform’s main branch that day, and each moved separately through the deployment path.

Change merged to main	What it carried	What the running cluster actually had
First merge request	A guided-JSON subclass with a token cap, plus a selector path-scoping fix	Pinned into the first image bump before the second merge landed
Second merge request	A contract correction for the structured-decoding request	Not in any image yet; its own container build was still running at handover

The first live backfill then ran against the image that had the first fix but not the second, and the inference server rejected the request combination that the second fix addressed. The graph stayed clean, which the retry design made possible, but the backfill did not complete. The second fix was on main, but it was not in the image the cluster was running.

That is not a story about a bad fix. It is a story about the delay between merging code and running it, and how a deployment can look finished while it is still catching up to what main already knows.

The second fix’s contract correction also had a design purpose worth showing. It put the structured-decoding mode behind an environment switch, so the request contract could change without another image rebuild:

Two dedented fragments, condensed: packages/rootweaver-graph/src/rootweaver_graph/graphiti_client.py, guided-decoding selection, historical commit 1e61f206. Unrelated branches between them omitted; shown lines unchanged.

def _guided_decoding_mode() -> str:
    return os.getenv(GRAPHITI_GUIDED_DECODING_ENV, "guided_json").strip().lower()

and within the guided client’s completion call:

guided_decoding = _guided_decoding_mode()
if response_model is None or guided_decoding == "off":
    return await super()._create_completion(
        model=model,
        messages=messages,
        temperature=temperature,
        max_tokens=max_tokens,
        response_model=response_model,
        reasoning=reasoning,
        verbosity=verbosity,
    )
...
if guided_decoding == "json_schema":
    request_kwargs["response_format"] = {
        "type": "json_schema",
        "json_schema": {
            "name": response_model.__name__,
            "schema": schema,
        },
    }

The retained default is guided JSON; the JSON-schema branch is the mode the July 2 fix would adopt on the generic chat path. A contract that is a deployment variable rather than a compiled-in choice is what let the later correction ship as configuration.

The ingestion path recorded the exception’s string representation. Some exceptions do not have a helpful string. A timeout can carry an empty message, which would leave a log line saying that an episode failed without revealing the type or the stack that explained where it failed. The historical record describes exactly that: a blank-looking failure that kept having to be rediscovered.

The historical source changed the handler to logger.exception and still returned False. The code did not make the document succeed. It made the unsuccessful attempt more inspectable.

Exact fragment, dedented: packages/rootweaver-graph/src/rootweaver_graph/graphiti_client.py, add_episode(), historical commit 1e61f206. Intervening explanatory comments omitted; executable statements unchanged.

except Exception:
    logger.exception(f"Failed to add episode '{name}'")
    return False

Explain this code line by line

1. The except branch catches any exception raised during the episode addition attempt.
2. logger.exception records the message with the full traceback and exception type. The document name identifies the failed item.
3. Returning False signals that this specific episode did not succeed, without aborting the surrounding batch.

The Python Software Foundation’s logging reference explains that exception-logging behaviour. The change went in through a merge request authored by the coding agent. A regression test was written to fail against the old code and pass against the fix, and the package’s full test suite ran green across two seeds.

The important engineering change was not that documents suddenly succeeded. It was that failures could now be seen and classified.

One symptom concealed independent causes

Once the exceptions became visible, the July 1 record identified two independently fatal paths. The embedder could run out of time while processing a large batch. The extraction model could return text that did not parse as the required JSON.

Failure	Evidence in the July 1 report	Intervention it needs
Embedding read timeout	Large batch against a 30-second client timeout	Configurable timeout and bounded sub-batches
Extraction JSON error	A large response failed parsing	Inspect the actual client and structured-output request path
Job completion despite failures	Per-item errors caught and continued	Inspect item outcomes and graph/marker state
Model-service cycling	Connection failures during a full backfill	File as its own dependency-resilience work item

Those require different interventions. Waiting longer for an embedding request cannot repair malformed extraction output. A schema correction cannot make an overloaded embedding batch finish before its client timeout. Treating both as one vague graph-ingestion failure makes it easy to keep changing the wrong layer.

Job health and document outcomes are separate

Pause flowExpand diagram

Close expanded diagram

Cause A: the embedder’s 30-second ceiling

The first cause was an embedding timeout, and it was the cleaner of the two. The embedding client was configured with a thirty-second timeout and sent the whole document batch in one request. The embedding model behind that request was Qwen3-Embedding-4B, a four-billion-parameter model producing 2560-dimension vectors. The July 1 report notes the inference server doing extraction was contending for the same graphics card, which slowed the embedding service’s response.

The retained embedder source shows the shape of the problem directly. One helper builds a single HTTP client with a thirty-second timeout, and the batch path funnels the whole workload through one POST to it:

Exact fragment, dedented: packages/rootweaver-graph/src/rootweaver_graph/embedder.py, _embed() and create_batch(), historical commit 1e61f206 — the state that produced the timeout. Docstring and interface comments omitted; executable statements unchanged.

async def _embed(texts: list[str]) -> list[list[float]]:
    async with httpx.AsyncClient(timeout=30.0) as client:
        resp = await client.post(
            EMBEDDING_SERVICE_URL,
            json={"model": EMBEDDING_MODEL, "input": texts},
        )
        resp.raise_for_status()
        return [d["embedding"] for d in resp.json()["data"]]

class LocalEmbedder(EmbedderClient):
    async def create_batch(self, input_data_list: list[str]) -> list[list[float]]:
        if not input_data_list:
            return []
        return await _embed(input_data_list)

Explain this code line by line

1. The thirty-second timeout is a client-side setting on a single shared HTTP client.
2. _embed posts one request whose body is the entire texts list — the code puts no bound on how large that list can be.
3. The July 1 report traces the failing call to Graphiti’s deduplication path, which asks the embedder for a large batch during a backfill.
4. create_batch forwards that whole list to _embed unchanged, so a large batch becomes one large request.
5. Against a contended GPU, that one request is the one that can outlast the client-side thirty seconds and fail the episode.

The proposed fix was to raise the configurable timeout and split the batch into bounded sub-batches so that a slow embedding request could finish without hitting the client ceiling.

The embedder proposal was concrete: split one large request into ordered smaller requests and make the timeout configurable. For a synthetic batch of 100 items with a limit of 32, the request sizes would be 32, 32, 32 and 4. That arithmetic is not a throughput benchmark. It explains the shape of the change and the ordering contract it needed to preserve.

Synthetic explanation of the reported sub-batch proposal, not an exact production implementation.

items = list(range(100))
batches = [items[start:start + 32] for start in range(0, len(items), 32)]
assert [len(batch) for batch in batches] == [32, 32, 32, 4]
assert [item for batch in batches for item in batch] == items

Explain this code line by line

1. items stands in for a hundred work units that must be embedded in order.
2. The list comprehension slices them into chunks of at most 32, preserving order.
3. The first assertion checks the request sizes a real batching implementation would produce: 32, 32, 32 and 4.
4. The second assertion checks that concatenating the sub-batches reproduces the original sequence exactly — no item lost, reordered, or duplicated.

The first assertion checks request sizes. The second checks that reconstruction preserves every item’s position. Without that second contract, a faster or more resilient embedder could quietly associate vectors with the wrong inputs — a failure that would look like success and corrupt semantic matching downstream.

Cause B: the extraction response that was not truncation

The second cause was harder. The extraction response from the Qwen3-8B model was structurally invalid JSON despite the schema enforcement and the token cap. The error appeared at a specific character position deep in the response, suggesting a malformed structure rather than a simple truncation.

The first guess was truncation: a large response cut off mid-way would produce exactly this kind of parse failure. The evidence did not match that guess. A truncated response fails at the end of the payload, where the cut lands. This error position meant something inside the payload itself was malformed — the structured-output instructions and the model’s actual output were interacting in a way the parser could not accept.

The open question was whether structured-output enforcement was actually taking effect on the production request path, not just in isolated probes. The answer required inspecting the actual request that the extraction client sent to the inference server. The July 2 follow-up’s corrected extraction client and explicit token-limit forwarding is the resolution of exactly that question: the request path was wrong in a way that only became visible once the exception text was being logged properly.

Why the token cap never reached the model

The retained source shows how the token cap was supposed to travel. A helper reads GRAPHITI_MAX_TOKENS from the environment with explicit validation, the configuration object carries it, and the client is constructed with that configuration plus the same value passed again as its own parameter:

Exact fragment, dedented: packages/rootweaver-graph/src/rootweaver_graph/graphiti_client.py, client construction, historical commit 1e61f206 — the state the July 2 fix replaced. Intervening explanatory comments omitted; executable statements unchanged.

graphiti_max_tokens = _graphiti_max_tokens_from_env()
llm_config = LLMConfig(
    api_key="not-needed",
    base_url=vllm_base_url,
    model=llm_model,
    small_model=llm_model,
    max_tokens=graphiti_max_tokens,
)
llm_client = GuidedJSONOpenAIClient(
    config=llm_config,
    max_tokens=graphiti_max_tokens,
)

Explain this code line by line

1. The environment helper reads GRAPHITI_MAX_TOKENS, rejecting empty, non-integer and non-positive values with a logged warning instead of a silent fallback.
2. The validated value lands in the LLMConfig object that the Graphiti library receives.
3. The same value is passed a second time as the constructor’s own max_tokens argument.
4. That double passing is deliberate. The July 2 follow-up reports that the corrected generic client’s initialiser reads the configuration value and then overwrites its token attribute with its own parameter default of 16384 — so an environment-configured cap would have been silently dead unless it was also forwarded explicitly.
5. The historical construction targets the guided-JSON subclass. The corrected client switched to the generic chat path with the token limit forwarded explicitly, and the guided subclass remained as deliberately orphaned code for a later cleanup.

The override explains why a configuration that looked correct could still leave the real request carrying a different value than the operator set. It is the same shape of lesson as the empty log: the outer system held one value while the actual request path carried another. Verifying that parameter forwarding reaches the request is now a standing check rather than an assumption.

Fixes still needed a stable opportunity to run

The July 2 follow-up reports the batching and timeout change merged, alongside a corrected extraction client that forwarded the token limit explicitly. It describes a successful single-document probe: total nodes moved from eleven to twelve and completion markers from three to four, without the earlier JSON error in that probe.

Those total-node observations are not directly comparable to the earlier Entity-only count. They measure different scopes. I have not retained the raw original probe output here, so the numbers remain attributed to that handover rather than presented as newly measured results.

Each code fix also had to travel a fixed path before it could run against real documents: the change merges to main, a container image builds, the CronJob’s image reference is updated through the GitOps repository, and the controller reconciles the cluster. Only then does a new backfill run against the fix. The July 1 record describes this as roughly a forty-minute loop per fix, end to end.

That cadence shaped the work. A fix that cannot be tested locally has to survive a deployment loop before it can be evaluated, which means the diagnosis has to be as precise as possible before each loop starts.

The full backfill still did not complete. The same follow-up describes the model service cycling under the GPU-priority arrangement, creating connection failures. That is another layer: code can handle the request correctly when a dependency is available and still fail when that dependency is deliberately removed.

The important change was learning to name those layers separately. Visibility exposed timeout and format problems. The follow-up supplied bounded evidence for code fixes. Stable dependency access and full-corpus completion remained work to do.

The graph would only become a useful memory if successful items and failed items stayed distinguishable across retries. More visible progress was useful; it was not permission to forget the missing part.

Different failures need separate explanations, and unfinished items remain visible for retries. Conceptual artwork; placement pending review.

The retries were safe by construction

One design in the June 30 record was worth naming: content-hash idempotency. A document’s completion marker is written only when the ingestion of that document succeeds. A document that fails leaves no marker. A later run reprocesses exactly the unmarked documents and skips the marked ones.

That gives retries a property that is easy to lose: they make progress without making garbage. A failed document does not leave half-written entities behind. The graph fills cumulatively across runs, and the set of documents that still need work is exactly the set without markers.

Empty logs are a symptom too

One more lesson from the July 2 record: the ingestion job’s own logging handler was never configured at the entry point. Even after the client library’s exception logging was fixed, the job’s container logs stayed empty on failure. The historical record describes the cost: a stalled backfill that looked like a hang until a manual probe revealed the real error. The follow-up filed the entry-point logging fix as its own small work item, to be done before the next backfill so that failures are visible — and the handler work was not completed inside this episode’s window.

What I would carry into the next run

I do not need to make the historical result look cleaner than it was. It was progress from a poorly explained failure to specific code changes and a better understanding of the remaining dependency problem.

Lesson	Practical response
A count is not a corpus	Track selected, attempted, completed and failed items separately
Silent failure is a design flaw	Configure logging at the entry point before the first run
Client library defaults can override config	Verify parameter forwarding reaches the actual request
Batching has a latency ceiling	Bound batch sizes and make timeouts configurable
Schema enforcement must be verified live	Inspect the production request path, not just probes
Retry markers are the ledger	Only mark completion on genuine success
Deployment lag is real	A fix on main is not a fix in the running container
Diagnosis precision beats fix speed	Name the layer before writing code for it
Model contention is a shared-GPU reality	Budget for interference between workloads
Intent and validation are different gates	A deployed system is not a working one

Next: the proposed second chapter follows a smaller model into the same platform. Its fit on the GPU mattered, but so did the benchmark that justified it and the consumers still expecting an old model name.

Terms and references

Glossary — plain-English definitions

Knowledge graph

A representation of entities and their relationships, built so that queries can traverse connections rather than only match text.

Hybrid search

Combining multiple retrieval signals — vector similarity, keyword matching and graph traversal — into one merged result set.

Embedding

A numeric representation of text used for semantic matching; producing one has latency and ordering contracts.

Structured output

Constraining a language model's response to a schema so downstream code can parse it reliably.

Episode

Graphiti's unit of ingestion: one attempt to add one document's content to the graph.

Traceback

The recorded chain of calls leading to an exception.

Per-item outcome

The success or failure of one selected document, separate from a whole-job exit code.

Retry marker

A record that a document has been successfully processed; its absence is what identifies work still to do.

vLLM

A self-hosted inference server exposing language models behind an OpenAI-compatible HTTP interface; its request contracts had to be verified on the live path, not only in probes.

GitOps

Managing cluster configuration by committing desired state to a repository and letting a controller reconcile the running system against it — so a merged fix still has to travel through an image build and a repository update before it runs.

CronJob

A scheduled job object in Kubernetes; this episode's ingestion ran as manually triggered one-off jobs while the schedule itself stayed suspended.

Content-hash idempotency

Marking each document's completion by a hash of its content, so retries reprocess only unmarked documents and leave successful ones alone.

Primary reference: Graphiti by Zep.

How this episode was reconstructed

No previous Season 5 map or pre-crafted opener was recovered in scoped searches; the title and numbering are new editorial proposals. Core July 1, June 30 setup and July 2 follow-up are explicitly declared. Raw 46KB response, original pod logs, complete corpus and raw probe outputs were not independently replayed. Larger graph/embedding/hackathon work is outside this chapter.

No original native conversation replay or complete period inventory is claimed. Sources are scoped by named historical handovers, ADRs, reports and Git commits. Relevance search is not exhaustive coverage. Raw credentials, account identifiers, and Habi content are absent from the public draft. Current external documentation is explanatory and not a historical deployment observation.