The Second Restructure — Cutover Waves + the Hero Bug

By the middle of June, Rootweaver’s application namespace had become a poor description of the system inside it. Retrieval, memory, indexing, journals, frontends and operations jobs all lived in the same bucket. I wanted boundaries that made the platform easier to reason about and gave each function its own resource and access controls. Moving the workloads meant finding the dependencies hidden outside them.

Functional boundaries still need working connections. Conceptual illustration.

The previous episode followed the agent’s memory-maintenance loop. This episode returns to June 16–18 and the infrastructure underneath those workflows. The earlier restructure had changed the codebase. This one changed where its services ran, which names they resolved and which connections the cluster would admit.

I directed and reviewed the work, with coding agents assisting with implementation and investigation. This account draws on the decisions, reports and GitOps history retained from that week. The source note at the end explains the reconstruction’s limits.

Choose boundaries before moving workloads

ADR-080, Cluster-Wide Namespace Taxonomy and rootweaver Decomposition, recorded roughly 100 pods in the old namespace: about 29 Deployments, four StatefulSets, 20 CronJobs and 28 Services. The problem was more than the length of the pod list. Unrelated functions shared an operational boundary that had grown by default.

A namespace gives Kubernetes a scope for names, permissions and resource accounting. It does not provide network isolation merely by existing. For that, the selected pods need appropriate NetworkPolicies and a network implementation that enforces them. I wanted the resulting boundaries to carry operational meaning, rather than just make a long pod list look tidier.

The recorded decision rejected a coarse four-namespace split in favour of per-function boundaries. Stable infrastructure namespaces would retain their names and receive classification labels; recreating them solely to impose a naming scheme would add risk. The application plane would move into eleven new namespaces.

Boundary	Function
rw-rag, rw-embed	Retrieval serving and embeddings
rw-memory, rw-mcp	Agent memory and tool gateways
rw-ingest, rw-codeintel	Corpus indexing and code intelligence
rw-journal, rw-sre	Journal processing and operations
rw-frontend, rw-platform	User interfaces and shared application utilities
rw-data	The five stateful stores, moved in a later wave

The first uncertainty was whether that design required a larger network migration. The June 16 technical journal still repeated the earlier assumption that kube-router could not enforce the required cross-namespace selectors and that a broad pod-network address rule was the interim answer. The updated ADR records a different result: combined namespace and pod selectors worked on the then-running k3s version, including a test with client and server deliberately placed on different nodes.

Those agent-reported tests changed the plan. Namespace decomposition could proceed on the existing CNI, while Cilium became optional later work. That kept the migration focused on the application’s boundaries instead of making it depend on replacing the cluster’s network layer.

Prepare the route, then pilot the move

The migration was staged to keep rollback manageable. Groundwork came first, a frontend pilot followed, the application services moved next, and the stores came last. An application could move while continuing to reach a database in the old namespace. Moving a database introduced backup, restore and caller-cutover work that deserved its own stage.

Three pieces of preparation made that sequence practical. A reusable namespace template supplied quota, resource defaults, RBAC and base network policy. Explicit service names removed the assumption that every dependency lived beside its caller. A reflector arrangement made the registry pull secret available in the new namespaces without treating a Secret in one namespace as a cluster-wide credential.

Consider an address such as tei-embedding:8090. The short name is resolved relative to the caller’s namespace. After an indexing pod moves to rw-ingest, that address no longer identifies a service remaining elsewhere. tei-embedding.rw-embed.svc.cluster.local:8090 makes the intended receiver explicit. Correct DNS gets the request to the right destination; it does not grant permission to connect.

The frontend pilot also changed assumptions before the larger move. Its June 17 implementation report says the console’s upstream addresses were already explicit enough to avoid rebuilding the image for that step. But the expected secrets were plain runtime Secrets rather than existing sealed manifests ready to copy. The report also records desktop-local mounts that required deliberate placement, and a suspended source Flux Kustomization that made the cutover a coordinated resume window.

The report captured preparation, before cutover. Once the manifests passed Kustomize checks, the next step was the coordinated move into rw-frontend: bring up the target, prune the old copies during the source’s resume window, then check the moved services.

Stage	What changed	What stayed behind
Groundwork	Labels, templates, explicit service addresses and pull-secret reflection	Workloads remained in their original namespaces
Frontend pilot	Four interfaces prepared for rw-frontend, with a coordinated source-prune window	The rest of the application plane
Application plane	Services moved into nine functional destinations	The five stores remained in rootweaver for this wave
Stateful wave	Backup, restore and caller cutovers for rw-data	Detailed in the next episode

The receiver remembered the old namespace

Wednesday’s application-plane moves followed the template, but the dependencies were not identical. The indexers moved into rw-ingest; the embedding service had already moved into rw-embed. The callers’ outbound policy had been updated. The receiver’s inbound allow-list still selected those callers in rootweaver.

The caller moved; the receiving gate still expected the old identity. Conceptual illustration.

This is the central defect in commit 96fd7669. A combined peer with a namespaceSelector and a podSelector means that both conditions must hold for that peer. The application label survived the move. Its namespace did not. A rule that previously admitted the indexer therefore stopped matching it.

        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: rootweaver
          podSelector:
            matchLabels:
              app: vault-delta-indexer

Explain this code line by line

1. Line 1 begins one allowed peer in the receiver’s inbound list.
2. Line 2 introduces the namespace label conditions.
3. Line 3 selects the rootweaver namespace; this is the value changed by the repair.
4. Line 4 adds a pod selector within the same peer, making the namespace and pod conditions apply together.
5. Line 5 introduces the pod label conditions.
6. Line 6 selects pods carrying app: vault-delta-indexer. The excerpt is a peer fragment, not a complete standalone policy.

The indentation carries the mechanism. These selectors are part of one peer entry, so the rule admits the selected application within the selected namespace. Separating them into different list entries would describe a different, broader set of peers. Changing a value while preserving that structure is a small edit with a large effect on who can connect.

The repair changed the consumer’s namespace in the receiver’s allow-list. The corresponding excerpt after the change is:

        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: rw-ingest
          podSelector:
            matchLabels:
              app: vault-delta-indexer

Explain this code line by line

1. Line 1 begins one allowed peer in the receiver’s inbound list.
2. Line 2 introduces the namespace label conditions.
3. Line 3 selects the rw-ingest namespace; this is the value changed by the repair.
4. Line 4 adds a pod selector within the same peer, making the namespace and pod conditions apply together.
5. Line 5 introduces the pod label conditions.
6. Line 6 selects pods carrying app: vault-delta-indexer. The excerpt is a peer fragment, not a complete standalone policy.

The repair updated three moved consumers and added dlq-replay-connector-docs, a fourth consumer that already had outbound access configured but had never appeared in the receiver’s list. Checking the inbound rule exposed a pre-existing gap alongside the migration regression.

Pause flow

The commit message reports an unreachable embedding URL, a connection error, zero processed items and a failed item. It also reports that the Kafka offset did not advance, so the next KEDA-spawned job received the same work again. Four firing lag or unconsumed-topic alerts are recorded in that message. Those observations explain how an indexing failure could continue behind an otherwise plausible deployment view: restarting the consumer did not remove the condition that prevented it from finishing.

The ingest cutover and policy-repair commits are about two hours and 47 minutes apart. That brackets the changes in the source history; the exact runtime recovery time is not retained. The important failure mechanism is clear in the recorded diagnosis: new jobs kept encountering the same closed path to the embedding service.

The practical rule is to treat a moved workload as a change to both ends of every connection. Search for the caller in receiver policies, including receivers moved in earlier waves. A dependency matrix helps identify the edges; reading the complete receiver policy tells you what those edges actually allow.

The first diagnosis was still too narrow

The same day supplied a stronger lesson than a single missed edit. When MLflow moved to rw-platform, it could not reach research-postgres. An initial attempt widened its outbound rule to a pod-network address range. The later correction’s commit message explicitly says that attempt did not help.

The receiver was the problem again. Research-postgres’s allow-list was bundled in the SRE agent’s policy file. It admitted MLflow through a same-namespace pod selector. Once MLflow moved, that selector no longer described the caller. The historical correction at 9f9e3182 changes the receiver to an explicit namespace-and-pod peer and removes the broad outbound workaround.

The connection error had led the investigation toward the caller’s egress. The fix required reading the receiver’s whole policy, including its default-deny context and an allow-list stored somewhere less obvious. The correction’s commit message records that an earlier, truncated search had missed both. The investigation had to follow the dependency further than the first search result.

Other corrections made the cutover pattern more specific. Moved pods needed the labels their policies selected. MLflow needed more quota. A dormant PVC changed what Flux should wait for. Each move still required checking identity, capacity and readiness against the actual workload.

The nine application destinations did not move in nine identical steps. Platform utilities needed more than one change, and the SRE plane moved after the other services. A successful earlier move supplied a pattern to reuse, but the next workload could still have a different dependency.

Review ownership as well as connectivity

Network policy was only one kind of dependency. A ClusterRole is not namespaced, and neither is a ClusterRoleBinding. Moving a workload does not give a cluster-scoped object a new identity. If source and target Flux Kustomizations both refer to the same name, pruning the source can remove an object the target expects to own.

The SRE-plane migration commit records renaming cluster-scoped RBAC objects with the destination namespace in their names and checking for shared names between the rendered builds. The frontend pilot used the same precaution. That gave the source and target distinct object names, so pruning the old configuration would not target the new configuration’s role or binding.

Follow the connection beyond the moving workload. Conceptual illustration.

Fresh review mattered because the moving workload was only one place to look. The SRE migration touched receiver policies, monitoring routes, probe targets and console upstreams. A service could have its new address in application configuration while an allow-list or monitoring target still described the old boundary.

A further review catch came early on Thursday, after the store cutovers. Radar’s outbound rule still bundled two destinations under the old namespace. Commit d6696784 split the database connection into a rule targeting rw-data. The retained ledger attributes that catch to three read-only Codex review passes with different lenses.

That correction belongs after the database moves in the chronology, even though it reinforces this episode’s lesson. The caller’s configuration and its network rule could disagree about where the database lived. Moving another layer changed the connection again, and review had to follow it.

What the restructure established

The application-plane cutover turned the namespace plan into a set of concrete dependencies. The embedding receiver and research-postgres exposed the same weakness from different callers: updating the moving workload was only half the connection. The RBAC work added a separate ownership problem, because some resources did not have a namespace to move with them.

I would still choose functional boundaries and a staged migration. The changes made the dependencies easier to describe, but also made assumptions visible that the shared namespace had concealed. The runbook needed to capture those assumptions before the next move repeated them.

The lessons I would carry into another migration are practical:

• Describe each connection at both ends. Record the caller’s namespace, receiver, port, outbound rule and inbound rule. Update the receiver in the same reviewed change when a caller moves.
• Read the whole receiver policy. A broad outbound workaround can add permissions without fixing an inbound rejection. Follow a search hit into its full context before changing the rule.
• Keep source checks and runtime checks separate. Composition, reconciliation, startup and a successful dependency request answer different questions. Scheduled and dormant workloads need deliberate coverage.
• Give cluster-scoped resources explicit ownership. Namespace changes do not distinguish identically named ClusterRoles for two reconcilers.
• Use metrics that show completed work. In this case, a running consumer was less informative than advancing offsets and successful indexing. The historical alerts reported stalled progress, not a complete diagnosis.

I would keep the staged order. It separated application moves from state restoration and exposed assumptions while rollback was still comparatively manageable. I would strengthen the receiver discovery and end-to-end checks around it. The aim is a set of boundaries whose dependencies are visible enough to test, rather than a collection of new namespace names.

The stores followed the app plane later that Wednesday. Episode 13, “Fix One Layer, Expose the Next,” follows the stateful cutovers and the indexing failures that remained once connectivity was repaired.

Glossary · six terms and official references

Namespace

A scope for resource names and controls. Namespace creation alone does not isolate traffic. Official reference

NetworkPolicy

Rules selecting pods and allowed network peers. Inbound and outbound isolation are evaluated for their respective selected pods. Official reference

Combined selectors

A namespace selector and pod selector inside one peer select pods satisfying both conditions. Separate peer entries admit their alternatives. Official reference

FQDN

A fully qualified domain name makes the service namespace explicit; a short service name is resolved in the caller’s namespace. Official reference

Flux Kustomization

A Flux resource that composes and reconciles declared objects. Pruning removes previously managed objects absent from the desired configuration. Official reference

ScaledJob

KEDA’s mechanism for creating Jobs in response to event-source demand. Creating another Job does not itself repair a dependency that makes processing fail. Official reference

About this reconstruction

The core period is June 16–18, 2026, centred on Wednesday's application-plane migration. The Thursday radar review is marked as a retrospective. Sources are ADR-080, the frontend authoring report, bounded technical-journal entries, the retained June evidence ledger and selected historical GitOps manifests and messages. The source/claim ledger beside this draft records the inventory and omissions.

The five store migrations, namespace teardown and later indexing aftermath are reserved for Episode 13. Parallel budget and agent-harness work, personal reflections, employer material, credentials and native conversation bodies are outside this reconstruction. The pre-crafted Episode 12 remains unchanged for comparison. Conceptual artwork illustrates mechanisms; it is not a historical screenshot. This review did not independently rerun the historical tests or replay the complete migration.