The Ops Day
The platform could tell me something was wrong. The harder question was what happened between that signal and my first useful action.
By Saturday 27 June, Rootweaver had accumulated alerts, dashboards, bots and several ways to reach them. Some of the operating surface still came from an earlier life: the Discord server had been a streaming server. Cost digests and operational messages competed for attention. A receiver still pointed at a namespace that no longer existed. Buttons promised useful actions, but each action needed its own authority, state and failure handling.
Signals need distinct destinations before the next human action is clear. Conceptual artwork, not a historical photograph.
I wanted a shorter path from seeing a problem to understanding what I could safely do. I set the direction, reviewed the choices and confirmed the device experience; coding agents assisted with implementation and investigation. The day’s records show that improvement coming through several corrections, rather than one clean deployment.
This reconstruction follows the retained session handovers, implementation reports, ADR and historical source. The generated daily journal is a discovery aid, not a complete record of this work: its summary does not capture this operations arc adequately. Where an outcome survives as a historical report, I say so. Inspecting the code now establishes what that code does; it cannot recreate every observation from the day.
First, make an alert’s destination mean something
The initial problem was attention. A budget digest and a firing service alert both deserved to reach me, but they did not ask for the same response. Putting them together meant doing the classification myself every time I opened the channel.
The Discord cleanup separated critical alerts, warnings and the default alert stream. Cost reporting moved to its own channel; formatting tests acquired a test channel. The morning handover reports that work complete while still describing the interactive relay as a specification awaiting review. That ordering matters. The first improvement was making messages easier to recognise; the buttons came later.
| Signal | Destination | What I should understand immediately |
|---|---|---|
| Critical alert | Critical channel | A higher-priority operational signal |
| Warning | Warning channel | An issue to investigate without treating all messages alike |
| Unmatched severity | Default alert channel | A fallback destination rather than a discarded message |
| Cost digest | Cost channel | Information about budget and pace |
| Formatting test | Test channel | A controlled check of the message presentation |
ADR-087 records my request for colour and severity separation. The first implementation used the masgustavos/alertmanager-discord fork, built from source and hosted in the private registry. The channel configuration moved into a SealedSecret. This introduced a maintenance obligation: a self-built image needed its own rebuilds. Moving a webhook out of the manifest improved that configuration; it did not prove that every earlier copy of the secret had disappeared.
The Prometheus project’s Alertmanager documentation describes the routing, grouping and silence mechanisms. My installation still needed to prove its own wiring. A sensible routing table was only useful if the receiver existed and the message arrived in the intended place.
The handover distinguishes two test paths. A direct Discord test checked the bot and formatting. Its pipeline mode sent a synthetic alert through Alertmanager and the forwarder. The distinction is practical: the first could render a perfect message while a receiver upstream remained broken.
That was exactly the sort of broken wire found in the SRE receiver. It still named the old rootweaver namespace after the services had moved into the rw-* namespaces. The retained records report correcting the destination and observing an HTTP 200 from the collector. That supports the receiver test. It does not demonstrate that a downstream model completed an investigation.
The relay had to own the interaction
Once the alert had a sensible destination, I wanted to act from the phone where I saw it. Opening another tool, reconstructing the labels and building a silence request was friction at the wrong moment.
The design chose a small HTTP interaction service rather than reviving a persistent gateway bot. Discord would send a signed request when I pressed a button; the relay would verify it and dispatch the action. The plan tied this choice to avoiding the resource profile of the earlier gateway approach. It is a recorded design rationale, not a measured memory benchmark in this episode.
The review decisions were specific: both one-hour and twenty-four-hour Silence buttons, a graceful Triage fallback, and a dedicated interactions hostname. The new service replaced the earlier forwarder because it needed to post app-owned messages and handle their controls. Severity routing remained; the message now had a return path.
Discord’s interactions documentation explains this model. A button click creates an interaction for the application. In Rootweaver, that request reached /interactions through the tunnel, then passed signature verification before the handler interpreted the action.
One alert, three different action paths
Three different contracts sat behind those buttons. Ack updated the message. Silence requested a bounded suppression of matching notifications. Triage requested an assessment, or returned the configured fallback. Treating all three as “the bot handled it” would hide the important differences.
Follow one warning through the state and authority checks
Consider a fictional warning about a worker. This is an explanatory input, not a recovered production alert or a full Discord payload.
Synthetic Alertmanager-style input. Only the fields needed for this explanation are shown.
{
"status": "firing",
"fingerprint": "example-warning-01",
"labels": {
"alertname": "WorkerQueueHigh",
"severity": "warning",
"service": "demo-worker"
}
}The severity-specific webhook path selects a channel. The handler skips resolved alerts and alerts without a fingerprint. After a successful Discord post, it stores the fingerprint with the labels and message context. That last ordering is visible in the historical source: the state entry is written after the message call succeeds.
The fingerprint connects a later click to those stored labels. A button identifier carries an action, a parameter and that fingerprint. For example, silence:1h:example-warning-01 means “request a one-hour silence for the context associated with this alert.” It is not an arbitrary command to run against the cluster.
State lived in memory. The design specified an eight-hour lifetime, and the inspected AlertStore sweeps expired entries on reads and writes. Restarting the process loses the map. The interaction handler rejects Ack and Silence when their context has expired and points the user towards Grafana or Alertmanager. Triage is handled differently: its code can proceed with empty stored labels and annotations recovered from the message. The controls therefore do not all have identical context requirements.
For Silence, a valid Discord signature is only the first check. It identifies the request’s authenticated delivery path; the handler still compares the acting user with the configured owner.
Exact fragment, dedented for reading: services/rw-alert-relay/app/main.py, interactions(), historical commit e81e21aa. Credentials and owner identifiers are omitted.
if action == "silence":
if user_id != settings.ryan_discord_id:
return _ephemeral("not authorized to silence alerts.")
try:
payload = build_silence(ctx.labels, param, created_by=f"discord:{user_id}")
sid = await alertmanager.create_silence(payload)Explain this code line by line
- The first condition selects Silence after the earlier signature, identifier and context checks.
- The user comparison checks who clicked. Receiving a signed Discord request is not sufficient permission to silence an alert.
- An unauthorised user receives a private response; the silence API is not called by this branch.
- The try block begins the operation that can fail independently of Discord’s request validation.
- build_silence uses the stored alert labels and chosen duration. created_by records the acting Discord user.
- create_silence calls Alertmanager and returns its silence identifier. This requests suppression of notifications; it does not repair the worker.
The inspected builder permits 1h and 24h, sorts the stored labels into equality matchers, excludes __name__, and rejects an empty usable matcher set. Each matcher says which alert label must equal which value. That limits the request to the alert context rather than silencing every warning.
Here is the shape produced for the fictional warning, using a fixed example time and a synthetic attribution identifier. This is not a recorded API response.
Synthetic expected output from build_silence() in services/rw-alert-relay/app/alertmanager.py, historical commit 40ce19e9. Example time and user are invented.
{
"matchers": [
{"name": "alertname", "value": "WorkerQueueHigh", "isRegex": false, "isEqual": true},
{"name": "service", "value": "demo-worker", "isRegex": false, "isEqual": true},
{"name": "severity", "value": "warning", "isRegex": false, "isEqual": true}
],
"startsAt": "2026-06-27T12:00:00+00:00",
"endsAt": "2026-06-27T13:00:00+00:00",
"createdBy": "discord:example-owner",
"comment": "silenced via Discord for 1h"
}This makes the boundary concrete. The interface removes JSON assembly from the human action, but the code still needs identity, retained labels and a bounded duration. The deployment did not grant the relay permission to restart a workload.
What the smoke test established, and what it did not
The later implementation report records thirty passing unit tests and a deployment of the relay’s second image. The retained GitOps commit records the receiver migration. Those are useful stages, but the report’s own smoke-test table is more precise than its “live end-to-end” headline.
| Path | Historical reported result | Evidence boundary |
|---|---|---|
| Webhook to Discord | Warning embed and five buttons rendered | One reported delivery smoke test |
| Signed registration PING | Endpoint validated and responded | Signature-verification path exercised |
| Ack | Footer changed and button became disabled in Discord’s API | Message state changed; service recovery not tested |
| Triage | Deferred response followed by graceful fallback | No live model assessment established |
| Silence | Not tested live | Inspected implementation is not a successful live API call |
Ack also produced a misleading observation: the mobile client appeared not to grey out the button. The report says the Discord message API showed disabled=true. That is a useful debugging distinction. The client view and the retained message state were different observations, so an apparently unchanged button did not automatically mean the handler had failed.
Triage’s initial deployment left the SRE-brain URL unset. The inspected client returns a fallback in that state. Although the wording says “queued,” the source shown here returns a message; that wording is not evidence of a durable queue or a later completed assessment. With a configured endpoint, the client has a separate HTTP path and marks the model’s assessment as display-only content referring to untrusted alert input.
The first version also left operational questions open. Its readiness route examined minimum configuration, not the health of every upstream dependency. Its posting handler caught a Discord failure, continued, and returned the number posted. A comment about redelivery does not itself establish reliable retries: the source does not propagate every failed post as a failing webhook response. I would want the delivery contract checked explicitly before relying on that behaviour during an incident.
A dashboard link could lose the whole alert
One of the most useful failures came from an optional convenience. The initial code used the alert’s generatorURL for the Grafana button. That URL could actually be a long Prometheus expression link.
The historical fix records that a live smoke test hit Discord’s 512-character button-URL limit. The message was rejected with HTTP 400. The failure affected the entire alert message, not just the link button. I inspected the code change; the original HTTP exchange survives here as a reported observation in the commit and implementation report.
The correction had two parts. The handler used the configured Grafana base URL instead of the alert’s generated query URL. The component builder also checked length and omitted an invalid link. An optional dashboard shortcut could then disappear without taking the alert with it.
Exact condition from services/rw-alert-relay/app/discord.py, build_components(), historical commit 40ce19e9. The constant is 512 in that source.
if grafana_url and len(grafana_url) <= _BUTTON_URL_MAX:Explain this code line by line
- grafana_url must contain a value before the builder considers adding the link button.
- len(grafana_url) checks its character count against the source’s configured maximum.
- Only a passing value enters the following button-construction block. Other alert controls remain in the row.
- This is one guard against the observed URL-length failure. It does not validate every possible message-format error or prove reliable delivery.
The design lesson is specific: keep optional presentation features from invalidating the essential signal. A useful control surface needs failure behaviour as carefully considered as its successful path.
The dashboard’s identity boundary needed another pass
The place an alert sends me also matters. The same day’s Grafana report describes removing a second local login after Cloudflare Access had authenticated the user through Google. Grafana would consume the identity header injected by the access proxy.
Grafana Labs documents its auth-proxy mechanism, including the source-address whitelist intended to reduce header spoofing. The local report shows why configuring that option was not enough for this network path.
The legitimate tunnel-to-NodePort hop appeared to Grafana as the node’s bridge gateway address. The first configuration trusted that address. A later security review found that a wired LAN request could be rewritten to the same source address by Kubernetes networking. A request carrying a forged identity header could therefore look as though it came through the trusted proxy. The source-address check could not distinguish the paths it was meant to distinguish.
The report records a node-level restriction on direct access to that Grafana NodePort, applied before destination translation, plus moving the local administrator password into a SealedSecret. It also records correcting the emergency login procedure because the database no longer contained the default admin user. These are reported repairs and HTTP checks, not a fresh audit of today’s installation or an independently replayed historical exploit.
This belongs beside the relay work because it changes what “open the dashboard” means. An authenticated header is useful only when the route delivering it preserves the intended trust boundary. Successful sign-in alone would not have shown that the bypass path was closed.
The communications failure looked like a client problem
Matrix then exposed a different misleading signal. One device or connection worked while another received Cloudflare 502 errors. That encouraged client-side explanations: cookies, browser transport, authentication, address family or the home router.
The incident report preserves the tests that narrowed those possibilities. Its wording is a historical diagnosis, not a universal rule about every Cloudflare 502.
| Suspected cause | Test recorded in the incident report | What it narrowed |
|---|---|---|
| Oversized access cookie | A private browsing session still failed | The existing cookie was not required to reproduce the failure |
| Client HTTP/3 behaviour | Forcing the phone to HTTP/2 still failed | Changing client transport did not remove the symptom |
| Authentication redirect | An unauthenticated Matrix API route worked from the unaffected Mac | That route was not uniformly blocked by the identity layer |
| IPv6 versus IPv4 | Both failed from an affected Mac | Selecting one address family did not fix that path |
| Home router | A desktop on the same router still reached the service | The router was not a sufficient explanation for every client outcome |
The more decisive comparison separated the origin from the tunnel. A direct request to Synapse returned HTTP 200 while affected requests through the public hostname failed. The report records repeated EOF errors associated with one connector connection and attributes the outage to a wedged HTTP/2 tunnel connection. That was stronger evidence than the symptom “the phone is broken.”
An edge-served trace page had also returned HTTP 200. That result did not exercise the application behind the tunnel. It could establish something about reaching Cloudflare’s edge while saying little about reaching Synapse. Choosing the right probe changed the diagnosis.
The first connector cycle did not produce a clean recovery. The report says the failing path shifted between client networks during the restart and transport change. A second connector was then added, and the recorded home-broadband origin-path check changed from thirty failures to thirty HTTP 200 responses. The handover also records my confirmation that the phone and Mac reconnected. I have not recovered and replayed the original request logs.
Both connectors ran on the same desktop. That reduced reliance on one connector process and allowed one-at-a-time restarts. It did not remove the host, power or local network as shared dependencies. A second-node connector was explicitly deferred.
Two routes can still depend on one shared place; two connectors on one desktop do not remove host failure. Conceptual artwork, not a historical photograph.
Cloudflare’s tunnel availability guidance supplies upstream context for replica connections. It does not independently establish the cause or repair of this June outage. The lesson I carry from the retained evidence is to compare an affected client route, an unaffected route and the origin before declaring where the fault lives.
The documentation loop was reading its own stale baseline
The day’s other feedback loop lived in the architecture documentation. The handover reports that the index still said thirty-three namespaces when the observed cluster had forty-two. It also retained four StatefulSets under the old rootweaver namespace after that monolithic namespace had gone.
Updating the detailed modules had not corrected the numbers the drift checker actually read. The checker parsed the index’s Quick Stats. The updater edited the modules. The next run compared the cluster with the same old index and rediscovered drift. The handover says that stale result could force another full model-assisted documentation run each night.
The historical drift_check.py change made the reader and writer share a table of field names, file patterns and numeric matches. The new writer updated just the matched number. It skipped unavailable measurements, missing matches and already-correct values. This was a narrow deterministic repair, not a model rewriting arbitrary architecture prose.
Exact fragment, dedented: scripts/drift_check.py, sync_doc_baseline(), historical commit 0ae5cd96. Earlier guards have already selected a file, match and non-null target.
current = int(m.group("n"))
if current == int(target):
continue
s, e = m.span("n")
new_text = text[:s] + str(int(target)) + text[e:]
if not dry_run:
path.write_text(new_text, encoding="utf-8")Explain this code line by line
- Read the number captured by the named group n, rather than treating the whole paragraph as editable content.
- Compare it with the measured target.
- Skip the write when they already match. Repeating the operation should not keep changing the file.
- Find the exact start and end of that numeric substring.
- Keep the text before and after it and substitute only the new number.
- Respect dry_run so a caller can inspect proposed changes without writing.
- Write the bounded replacement only when a real update is requested. This does not validate the accuracy of unrelated architecture prose.
For the recorded namespace discrepancy, the intended transformation was simply thirty-three to forty-two in the parsed field. That fixes the feedback loop’s reference point. It does not prove that every topology drawing, service description or stored source count is current.
There was another wrong turn in the same audit. The initial explanation for sync conflicts was two documentation runs racing. The handover says the differing timestamp formats and YAML ordering instead identified two different writers: the documentation updater and the NER enrichment service, operating on synchronised replicas.
The reported repair excluded the architecture directories from that enrichment path. The handover records forty-five tests and a merged baseline-repair change, plus a separately deployed enrichment exclusion. It still left a wider file-watcher debounce task and an empirical check of the next scheduled run outstanding. The richer Excalidraw topology also remained stale after the simpler topology was updated. Those open items prevent “the docs healed themselves” from becoming a claim that all documentation was now correct.
What the ops day actually changed
These changes share a practical thread. They reduced ambiguity between a signal and the next decision: what channel it belongs in, which action is permitted, which route a probe exercises, and which baseline an automated check compares.
| Lesson | Check I would carry into the next build |
|---|---|
| Presentation can hide delivery failure | Exercise the complete alert route, including optional links |
| A signed interaction still needs authority | Check the actor, action and retained context separately |
| A button label can overstate the backend | Inspect what “queued,” “acknowledged” and “ready” actually mean |
| One successful route does not cover another | Compare direct origin, public path and affected clients |
| Two processes may share one failure domain | Identify the host and network dependencies beneath them |
| A feedback loop needs the right reference | Update the baseline the checker reads, not just nearby documentation |
The platform had clearer alert destinations and a smaller human action from the message. The relay had reported working delivery, signature and Ack checks, alongside a Triage fallback and an untested live Silence path. Grafana’s access work needed a second hardening pass. Matrix’s reported recovery improved connector resilience while leaving host redundancy for later. The documentation change repaired a specific recurring baseline mismatch and exposed a separate ownership conflict.
That is enough progress to explain without calling the system autonomous. The human still had to choose what to investigate and verify that the underlying service recovered. Making that boundary clearer was the useful part of the day.
Next: The Self-Healing Turn follows the next question: what context a model should receive, how its assessment leaves the system, and which authority should remain with me as triage gains capability.
Terms and references
Glossary — plain-English definitions
Alertmanager
The Prometheus component that groups, routes and silences notifications; it does not repair the affected service.
Fingerprint
An alert identifier used here to retrieve stored labels and message context.
Silence
A time-bounded rule suppressing matching alert notifications.
TTL
Time to live: how long retained context remains usable before it expires.
Ed25519
A digital signature scheme used to verify Discord interaction requests.
Auth proxy
A trusted intermediary that supplies an authenticated identity to the application.
SNAT
Source network address translation: rewriting the source address seen by the receiving service.
Failure domain
A shared dependency whose failure can affect apparently separate routes.
Origin path
A request reaching the application behind the tunnel, rather than only the edge.
Drift baseline
The documented reference values that a checker compares with observations.
Idempotent update
An operation that makes no additional change when repeated against the same desired state.
NER
Named entity recognition: enrichment that identifies terms in documents; a separate document writer in the recorded conflict.
Official references: Prometheus Alertmanager, Discord interactions, Grafana auth proxy and Cloudflare tunnel availability.
How this episode was reconstructed
This is a curated reconstruction of Rootweaver operations on 27 June 2026. The generated technical log and reflection were checked for coverage; they do not adequately cover this arc and were not used to substantiate its operational claims. Dated session handovers, the routing ADR, relay and Grafana implementation reports, the Matrix incident report and inspected historical commits provide the evidence. No complete native-session replay, exhaustive day inventory or fresh production audit is claimed. Historical reports of tests remain distinct from inspected source and new synthetic examples. Some records use broader completion language than their check tables support; this draft preserves the narrower results.
The original pre-crafted chapter is retained unchanged for comparison. Provider-usage accounting is not repeated as a second central story after the earlier season’s usage chapter. Removal of an unused public ingress is not asserted as verified here because a separate supporting implementation record was not recovered in this scoped pass. The wider day also contains provenance, ingestion and quality work outside this operations narrative. Private personal-project material, financial details and credentials are excluded. Current primary documentation explains upstream mechanisms; it does not prove the June deployment. Private evidence snapshots and claim mappings remain outside public site assets.