Skip to content

A Green Run Didn't Clear the Alert

S5·E14 ·Rootweaver · Observability · Kubernetes · Reliability

Episode 13 ended with a repaired implementation and an experiment that still did not justify its rollout. The next investigation put a similar distinction into an operational setting. A quality-scoring job could finish successfully without satisfying the condition behind the alert that had brought it to my attention.

I needed a dependable daily quality signal, and a recovery check that tested the system actually responsible for producing it. The July investigation found a failed upstream read, a small retry budget and a misleadingly reassuring manual result. Recovering the scorer meant following those states through the application, the Job and its CronJob controller.

The engineer holds a working hand signal lamp inside a dark railway signal cabin while a separate mechanical dispatch register remains unattended.

A working hand signal does not establish that the scheduled dispatch system recorded a successful departure. Conceptual illustration.

This chapter covers 16 July 2026. It is a curated reconstruction from an investigation handover, a recovery report, and retained scorer, alert and GitOps source. I owned direction and review; agents assisted with investigation, implementation and verification. The live observations below remain attributed historical reports. The new examples inspect source and calculate illustrative states; they do not rerun the original jobs or reconstruct the complete native sessions.

Start with what the warning actually measured

The work was recorded as VW-1317 — QualityLens scorer retry and recovery. The alert concerned a scheduled pipeline that had stopped recording success. It was not a verdict that the model’s answers had suddenly become worse.

The historical rule compared two CronJob timestamps: its latest scheduling time and its latest recognised successful completion. The retained expression was:

alert-rules.yaml · DataPipelineCronJobNotSucceeding · GitOps de2cd074 · exact historical expression

(kube_cronjob_status_last_schedule_time{namespace=~"rootweaver|rw-.*"}
 - kube_cronjob_status_last_successful_time{namespace=~"rootweaver|rw-.*"}) > 14400
Explain this code line by line
  • The first metric supplies the CronJob’s most recent scheduling timestamp.
  • The second supplies its recorded successful-completion timestamp.
  • Subtraction produces a difference measured in seconds.
  • The namespace filters select the same platform namespace pattern on both sides.
  • The threshold is 14,400 seconds, or four hours. This expression compares the two recorded times; it is not the same calculation as the current time minus the last success.

The rule also used for: 1h. That required its condition to persist before the alert fired. The Prometheus authors’ alerting-rule documentation explains this pending period. It is a separate duration from the four-hour difference inside the expression.

This distinction gave the investigation a concrete acceptance question: had the CronJob’s success state advanced? A healthy service endpoint, a fresh application log and a successful standalone Job could each be useful observations without answering that question.

The scorer failed before it could assess an answer

Quality Lens read model generations from Langfuse, selected eligible outputs and sent them to a local judge. The intended output was a quality score attached to a generation. Fetching the generations was a dependency before any of that judging could happen.

The investigation handover attributes the morning failure to an unhandled httpcore.ReadTimeout while reading Langfuse observations. It also reports instability in langfuse-web, including repeated restarts. Those are the historical diagnosis and operational observations; this reconstruction has no original complete request trace with which to independently re-establish the upstream failure.

The retained Python makes the failure boundary visible. score_recent fetches pages through langfuse.api.observations.get_many. That read does not sit inside the per-generation judge-error handler. If it raises, execution can leave the scoring function before the normal completion log and subsequent metrics push.

A different failure path was deliberately softer: judge_generation caught a judge exception and returned no score, allowing the loop to count an error and continue. These behaviours belong to different stages. Handling a failed judge call did not automatically handle a failed attempt to fetch the work in the first place.

The historical Job had backoffLimit: 1, and the report says that budget was exhausted. I would not translate that field into a claim that exactly one container attempt occurred. The retained template used restartPolicy: OnFailure; Kubernetes counts retries in relation to both Job and container state. The Kubernetes Job documentation explains why the field needs that context.

The first debug copy changed the conditions

Before a successful debug run was obtained, the investigation encountered a second problem. Its copied workload needed the same pod label as the scorer: app: quality-lens-scorer.

That label selected the scorer’s permitted network egress. The namespace also had a default-deny policy and a separate DNS allowance. A copy could therefore resolve a hostname while still lacking permission to contact the service behind it.

network-policies.yaml · quality-lens-scorer-policy selector · GitOps de2cd074 · exact historical excerpt

podSelector:
  matchLabels:
    app: quality-lens-scorer
policyTypes:
  - Egress
Explain this code line by line
  • podSelector identifies the pods to which this policy applies.
  • matchLabels requires the specified key and value to be present.
  • app: quality-lens-scorer is the workload identity used by this selector.
  • Egress makes this a policy about outgoing connections. The following rules in the retained file permit the relevant service destinations.
  • This label determines policy selection. It does not declare which CronJob owns a Job.

The handover reports fast connection refusals from the incorrectly labelled probes, followed by a successful labelled debug job. It attributes the refusal to the network-policy path. There is no recovered packet capture here, and the symptom should not become a universal rule that policy denial always looks like connection refusal. Kubernetes’ NetworkPolicy reference describes the policy model; the historical response remains the handover’s observation.

The useful lesson was experimental: a copied workload must preserve the identity relevant to the boundary being tested. A failed probe with different access rules can diagnose the copy rather than the original failure.

In a subterranean pump room, the engineer checks a pressure gauge fitted to the correct brass connection while a similar unattached gauge rests nearby.

The test needs the correct connection and identity before its result can explain the production path. Conceptual illustration.

A label and an owner reference answered different questions

The labelled debug job exited successfully. That established that this probe could reach its dependencies and finish at that checkpoint. It did not make the probe a child of the CronJob whose status the alert watched.

A pod label can select network policy. A Job’s controller owner reference connects it to the parent controller. Similar names and matching labels are not a substitute for that relationship. The Kubernetes ownership documentation distinguishes ownership from label selection.

For an independently inspectable example, the upstream Kubernetes v1.34 CronJob controller selects Jobs through their controller reference and processes successful completion times for those children. This is a pinned explanatory source, not proof of the exact controller binary running in the July cluster.

Here is a synthetic summary of the manual-probe case, not an original Kubernetes response or a deployable manifest:

{
  "pod_label_matches_policy": true,
  "job_has_cronjob_controller_reference": false,
  "job_succeeded": true,
  "cronjob_success_record_advanced": false
}

The first field explains why the repaired probe could make its outgoing calls. The second explains why that successful probe did not establish parent-controller recovery. The final two fields can both be true without a contradiction: the application completed, while the CronJob’s recorded success remained unchanged.

Which success reaches the alert?

ownedunownedJob completesApplication resultOwner checkController reference?CronJob statusOwned success recordedAlert reads statusSchedule minus successManual probeParent stays unchangedManual success can prove an application path.It does not advance an unrelated parent record.
ownedunownedJob completesApplication resultOwner checkController reference?CronJob statusOwned success recordedAlert reads statusSchedule minus successManual probeParent stays unchanged
Simplified explanation of the ownership boundary. Moving dots show direction, not measured traffic or a replay of the incident.

Follow one failure through the alert calculation

Use relative hours to make the distinction concrete. Suppose the previous recognised success is at hour zero. The next scheduled run starts at hour 24 and fails. A manual debug job then succeeds at hour 27. Later, an owned verification job completes at hour 28.

Illustrative checkpointLast scheduleCronJob’s last successDifference used by the expression
New scheduled run fails24 h0 h24 h
Unowned manual debug job succeeds24 h0 h24 h
Owned completion is recognised24 h28 h−4 h

This is new explanatory arithmetic, not the original incident’s timestamps. The manual success does not change the calculation because it does not change the parent field. Once the later owned success is recognised, the difference is no longer above four hours. The expression can become false even though that verification completion occurred after the most recent scheduling timestamp.

The example also limits the conclusion. Advancing the controller’s success field is stronger recovery evidence than an unrelated green Job. It still does not prove that the next ordinary scheduled invocation will start and succeed. The required observation depends on which of those behaviours is being accepted.

Short retries and another daily opportunity

The repair changed two operational settings. The Job’s retry budget increased from 1 to 4. The CronJob’s single morning opportunity became a morning and evening schedule.

cronjob-after.yaml · CronJob timing · GitOps de2cd074 · exact historical excerpt

schedule: "0 7,19 * * *"
suspend: false
timeZone: "Europe/London"
concurrencyPolicy: Forbid
Explain this code line by line
  • The schedule requests a run at 07:00 and 19:00 in the selected timezone.
  • suspend: false leaves scheduling enabled.
  • Europe/London gives those hours their local-time meaning rather than assuming a fixed UTC offset.
  • Forbid prevents this CronJob from starting another concurrent Job while an earlier one remains active.
  • These fields describe the requested controller behaviour. Their presence in Git does not, by itself, establish that the running cluster applied them.

Short retries and an evening opportunity addressed different interruption lengths. Several closely spaced attempts might outlast a brief read failure. They could all finish before an unstable upstream service recovered from a longer outage. The second scheduled opportunity spread recovery across the day.

The investigation estimated that the enlarged retry sequence still covered only a few minutes. That was operational reasoning in the report, not a guaranteed deadline derived solely from backoffLimit. Runtime, restart behaviour and controller timing remain relevant. The Kubernetes CronJob documentation also makes clear that scheduling has limitations; a cron expression is not an exactly-once execution guarantee.

Inside a dark stone clock tower, the engineer adjusts two separate trigger pins on a mechanical timing drum beneath a narrow rain-streaked window.

A second opportunity later in the day covers a different interruption window from a few immediate retries. Conceptual illustration.

Re-running safely required a data contract

Adding another run would have been a poor recovery strategy if every overlap created duplicate scores or repeated all the expensive judging. The retained configuration kept a 26-hour lookback, so twelve-hourly runs intentionally examined overlapping work.

The Python used two protections. A pre-pass collected observation identifiers already carrying a quality score. The main loop then skipped those identifiers before considering an output for judging. Empty outputs were counted separately rather than assigned a fabricated zero-quality score.

quality_scorer.py · score_recent eligibility checks · platform d0347c3 · exact historical excerpt

gen_id = _attr(gen, "id")
if gen_id in already:
    stats.skipped_already += 1
    continue
output_text = _as_text(_attr(gen, "output"))
if not output_text.strip():
    stats.skipped_no_output += 1
    continue
Explain this code line by line
  • gen_id identifies the generation currently being considered.
  • Membership in already means the pre-pass found a quality score for that observation.
  • The counter records why the generation was skipped, and continue moves to the next one.
  • _as_text normalises the stored output into text.
  • Stripping whitespace tests whether there is anything meaningful to judge.
  • The final counter distinguishes missing output from an assessed answer. No score is written in this branch.

When an eligible output did receive a score, the write used a generation-derived identity:

quality_scorer.py · score_recent score write · platform d0347c3 · exact historical excerpt

langfuse.create_score(
    name="quality",
    value=score,
    trace_id=_attr(gen, "trace_id", "traceId"),
    observation_id=gen_id,
    score_id=f"quality-v1-{gen_id}",
    data_type="NUMERIC",
    comment=(reason or "")[:500],
)
Explain this code line by line
  • The score is named quality and carries the judge’s numeric value.
  • The trace and observation identifiers link it to the work being assessed.
  • The score identifier combines a fixed version prefix with the generation identifier. Reconsidering the same generation targets the same score identity.
  • The data type declares the stored value numeric.
  • The final field keeps a bounded explanation from the judge.
  • This shows the application’s repeat-write identity. It is not a recovered proof of every backend transaction or failure ordering.

In an ideal regular twelve-hour schedule, two 26-hour windows overlap by fourteen hours. That simple calculation explains why overlap was expected. It does not establish complete processing: the implementation also paginated its reads, and a record could become available outside the window. The repair improved recovery opportunities within the existing contract; it did not turn the scorer into an unbounded backlog processor.

Zero new scores could be a healthy completion

Both reports contain a successful run with no newly created scores. The numbers describe different runs and should remain separate:

Historical reportFetched generationsNo outputAlready scoredNewly scoredReported errors
Labelled investigation probe1481163200
Owned verification in the recovery report82503200

The skip categories account for the fetched totals in each reported run. There was no remaining eligible output in those summaries. scored=0 therefore did not mean the judge had rated the answers zero, nor did it establish that the scorer had failed to do its permitted work.

The converse also matters. The scorer could count a per-generation judging error and continue, and its metrics push was best-effort. A successful process exit was not a universal guarantee that every assessment and every metric delivery succeeded. The alert’s controller-success condition, the scorer’s completion counters and the meaning of a quality score were separate observations.

There were even two different success signals in the implementation. The application pushed its own run-success timestamp to the metrics gateway. The alert quoted above read the CronJob status exported by the cluster. A new application metric could be useful without changing the controller timestamp used by that particular alert.

What the closeout actually established

The investigation handover preserved an interrupted checkpoint with the fix still listed as open. The recovery report and inspected GitOps commit establish the committed repair state. Their document clocks overlap, so they should not be treated as a complete, ordered session replay.

The closeout reports the changed schedule and retry budget applied to the cluster, a successful CronJob-owned verification Job, an advanced lastSuccessfulTime, and no firing series for the named alert. That is the reported recovery evidence. The retained Git diff independently confirms the requested settings changed; it cannot independently prove the historical live status.

I would keep the next acceptance boundary visible: the report does not supply a later ordinary scheduled run as a separate observation. Nor does the retry/schedule repair resolve the underlying langfuse-web instability or calibrate the judge’s quality scores as correctness measurements. Those remain different pieces of engineering work.

What I would carry into the next build

Follow an alert to the state it actually reads before changing its rule. Reproduce the relevant workload identity when building a debug copy, and distinguish network-policy labels from controller ownership.

Choose recovery evidence that matches the intended guarantee. A manual success can demonstrate a useful application path. An owned completion can exercise the parent status update. An ordinary later scheduled success would add another observation. Each earns its own conclusion.

Make overlapping recovery safe enough to be useful. Inspect eligibility, skip counters and stable write identities before increasing the frequency. Keep zero new work distinct from a zero-quality answer, and keep a green Job distinct from complete quality assurance.

The next records move to document ingestion, where a nightly deadline, an overloaded dependency and one troublesome document could hide behind the same warning. That is a separate chapter about clearing one failure boundary only to expose the next.

Glossary and primary references

CronJob

A Kubernetes controller that creates Jobs according to a schedule.

Primary reference

Job

A workload object that tracks a task running toward completion. A separate manual Job need not belong to the CronJob it resembles.

Primary reference

Controller reference

The controlling owner relationship on a child object. It is separate from the labels used to select network policy.

Primary reference

Network policy

Rules selecting which pod connections are permitted. Different labels can put a copied probe on a different permitted path.

Primary reference

Retry budget

A limit on permitted retries. Its execution timing also depends on restart and controller behaviour.

Primary reference

Idempotent operation

An operation designed so repeating the same logical write does not create another distinct result. Here a stable generation-derived score identifier supports that intent.

Primary reference

Pending alert

An active expression that has not yet remained active for its configured firing delay.

Primary reference

Last successful time

The successful child completion recognised in the parent CronJob status, rather than any arbitrary successful application run.

Primary reference