Wiring the Trust Engine into This Blog

The trust engine needed to do more than explain how a claim could be checked. I wanted to see what happened when the pieces ran together, what the resulting signals meant, and where a failed check would actually stop something.

Episode 16 covered the libraries behind that idea: the source registry, grounding, the audit store and the two checks around the final wording. This chapter follows the wiring work in the same June window. I set the direction and reviewed the work; coding agents assisted with implementation, investigation and the reports used to reconstruct it.

The less tidy part is that a working gate, a working dashboard and a publisher that obeys the gate are three different things. Following the actual entry point made that distinction much clearer than the ticket title did.

A proof press can work without controlling the production run. Conceptual artwork, not a historical photograph.

Follow the entry point

The work item made the intention explicit: VW-995 — rootweaver-audit: wire TwoPassPublishGate into production blog_publish_flow + emit gate/publish metrics (Child G deferred wiring). That is the recorded target. The implementation gives us a more precise account of what this slice delivered.

In the retained June source, build_publish_pipeline() constructs real components: a vault connector, a source registry, the grounding service, the gate service and the deterministic voice renderer. This is useful integration work. It is a path through the libraries rather than a picture of how they might fit together.

The command that calls it, however, is an observability emitter. Its main() builds that pipeline, sends it demo_claims(), pushes the resulting metrics and returns zero if those calls finish without raising an exception. The two claims are a curated corpus referring to earlier blog episodes. They are not the text of whichever new article is about to be released.

Exact historical excerpt: rootweaver_audit_observability/emitter.py · main(), snapshot 51667774.

def main() -> int:
    """CronJob entrypoint: ground + gate the demo corpus, push the resulting
    gate/publish series to the pushgateway under the blog-publish-gate job."""
    from rootweaver_audit.config import Settings

    vault_root = os.environ.get("AUDIT_VAULT_ROOT", ".")
    gateway_url = os.environ.get("PUSHGATEWAY_URL", DEFAULT_PUSHGATEWAY_URL)
    pipeline = build_publish_pipeline(Settings(), vault_root=vault_root)
    emit_publish_metrics(pipeline, demo_claims(), gateway_url=gateway_url)
    return 0

Explain this code line by line

• main() -> int is the command entry point and returns a process exit code.
• Settings supplies the audit service configuration. The next two lines choose the vault root and metrics destination from the environment or defaults.
• build_publish_pipeline(...) assembles the real library path. Constructing it does not show that a website publisher calls it.
• demo_claims() supplies the fixed demonstration corpus. The emitter runs those claims and sends the resulting measurements to the metrics destination.
• return 0 reports successful command completion. There is no branch here that turns a blocked PublishDecision into a nonzero exit, and no website release operation in this function.

That last point changes the story. A blocked claim can be recorded correctly while the command still completes successfully. That is reasonable for a demonstrator whose job is to report gate activity. It would be insufficient as the only release control if a publisher treated exit zero as permission to publish.

The later manifest reinforces this reading. Its header calls the job a “demonstrator emitter”, runs it hourly, and puts its writes in a dedicated audit_demo schema. Keeping demonstration events apart from the production audit chain is a sensible precaution. It also means those events should not be presented as evidence that each newly published article has passed a mandatory check.

The manifest entered retained history just after midnight on 25 June UK time, following the 24 June work. I use that narrow boundary receipt to confirm the job’s intended role, rather than moving the whole episode into a later period.

What a two-pass decision actually tells me

Inside the library, the sequence still matters. Gate one checks the claims before voice rendering. If that stage and the intermediate validations pass, the renderer produces the final wording and gate two checks that output. A failed earlier stage can mean gate two never runs.

A publishing system needs to preserve that distinction. “No second gate result” cannot quietly become “second gate passed”. Equally, a counter recording a decision is not the same operation as a caller obeying it.

From curated claims to a recorded decision

Pause flowExpand diagram

Close expanded diagram

For example, imagine a claim whose sources are acceptable before editing, but whose final wording adds an unsupported number. The useful result is a failed final check with the affected text identified. The release caller would then need to hold that exact artifact. Recording “blocked” on a dashboard while publishing the changed text would leave the central problem untouched.

That is an explanatory example, not a recovered production incident from this window. The retained integration test does exercise real connector, registry, store and gate components against a fixture vault and test databases. Its NLI classifier is injected and always returns a strong entailment result. It therefore tests composition and recording without measuring the real model’s accuracy or proving website enforcement.

The June 23 handover was explicit about the unfinished operational steps: build and smoke-test the image, provision the database schema and role, then commit and reconcile the job. Its green CI report was progress against the code. It was not a substitute for those remaining steps.

The image needed the model, not just the package

The next obstacle was less philosophical. The image could omit the dependencies needed to do its advertised work.

VW-1003 — v2 audit-probe/blog-publish-gate image: bake DeBERTa NLI weights, grounding dev→runtime dep addressed that packaging gap. The grounding package kept its NLI stack in an optional extra. That lets callers use the lighter parts of the package without installing the model runtime. The observability image’s ordinary dependency export did not automatically include that extra.

The fix in the retained Dockerfile exports both requirement sets before installation:

Exact historical Dockerfile commands, shown without surrounding shell continuations: rootweaver-audit-observability/Dockerfile, snapshot 51667774.

uv export --package rootweaver-audit-observability --no-dev --frozen --no-hashes
uv export --package rootweaver-audit-grounding --extra nli --no-dev --frozen --no-hashes

Explain this code line by line

• The first export resolves the ordinary observability package closure from the lockfile.
• The second explicitly selects the grounding package’s nli extra, bringing the model runtime into the requirements set.
• --no-dev excludes development groups, --frozen uses the existing lockfile, and --no-hashes controls the requirements-file output. The Dockerfile then combines these exports, removes editable workspace references and installs the third-party dependencies before the workspace source.
• These are the original commands with their redirections and shell chaining omitted for readability. They are not a standalone reproduction of the entire build.

Astral’s uv documentation explains the optional-dependency mechanism. Here the important distinction was between a package being available and the runtime closure being able to load the model.

The Dockerfile also loads Moritz Laurer’s DeBERTa NLI model during the build, using a fixed model revision, and copies its cache into the runtime image. Offline environment settings then tell the runtime to use the retained files. That makes the tokenizer and weights part of the build concern instead of a surprise download when the job starts.

A casing is not a usable instrument until its working parts are present. This illustration accompanies the missing runtime dependencies and baked model assets.

There was also a test limitation worth keeping. The handover says the existing NLI-marked test constructed the wrapper and checked its device, but did not exercise the real model call. The proposed smoke test therefore had to load and run the classifier with networking disabled. A constructor test cannot establish that the deployment contains every file needed for inference.

The source shows the packaging change and a fixed revision. I have not rebuilt that historical image or recovered an independent offline-smoke transcript for this reconstruction. I can explain the fix without claiming that writing the Dockerfile proved its runtime behavior.

The bill pointed at a different workload

The same week, a cost investigation changed what deserved attention first.

The recorded direction was to attribute paid model usage to individual use cases and put a ceiling around it. VW-946 — Paid-API cost + quality governance: route Anthropic/OpenAI/Perplexity calls through Bifrost virtual keys (per-use-case budgets) + Langfuse quality lens captures the original plan, including Bifrost in the title. The implementation report says the gateway used for the delivered spine was LiteLLM. The original ticket name and the chosen implementation should not be silently treated as the same product.

Then the billing investigation pointed elsewhere. According to the retained report, the paid Anthropic usage being examined came from the nightly journal consolidator, rather than the five call sites at the centre of the gateway work. The consolidator already had several backends, but the paid one was first.

Figure retained in the report	Meaning
May: $12.51	Reported Anthropic total for the month
June through 20 days: $20.04	Reported partial-month total
About $30 for June	A run-rate estimate, not a completed-month invoice

The arithmetic behind that estimate is straightforward: $20.04 / 20 × 30 = $30.06. It is about 2.4 times the reported May total. I have the engineering report’s account of the billing exports, not the original CSVs, so these remain reported historical figures. They do not establish the cost of every provider or the running cost of the local hardware.

The immediate change became VW-972 — Make daily-consolidator LLM order env-driven (LLM_PRIORITY), default local-first — cut ~$30/mo Haiku + survive $0 Anthropic balance. Its practical purpose was to try the local backend first and reserve the paid path for later in the chain.

Exact historical assignment: daily_consolidator_claude.py · DailyConsolidatorClaude, snapshot 51667774.

    _DEFAULT_LLM_PRIORITY = ("vllm", "groq", "claude")

Explain this code line by line

• The tuple lists backend names in their default attempt order.
• The resolver can override the order using the LLM_PRIORITY environment variable.
• The call loop returns the first nonempty result. It does not compare several answers and select the best one.
• Consequently, a poor but nonempty local answer can still win. This is failure fallback, not an automatic quality competition.

The report records four out of four calls using the local model and no paid API cost for that run. It also reports a qualitative review of the output. That was a useful initial result. It was not a blinded model comparison, a long-run quality benchmark or proof that the fallback providers would never be used again.

Local-first needed an honest failure path

Changing the order exposed another small dependency: the local model name had to match what the server actually served. The initial report records a stale default requesting a different model. A request to a nonexistent model could fail and move the work to another backend, leaving “local-first” as an intention rather than the observed route.

The follow-up source adds discovery from the server’s model list and caches the selected name for the run. An explicit operator override still takes precedence. That qualification matters: discovery reduces one kind of drift; it does not make every possible configuration valid.

The stricter boundary was VW-978 — daily-consolidator vLLM model discovery: hard-fail on discovery failure instead of stale qwen3-14b default (silent Groq demotion in a cost path). In the retained implementation, model resolution sits before the completion call’s try block. A discovery failure therefore propagates instead of being converted into an ordinary empty result that advances the fallback chain. A later completion failure can still return None and allow fallback. Those are deliberately different failure classes.

This is the sort of detail I want the public record to retain. “We made it fail loudly” would be too broad. The change made a particular configuration/discovery failure loud while preserving ordinary fallback for the completion request.

Output attribution needed attention too. The report identifies an agent-name voice leak, a stale model footer and a mismatch in the session count. The follow-up source records which backend actually returned the text and uses that information in the footer. A report can contain a useful summary and still misstate who or what produced it; moving generation locally made that existing assumption visible.

Quality monitoring arrived before useful samples

Spend attribution could tell me which path was being used. It could not tell me whether the resulting answer was good enough. That became VW-980 — Per-use-case LLM output-quality lens via Langfuse + quality circuit-breaker (carved out of VW-946).

The work was split into three concrete stages:

Work item	Mechanism
VW-983 — VW-980 Phase 1 fan-out: shared Langfuse quality-lens helper + instrument the 4 remaining paid call sites	Shared instrumentation records generations and a use_case label.
VW-985 — VW-980 Phase 2: async LLM-judge quality scoring (free local vLLM) into Langfuse scores	An asynchronous local judge reads eligible outputs and attaches a numeric score.
VW-986 — VW-980 Phase 3: quality circuit-breaker (alert-only) — per-use_case quality gauge → Prometheus alert	Per-use-case gauges feed an alert; there is no automatic shutdown.

Langfuse’s documentation on model-based evaluation provides the upstream context for that scoring pattern. The particular rubric, local judge adapter, skip rules and alert wiring described here are Rootweaver implementation choices.

The scorer uses a rubric appropriate to the use case and asks the judge for a score and a reason. Here is a deliberately small illustrative exchange, not a recovered production payload:

Synthetic example following quality_scorer.py · judge_generation() and _parse_judge(). Values are invented for explanation.

{
  "use_case": "websearch",
  "input": "Explain why the job can finish without scoring any output.",
  "output": "Empty generations are skipped before the judge is called.",
  "judge_response": {"score": 0.8, "reason": "Direct and relevant."}
}

The score is an assessment against a rubric. It is not a probability that every fact is true. An out-of-range or unparseable response is skipped rather than written as a valid score. Empty outputs are also skipped, and existing quality scores are checked to avoid judging the same generation again unnecessarily.

That makes the actual handover unusually revealing. It reports fetched=28 scored=0 errors=0. The fetched records were described as having no usable output while the upstream cap was in effect. The integration could be deployed and the job could finish cleanly without having measured the quality of one real answer in that run.

Ready to assess is different from having samples to assess. Empty cups illustrate the reported zero eligible outputs, not a failed or successful quality result.

The alert guard was also specific: the reported rule required an average below 0.5 and at least five scored samples, held for 24 hours. Prometheus’s alerting documentation explains that for is a duration for which the condition must remain active. It does not mean “the last two completed runs were bad”. With nightly gauges, missing pushes, stale values and sample volume still deserve attention.

The metrics push is best-effort. If it fails, the scorer can log a warning and still exit successfully. The handover records exactly that operational trap: sender egress was allowed, but the receiver’s ingress policy also needed to admit the scorer. Fixing both ends made the path usable. A successful job alone had hidden the missing delivery step.

This is why I keep quality monitoring separate from the grounding gate. Grounding asks whether the claim has appropriate support. The rubric asks how useful the output is for its task. Alerting makes a measured problem visible. None of those roles, by itself, proves that a website release will stop.

What I would carry into the next build

The important improvement was learning to name the boundary each result actually covered.

Question	What this window gives me
Can the audit components be assembled?	Inspected factory and integration-test source, with an injected NLI classifier in the test.
Can the image include the inference stack?	Inspected dependency export and model-cache build steps; no fresh historical image run here.
Did local-first do anything useful?	A historical report of four local calls, followed by source-level discovery and attribution hardening.
Was the quality lens only a future idea?	No. Its code and deployment records exist, but the cited live run scored zero eligible outputs.
Did every blog release have to obey a failed decision?	That enforcement connection is not established by the inspected demonstrator and reports.

I would inspect the caller and its input first next time. Before adding another dashboard panel, I would want to know whether the job receives the current article, whether it checks the final bytes, and what happens to the release when the result is blocked or unavailable.

I would also pull the workload-level cost evidence earlier. The gateway work still had a purpose, but it was not the quickest answer to the usage the report had actually found. And I would keep measuring quality separately from whether a backend returned text at all.

The engine had become more concrete: real components, an executable integration path, metrics, a model-bearing image design and a quality-scoring loop. The records also exposed the limits of each result. That is a more useful account of progress than treating a successful job as proof of everything downstream.

Next: Episode 18, The Ops Day. The story returns to Rootweaver’s operational foundations: getting alerts to the right place, making the first human actions easier, and checking the access and delivery paths those signals depend on. Those foundations set up the season’s closing question: how much autonomy should the platform earn?

Terms and references

Glossary — plain-English definitions and primary references

Demonstrator

A bounded example that exercises a mechanism. Here the emitter uses a curated corpus, so its activity is not a record of every new website release.

NLI and DeBERTa

Natural language inference assesses whether one text supports or contradicts another. Moritz Laurer's model card describes the particular upstream model; its result is not a guarantee of truth.

Optional extra

A named group of package dependencies that must be selected explicitly. See Astral's uv documentation.

Virtual key

A gateway-issued credential used to attribute and constrain a client's model usage without giving it the provider's underlying key. See LiteLLM's virtual-key documentation.

LLM judge

A model asked to score an output against a rubric. It supplies an assessment that needs calibration and review. See Langfuse's evaluation guide.

Pushgateway

A service that retains metrics pushed by short-lived jobs so Prometheus can scrape them. See the Prometheus project's guidance, including its warnings about stale series.

Alert-only

A condition can produce a warning without disabling the workload. A Prometheus for duration requires the condition to remain active over time; it does not count job runs. See alerting rules.

How this episode was reconstructed

This is a curated reconstruction of June 21–24, with a narrowly identified early-June-25 UK manifest receipt. It is not an exhaustive account of every ticket or a replay of complete native sessions. Historical reports, recovered ticket titles and independently inspected Git source are distinguished in the linked review ledger. Original bills, live model-scoring transcripts and an independently observed release rejection from that week were not recovered. The current reading draft is not claiming that today’s website uses the historical emitter as a release control.

The unchanged pre-crafted episode is retained as a comparison reference. Its stronger descriptions of direct publisher wiring and already-demonstrated live quality are narrowed here to what the inspected records establish.