A Checked Skill Still Had to Earn Its Place
The previous episode followed a memory-first gate that was measuring the wrong question. Once the input was wrong, a precise metric could still describe the wrong behaviour. The next July thread moved that problem into the instructions themselves. An agent skill could be valid Markdown, fit the repository and pass review while its advice remained dated or its practical value untested.
I wanted reusable engineering guidance for Rootweaver without importing another project’s assumptions. The work began as a small port of two skills, expanded to five, then changed direction when I asked what actually grounded the advice. A later bounded trial asked whether invoking one of those skills changed the work.
A coherent recipe is a starting point. The ingredients, source and result still need checking. Conceptual illustration.
This chapter covers July 7, 2026, with the July 6 pilot specification as declared setup context. It is a curated reconstruction from the skills handovers, pilot and efficacy reports, and retained source. I owned the question, direction and review; agents assisted with authoring, research and checking. Reported verifier results and trial scores remain historical reports. New inspection does not rerun those agent trials or establish present-day task performance.
A port needed more than renamed tools
The starting material came from Addy Osmani’s agent-skills project. The retained pilot report identifies it as MIT-licensed and credits that source. Rootweaver’s adaptation replaced generic web-application defaults with the platform’s actual boundaries: source code, CI, image registry, GitOps, cluster reconciliation and runtime evidence.
That was a substantive part of the work. Advice to inspect a GitHub Actions job would be misplaced in a GitLab-driven workflow. A deployment checklist that ended when an image existed would miss the later GitOps and runtime stages. A security checklist had to consider retrieved documents and tool inputs, not just a public website’s request handlers.
The pilot deliberately covered two areas first: security and hardening, and CI/CD automation. Mechanical checks tested the document shape and looked for generic tool references that should not have survived the adaptation. A separate agent verifier checked platform consistency. The report records those checks as passing; it also says a client-specific skill-load smoke was approximated by frontmatter validation and direct inspection because the authoring client did not expose the other harness’s native skill tool.
That limitation belongs beside the result. Finding a skill file and validating its frontmatter can show that the artifact is well formed. It cannot prove that another application’s skill discovery loaded it, that an agent invoked it on a real task, or that the invocation improved the result.
There was a more personal provenance defect too. The first port contained the authoring agent’s own identity in instructions intended for future agents. If copied forward, those instructions could attribute a later non-Codex contribution to Codex. The historical handover says the hardcoded labels were replaced with dynamic producer identification, and the follow-on specification added an explicit check against repeating that mistake.
This is an easy defect to overlook because the document can be perfectly readable. Its instructions are wrong about who will use it. A reusable workflow needs the actual future producer to identify itself; the identity of the agent that first wrote the template is historical metadata, not a universal default.
Three different questions about quality
After the pilot, three more skills covered observability, performance and shipping. The July records describe five completed ports. My next question was whether their disciplines were grounded in research and established practice. The answer recorded at that point was that consistency had been checked, but external citations had not.
That is a meaningful distinction. A verifier can compare a document against a platform map and catch a wrong service name. It cannot establish that a claimed industry definition is current merely because every local document repeats it. Internal agreement is evidence of consistency, not external truth.
| Check | Concrete question | Evidence it can provide | Limit |
|---|---|---|---|
| Platform consistency | Does the guidance fit this system? | Paths, ownership, tools and workflow references agree | The advice can still be outdated |
| External grounding | Does a primary source support the advice? | A dated, attributable source and a traceable claim | A citation alone does not show practical benefit |
| Task efficacy | Does invoking the skill improve this task? | A bounded control/treatment comparison | One draw does not establish a general effect |
The later research pass added primary references and found one concrete problem in the shipping guidance’s DORA framing. Other skills received references without a substantive correction. A null result can be useful when a real source check precedes it; it should not be assumed merely because the prose sounds familiar.
There is a further correction to preserve in this reconstruction. The historical edit discussed reliability as a fifth DORA key. I am not copying that shorthand as current guidance. Nathen Harvey’s DORA history explains that the framework changed over time. The current delivery-metrics guide includes deployment rework rate in its five delivery measures. The old skill and its reported correction are historical evidence; neither replaces reading the primary definition for the claim being made now.
That does not invalidate the useful July decision to require grounding. It shows why grounding is a continuing traceability obligation rather than a permanent certificate attached to a document. A source needs a date, a claim and the right scope. An old summary can lag behind the source it cites.
Follow the mechanism
The diagram is an editorial model of the three checks. They are separate routes to evidence, not three automated stages proved to run on every later task. A failure in one cannot be repaired simply by collecting more passes in another.
A real task gave the skill a fairer test
The later trial used an actual engineering surface: Dockerfiles installing the uv tool through unpinned or insufficiently verified fetch paths. The question was not whether a skill could explain security in the abstract. It was whether invoking the security skill changed the completeness and discipline of a plan for those files.
The author of a skill already knows its contents. Asking that same context to act as an uninformed control would contaminate the comparison. The report instead describes two fresh agent contexts on the same task: a control without the skill, and a treatment told to invoke the skill first. Neither saw the grading rubric. Six expected differences were registered before the two outputs were examined.
A small comparison can answer a bounded question. It cannot establish what happens in every environment. Conceptual illustration, not the trial dataset.
The historical score was five of six rubric items for the control and six of six for the skill-assisted arm. Both proposed the same core repair. The useful difference was largely scope discipline: the treatment explicitly cleared clean files from the work and made trust-boundary reasoning more consistent. It did not discover a vulnerability that the control missed.
Even the supposed oracle needed correction. The report says an initial search missed a pip-based installation because a comment matched first. The corrected trial fixture counted ten affected Dockerfiles. A later implementation handover described eleven changed files across a broader surface. Those counts should not be silently reconciled into one number: they describe different inventories at different stages. The draft follows the retained trial’s denominator when discussing the trial and the later source when discussing the actual change.
A grep result is therefore inspectable evidence, but only if the search pattern, scope and interpretation are correct. Calling it ground truth does not make the first query infallible. The trial was stronger because it corrected that mistake before treating the output as its scoring reference.
What the Dockerfile change actually pinned
The retained July implementation source contains this instruction. It is an exact historical line from the gateway Dockerfile at revision c3ec5d23; the version and digest are evidence of that change, not a current recommendation to install an old release.
gateway-Dockerfile · builder stage · platform c3ec5d23
COPY --from=ghcr.io/astral-sh/uv:0.11.27@sha256:4d01caf3b22dfd11003455e2e68153da08c4ee1fa54fdbd166c6282d22693419 /uv /uvx /usr/local/bin/Explain this code line by line
COPY --fromselects files from another container image during the build.- The version tag names the intended historical release; the digest pins the selected manifest rather than trusting a mutable tag alone.
/uvand/uvxare the two source binaries./usr/local/bin/is their destination in this builder stage.- This line does not establish publisher trust, vulnerability status or final runtime contents. Those claims need their own evidence.
The tag helps a person recognise the intended release. The digest identifies the selected image manifest. The COPY operation takes the two tool binaries from that image into the builder stage. This avoids fetching an installer script during the build merely because a URL still resolves.
Pinning the source does not answer every supply-chain question. A digest identifies selected content; it does not establish that the publisher was trusted, that the dependency was vulnerability-free, or that a later runtime image contains only the intended tools. Those are additional checks. Docker’s build guidance explains the reproducibility role of digest pinning, while Astral’s container guidance documents the supported integration pattern.
The source also distinguishes the builder from the runtime stage. Dependencies and package wheels are assembled before the final image is created. That separation is useful, but a review must inspect what the final stage copies. A label saying builder-stage-only cannot substitute for following the actual instructions into the runtime filesystem.
Here is a synthetic evidence record for that review, with no real digest or registry proof implied:
{
"check": "tool image pin",
"source_instruction": "Dockerfile COPY --from",
"selected_content": "digest present",
"publisher_verification": "separate check",
"runtime_placement": "inspect final stage"
}The fields keep three claims apart: the build selected fixed content, the content came from an accepted origin, and the final runtime contains the expected files. The example is a review aid. It is not a retained attestation or evidence that all three checks were performed that day.
A useful result with a small denominator
The report’s modest verdict is better supported than a claim of universal improvement. One treatment output satisfied one more expected rubric item than one control output, on one skill and one task. That supports keeping and studying the guidance in that setting. It does not supply an effect size for all agents or a confidence interval over future work.
Some original commentary went further, suggesting that a checklist guarantees consistency on every invocation or must help weaker models more. Those are hypotheses here. The retained single comparison does not establish either claim. A skill can be skipped, misread, conflict with task instructions or guide an agent toward the wrong scope. Repeat trials and evidence of actual use would be needed to test consistency and model-dependent benefit.
It would also be a mistake to score only the treatment’s wins. The control surfaced a registry-egress risk that the report says the skill arm underweighted. That is useful adverse evidence. A process intended to improve security should retain that observation instead of flattening the outcome into “skill wins.”
| Retained result | Supported conclusion | Still open |
|---|---|---|
| Five ports passed consistency review | The reported local checks found no remaining platform mismatch in that scope | Actual loading and use in every client |
| Primary references were added | The guidance became easier to inspect and challenge | Whether every claim remains current |
| Trial scored 5/6 versus 6/6 | The treatment was more complete under that rubric on that draw | Repeatability and broader task benefit |
| Both arms chose the same core repair | The skill’s value was not new vulnerability discovery in this test | Whether another task would show a different effect |
The trial and the later implementation also needed separate inventories. A controlled comparison should keep its fixture stable. A real change should re-check the files that exist when it is applied, including clean files deliberately left outside scope. Otherwise a score about a small fixture can drift into an unsupported claim about an entire repository.
Define the work that belongs in the change, and leave unrelated material outside it. Conceptual artwork bound to the scope discipline observed in the trial.
Lessons I would keep
I would credit an upstream project at the point its contribution enters the story. Adaptation is real work, but it does not erase the origin of the method. The local port should explain what changed because the environment was different.
I would keep producer identity out of reusable imperative text. The skill can require provenance, while the actual agent supplies its observed identity when it acts. That preserves authorship instead of copying an old author’s label into a new contribution.
I would ask three separate questions before calling a skill good: does it fit, is it grounded, and did it help? Those questions need different evidence. Passing more formatting checks cannot answer the last two.
For a follow-up trial, I would keep the real fixture, bounded scope, concealed rubric and pre-registered expectations. I would add repeated draws and a second task before making stronger claims about reliability or model effects. I would also retain any useful finding from the control, because the purpose is to improve the work rather than defend the treatment.
Next in the sequence
The next July record returns to the publishing machinery. A claim ledger could contain the right evidence while a small header mismatch stopped the parser from seeing it. The July 8 fix is a useful continuation: before judging the quality of evidence, the system has to recognise what the author actually supplied.
Glossary — plain-English definitions
Agent skill
Reusable instructions that a client may discover and invoke. Primary concept reference.
Grounding
Linking a claim to an appropriate inspectable source. Episode evidence and limits (retained historical evidence).
Control arm
The comparison condition without the selected intervention. Episode evidence and limits (retained historical evidence).
Treatment arm
The condition with the selected intervention. Episode evidence and limits (retained historical evidence).
Pre-registration
Stating expected comparisons before examining the trial outcomes. Episode evidence and limits (retained historical evidence).
Image digest
A content identifier for a selected image manifest, distinct from publisher trust. Primary concept reference.
Rubric
The explicit criteria used to score an output. Episode evidence and limits (retained historical evidence).
External validity
How far a result can reasonably be carried beyond the tested setting. Episode evidence and limits (retained historical evidence).
Sources and limits
The source and claim ledger (retained in review ledger) retains the pilot, grounding and trial reports, the historical shipping skill and Dockerfile source. It labels the primary-source clarification made during this reconstruction. Original agent trial transcripts, a repeated experimental sample and registry signature records are not included. The old reports are comparison evidence, not instructions to operate today’s platform or proof that their broadest conclusions were correct.