Skip to content

ADR-0014: Agent skills are script-first — analysis and validation run in scripts, not in LLM context

  • Status: Accepted — tooling tracked in #134 (scenario validation), consumers in #128 (paired stats), #131 (##PERF## pass-through)
  • Date: 2026-07-24

Context

Much of the recurring agent work in this repo is not code change but data work: parsing benchmark tables, computing A/B deltas, judging run-to-run noise, validating scenario configs and result plausibility, transforming campaign CSVs for the papers repo. Done "in context" — the model reading raw tables and doing arithmetic in its head — this work is:

  1. Error-prone. The #47/#68/#71 EnableMacMetric episode showed the exact failure: eyeballed deltas with opposite signs across pairs read as signal when they were noise. A scripted, thresholded verdict does not make that mistake twice.
  2. Token-expensive. A classified campaign CSV is ~25 columns × dozens of rows; a CI job log is hundreds of lines. Loading them into context on every session burns budget the CLAUDE.md token-economy policy exists to save — and the numbers themselves must stay byte-exact, which summarising in prose does not guarantee.
  3. Non-reproducible across sessions. An in-context judgement leaves no artifact; the next session (any model) re-derives it, possibly differently. ADR-0013 solved this for findings (issues); this ADR solves it for procedures.

Decision

Recurring parse/analysis/validation loops ship as executable scripts inside the agent skills (.claude/skills/), and the skill instructions direct the model to run the script and read only its compact output. Rules:

  1. Raw data stays out of context. Job-log tails and CSVs are saved to files and handed to scripts; only verdict lines (a delta grid, a FAIL/WARN/OK summary) should be read back. Never eyeball tables, compute deltas by hand, or hand-type result numbers.
  2. Scripts are stdlib-only Python (or POSIX shell), runnable in any sandbox with no installs, and are validated against real repo data before landing (e.g. the rescued #122 campaign CSVs, the committed ##BENCH## formats).
  3. Thresholds live in one place. Scripts read shared threshold files (e.g. ns3/tools/anchors.yml, #59) rather than duplicating constants that would drift.
  4. New recurring analysis extends a script rather than being performed in context; one-off exploration is fine in context but graduates to a script the second time it recurs.

Current inventory:

Script Skill Job
bench_parse.py benchmark-results parse ##BENCH## cells, A/B deltas, noise verdict
sweep_summary.py benchmark-results campaign-CSV schema check, per-point deltas, stddev-aware noise call, papers export
scenario_check.py benchmark-results pre-flight scenario degeneracy checks; result plausibility + anchor floors (#134)
check_invariants.sh protocol-review golden-rule checks on a diff
figdata.py figures (papers repo) campaign CSV / ##BENCH## → paper plot data, diff report, preliminary-flag audit

Alternatives considered

  • Keep procedures in prose (SKILL.md/issue notes only). The procedure survives, but the arithmetic still happens in context — the #47-class errors and the token cost remain. Rejected.
  • A single monolithic CLI tool in tools/. Couples agent workflow tooling to the shipped software; skills keep agent-facing tooling discoverable at the point of use and out of release artifacts. Rejected.
  • CI-only enforcement (no local scripts). CI gates (e.g. #59 anchors) catch regressions post-push but cannot do pre-flight checks before an expensive dispatch (#121) or ad-hoc A/B analysis mid-session. The two are complementary; CI enforcement alone is insufficient.

Consequences

  • Session context spends tokens on decisions, not on tables; verdicts are identical across sessions and models.
  • The scripts are themselves code: they need the same review care, and their formats (e.g. ##BENCH##, the classified CSV schema) become interfaces — a harness output change must update the parsers in the same PR.
  • The papers repo participates via its figures skill (figdata.py), so numbers cross the repo boundary only through validated transforms (papers-repo golden rules 1–3), never by hand.

Amendment (2026-10-09): shared scripts live in tools/, skills hold procedures

.claude/ is excluded from both release packages (the install-bundle zip and GitHub's source archives, via export-ignore). Shipped tools nevertheless imported the benchmark-results skill's scripts:

  • run-scenarios.py used pool_runs;
  • check-leo-walker.sh used scenario_check;
  • family-charts.py used bench_parse and stats_util;
  • leo-summary.py used stats_util.

So the released tools broke, and the methodology links 404'd (#604).

The rule from now on:

  • A script that any shipped tool, workflow or doc depends on lives in tools/. The benchmark scripts are in tools/bench/ today.
  • .claude/skills/<name>/ holds the procedure (SKILL.md), plus scripts only an agent runs.
  • The release-package gate (tools/release/check-bundle.sh, run in CI) fails if a shipped file references .claude/.

The decision above (script-first, verdicts not raw data in context) is unchanged.