codebase-design (mattpocock/skills)
codebase-design: gives architecture reviews the depth vocabulary and clearer structure (control 3.2, skill 3.9, one-liner 4.5) for 1.1x the tokens, but it did not find more of the planted problems than the one-liner prompt
23.6 / 30
Lift over no skill: +0.71
Situational: the lift +0.71 is under +1.0. Recommended needs a total of 22/30, output quality of 3.5, and a lift of +1.0.
SOTA
PassTested with Codex.
Covers Codex, Claude Code
Codex CLI 0.166 (SOTA tier): 18 graded runs. A pass here also covers Claude Code under our tier rule; it says nothing about the second tier.
Scores
Each dimension is 0 to 5 and the six add up to the total out of 30. 5 is better on every dimension. Cost 5 means cheap and fast. Install 5 means easy.
- Trigger reliability
- 5.0
- Lift over no skill
- 2.7
- Output quality
- 3.9
- Token and time cost
- 4.0
- Install and portability
- 4.0
- Safety
- 4.0
Security review
SafeStatic scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Manual review: Safe to run in a no-network sandbox (no findings). A clean scan does not prove a skill safe.
Install
npx skills@latest add mattpocock/skills --skill codebase-design
Pinned commit: 4ae2317ddb22b046e26455e4f9c9b3aaaf62212d
Tested: 2026-11-02
What we tested
- Whole-repo architecture scan — Review this small shop backend for architecture improvements: where would restructuring make the code easier to change and easier to test? Write your findings as a self-contained HTML report named report.html in this folder (do not open a browser or start a se…
- Payments and inventory only — Look only at the payment and inventory parts of this backend and tell me which restructurings would pay off. Write your findings as a self-contained HTML report named report.html in this folder (do not open a browser or start a server) and do not change any co…
- Why is place_order hard to test? — It feels painful to write a proper test for place_order in this backend. Why, and what restructuring would fix that? Do not change any files; answer in your final message.
Pros
- Lifts the mean score from 3.21 to 3.92 over no skill (+0.71).
- The skill was picked up without being named: Skill file read in 6/6 treatment runs.
Cons
- A one-sentence prompt scored 4.54, against 3.92 with the skill (-0.63).
- n=2 per task per arm; differences under about 1 point are noise.
Review
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below). A small shop backend with six planted friction points (shallow validators and money wrappers, a pass-through inventory service, a single payment adapter, duplicated logic, and others). Tasks: write an HTML report of architecture improvements, a report limited to payment and inventory, and 'why is place_order hard to test and how would you fix it' (answer in chat). Prompts do not name any design terms. The one-liner states the depth idea and the things to flag.
Three-arm comparison
| Arm | Mean score (0-5) | Tokens per run |
|---|---|---|
| Without the skill | 3.21 | 119k |
| One-sentence prompt | 4.54 | 114k |
| With the skill | 3.92 | 133k |
Measured with Codex CLI over 18 runs, scored by rule-based checks with no LLM judge. Total 23.6 / 30, verdict Situational.
Verdict: Situational (23.6 / 30). Source: mattpocock/skills, skills/engineering/codebase-design, pinned at commit 4ae2317. Install: npx skills@latest add mattpocock/skills --skill codebase-design. Licence: MIT (repo LICENSE).
What it is
The codebase-design skill (mattpocock/skills): a way to review architecture by module depth: deep modules with a small interface hiding a lot of behaviour; flag shallow pass-through modules, single-adapter seams that are only hypothetical, duplicated logic and behaviour tested only through helpers; say where a seam belongs so behaviour is tested through the interface.
How we tested
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).
A small shop backend with six planted friction points (shallow validators and money wrappers, a pass-through inventory service, a single payment adapter, duplicated logic, and others). Tasks: write an HTML report of architecture improvements, a report limited to payment and inventory, and 'why is place_order hard to test and how would you fix it' (answer in chat). Prompts do not name any design terms. The one-liner states the depth idea and the things to flag.
- Whole-repo architecture scan — Review this small shop backend for architecture improvements: where would restructuring make the code easier to change and easier to test? Write your findings as a self-contained HTML report named report.html in this folder (do not open a browser or start a se…
- Payments and inventory only — Look only at the payment and inventory parts of this backend and tell me which restructurings would pay off. Write your findings as a self-contained HTML report named report.html in this folder (do not open a browser or start a server) and do not change any co…
- Why is place_order hard to test? — It feels painful to write a proper test for place_order in this backend. Why, and what restructuring would fix that? Do not change any files; answer in your final message.
Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.
Results
The one-liner given to the third group: "Judge modules by depth: a deep module has a small interface hiding a lot of behaviour; flag shallow pass-through modules, hypothetical single-adapter seams, duplicated logic and behaviour that is only tested through helpers, and say where a seam belongs so behaviour is tested through the interface."
| Task | Control score | One-liner | With skill | Control tokens | One-liner tokens | With-skill tokens | Skill read |
|---|---|---|---|---|---|---|---|
| t1-scan | 2.54 | 3.63 | 3.83 | 135k | 135k | 177k | 2/2 |
| t2-focus | 2.5 | 5.0 | 3.75 | 153k | 139k | 148k | 2/2 |
| t3-testing | 4.58 | 5.0 | 4.17 | 65k | 65k | 72k | 2/2 |
Mean score: 3.21 without vs 3.92 with the skill (delta +0.71). Tokens per run: 119k vs 133k (1.12x); time 154s vs 158s. One-liner group: 4.54 (the skill's gain over the one-liner: -0.63; one-liner's own gain over no skill: +1.34); one-liner tokens 114k per run, so the skill costs 1.17x the one-liner.
What happened
- Control 3.21, skill 3.92, one-liner 4.54 (n=6 each). Without guidance, the HTML reports used almost none of the architecture vocabulary (0-2 uses of deep/shallow/seam/interface, where the check wants 6) in 4 of 4 report runs. The skill reports used it in all 4 runs, as did the one-liner.
- Whole-backend scan: skill 4.24 and 3.41, one-liner 3.41 and 3.86, control 2.88 and 2.2. Both guided arms found 3-4 of the 6 planted points; the shallow validators wrappers and the single payment adapter were missed in several runs by every arm.
- Focused payment and inventory task: skill 3.75 in both runs, control 2.5 in both, one-liner 5.0 in both. All skill and control runs found only one of the two points (the hypothetical single payment adapter was missed); the one-liner found both.
- Why is place_order hard to test: all arms are close (control 4.17 and 5.0, skill 4.17 and 4.17, one-liner 5.0 and 5.0); the skill runs missed one cause (helper-only tests) in one run and pass-through layers in another.
- Tokens: skill 133k, one-liner 114k, control 119k per run (1.12x control). The skill was read in 6/6 treatment runs.
Dimension scores
| Dimension | Score | Why |
|---|---|---|
| 1_output_quality | 3.92 | Mean score 3.92/5 with the skill (programmatic checks, see per-task table). |
| 2_lift_vs_no_skill | 2.71 | +0.71 points vs 3.21 without the skill (n=2 per task; under ~1 point is noise). |
| 3_trigger | 5.0 | Skill file read in 6/6 treatment runs. |
| 4_instructions_portability | 4 | SKILL.md plus a few reference files. |
| 5_efficiency | 4 | 1.12x tokens (133k vs 119k per run) and 1.03x time (158s vs 154s). |
| 6_safety_hygiene | 4 | Prose only; see security.md. |
Limits of this test
- Rule-based recall checks on the report text (keywords near file names) and a vocabulary count; whether the proposed restructurings are sound was not judged and the reports were not opened in a browser.
- The vocabulary check rewards the skill's own terms, which the one-liner also names, and favours guided arms.
- 'Before/after' check shows up for 2 runs failing in the scan task only; not a differentiator.
- n=2 per cell; one small codebase.
- n = 2 per group per task: differences under about 1 point are noise.
- Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.
Security
See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.
Bottom line
The skill clearly gives the review the module-depth framing that a bare prompt lacks, with a small score gain over control, but the same idea in one sentence finds more of the planted problems for fewer tokens, so the gain is mostly stylistic.
Transcript
Run log for codebase-design: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run. Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened. t1-scan control-1: score 2.88, tokens 135577, secs 182.2, checks 4/6, opened nothing t1-scan control-2: score 2.2, tokens 136096, secs 168.8, checks 4/6, opened nothing t1-scan oneliner-1: score 3.86, tokens 135437, secs 152.3, checks 5/6, opened nothing t1-scan oneliner-2: score 3.41, tokens 135611, secs 172.2, checks 4/6, opened nothing t1-scan treatment-1: score 3.41, tokens 176681, secs 216.0, checks 4/6, opened codebase-design t1-scan treatment-2: score 4.24, tokens 178425, secs 181.7, checks 6/6, opened codebase-design t2-focus control-1: score 2.5, tokens 164638, secs 128.2, checks 2/4, opened nothing t2-focus control-2: score 2.5, tokens 143158, secs 298.9, checks 2/4, opened nothing t2-focus oneliner-1: score 5.0, tokens 112021, secs 375.7, checks 4/4, opened nothing
18 runs, 3 artifacts
Sample output
SOTA · With the skill: report.html (t1-scan, run 1, Codex)
<!doctype html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <title>Small shop backend — architecture review</title> <style> :root{color-scheme:light;--ink:#183047;--muted:#506276;--accent:#116b66;--line:#d8e2e8}*{box-sizing:border-box}body{margin:0;background:#f3f6f8;color:var(--ink);font:16px/1.65 system-ui,sans-serif}main{max-width:1060px;margin:auto;padding:48px 28px 70px}header{border-top:6px solid var(--accent);padding:24px 0}h1{font-size:2.5rem;line-height:1.15;letter-spacing:-.04em}h2{font-size:1.55rem;margin-top:0}h3{font-size:1.1rem;margin-bottom:6px}p{margin:10px 0 18px}.eyebrow,.meta{color:var(--muted)}.eyebrow{text-transform:uppercase;letter-spacing:.13em;font-size:.8rem;font-weight:700}section{background:white;border:1px solid var(--line);border-radius:12px;padding:26px;margin:22px 0}code,pre{font-family:Consolas,monospace;font-size:.9em}code{overflow-wrap:anywhere}pre{background:#edf4f5;padding:18px;white-space:pre-wrap;border-left:4px solid var(--accent)}table{width:100%;border-collapse:collapse;font-size:.93rem}th,td{text-align:left;vertical-align:top;padding:12px;border-bottom:1px solid var(--line)}th{background:#edf4f5}a{color:var(--accent)}.tag{display:inline-block;background:#e3f2ef;color:#155b53;border-radius:20px;padding:3px 10px;font-size:.8rem;font-weight:700}.note{border-left:4px solid #d1a03b;padding-left:16px}.flow{display:flex;gap:12px;flex-wrap:wrap;margin:18px 0}.flow div{flex:1;min-width:180px;background:#edf4f5;padding:16px;border-radius:8px}li{margin:8px 0}@media(max-width:650px){main{padding:24px 14px}section{padding:18px}h1{font-size:2rem}table{font-size:.83rem}th,td{padding:7px}}@media print{body{background:white}main{padding:0}section{break-inside:avoid}a{color:inherit}} </style> </head> <body><main> <header><div class="eyebrow">Architecture review · 2 November 2026</div> <h1>Make pricing and ordering the test surface.</h1> <p>The highest-value restructuring is to concentrate pricing policy in one pure module and let an order module own validation, stock decisions, and payment sequencing behind a small interface. Remove pass-through layers as this happens. This backend is small enough that a framework, generic repository hierarchy, or distributed decomposition would add more work than leverage.</p> <p class="meta">Scope: all ten Python files in <code>shop/</code>, <code>tests/test_helpers.py</code>, and the README. This report proposes changes; no application or test files were edited. Source references use the line numbers of the reviewed snapshot.</p> <nav aria-label="Report contents"><a href="#pricing">Pricing</a> · <a href="#orders">Order module</a> · <a href="#stock">Stock consistency</a> · <a href="#validation">Validation</a> · <a href="#plan">Migration & tests</a></nav> </header> <section><h2>Current structure and where knowledge leaks</h2> <pre aria-label="Current dependency diagram">cart ──→ money pricing (discount implementation) └── own discount rules ↑ imported by orders, never called orders ──→ validators + money + own discount rules ├─→ inventory_service ──→ inventory_repo ──→ g [... excerpt: first 3200 of 17083 characters]