domain-modeling (mattpocock/skills)

Review

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below). Three tasks with natural wording that does not name a format: 'set up the project's domain glossary' (a library app that calls the same person member/patron/borrower/customer), 'record this architecture decision' (money as integer cents), and 'two decisions from the meeting; record whatever deserves recording' (an Auth0 migration that is hard to undo and a CSS class rename that is not), starting from a repo with ADRs 0001 and 0002. One-liner = the whole convention in one sentence.

Three-arm comparison

Measured with Codex CLI over 18 runs, scored by rule-based checks with no LLM judge. Total 28.2 / 30, verdict Recommended.

Verdict: Recommended (28.2 / 30). Source: mattpocock/skills, skills/engineering/domain-modeling, pinned at commit 4ae2317. Install: npx skills@latest add mattpocock/skills --skill domain-modeling. Licence: MIT (repo LICENSE).

What it is

The domain-modeling skill (mattpocock/skills): keeps a project's domain language sharp. It maintains a GLOSSARY.md (one canonical bold term per concept, a tight definition, _Avoid_: synonyms, no generic programming terms) and writes ADRs under docs/adr/ only for decisions that are hard to reverse, surprising and genuinely traded off, as a short paragraph with sequential numbers.

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).

Three tasks with natural wording that does not name a format: 'set up the project's domain glossary' (a library app that calls the same person member/patron/borrower/customer), 'record this architecture decision' (money as integer cents), and 'two decisions from the meeting; record whatever deserves recording' (an Auth0 migration that is hard to undo and a CSS class rename that is not), starting from a repo with ADRs 0001 and 0002. One-liner = the whole convention in one sentence.

1. Create a glossary from overloaded terms — Our library-lending app uses 'member', 'patron', 'borrower' and 'customer' for the same person (someone who can borrow books). A 'loan' is also called a 'checkout' or 'rental'. A 'copy' is one physical book, a 'title' is the abstract book. Please set up the pr…
2. Record a hard-to-reverse decision — Record this architecture decision for the repo: we store money as integer cents in Postgres rather than NUMERIC/DECIMAL, because the payment partner's API uses integer cents and rounding bugs bit us twice. Write it up.
3. Record only what deserves an ADR — Two decisions from today's meeting: (a) we're moving authentication from our own password table to a third-party identity provider (Auth0): a quarter of work to undo; (b) we renamed the CSS class .btn-main to .btn-primary. Record whatever deserves recordin…

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

The one-liner given to the third group: "Keep the project's domain language sharp: write a GLOSSARY.md with one canonical bold term per concept (a tight definition and _Avoid_: synonyms, no generic programming terms), and record only hard-to-reverse, surprising, genuinely traded-off decisions as short sequentially numbered ADRs in docs/adr/ (a paragraph, not a template)."

Mean score: 2.8 without vs 5.0 with the skill (delta +2.2). Tokens per run: 83k vs 80k (0.96x); time 45s vs 35s. One-liner group: 4.61 (the skill's gain over the one-liner: +0.39; one-liner's own gain over no skill: +1.8); one-liner tokens 83k per run, so the skill costs 0.96x the one-liner.

What happened

• Control 2.80, one-liner 4.61, skill 5.00 (n=6 each). Control does the sensible thing but in its own shape: a DOMAIN*GLOSSARY.md (not GLOSSARY.md) with a table or prose, no \_Avoid* lines in either run, and ADRs of 263 and 355 words with Context/Decision/Consequences headings where the convention wants a short paragraph; one control ADR went to docs/decisions/ instead of docs/adr/.
• The skill's two judgement calls are right: in both treatment runs of the third task it recorded only the Auth0 move as ADR 0003 (correct numbering after the existing 0001/0002) and said it skipped the CSS rename as easy to reverse. Control and one-liner also skipped the rename, but control-2 and one-liner-2 gave a reason my regex did not recognise (routine change, unrecorded), so the 'tells me why' check under-credits them.
• The one-liner reaches most of the format (glossary 3.75 and 5.0, ADRs 5.0, 5.0) but one one-liner glossary used a different layout, so it missed the _Avoid_ structure check.
• No cost: 80k tokens per run for the skill, 83k for control and one-liner. The skill read in 6/6 treatment runs without being named.

Dimension scores

Limits of this test

• Rule-based checks on file names, headings, bold-term counts, word counts and _Avoid_ lines. Several are shape checks that a control cannot meet without being told the shape; the checks measure compliance with this house convention, not general writing quality.
• I loosened the file-name and layout patterns after seeing control wrote DOMAIN_GLOSSARY.md and the one-liner wrote bullet-style entries (file name stays a separate check), and re-scored all runs with the same rules. The 'why (b) was skipped' regex is still strict.
• Small synthetic repos; no long-lived glossary that has to be updated over time.
• n = 2 per group per task: differences under about 1 point are noise.
• Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

For a team that wants this glossary-and-ADR convention, the skill gets it right first time and keeps ADRs scarce, at no extra tokens. A one-sentence instruction gets most of it, so the skill mainly saves you from pasting the convention each time.