Agent Skills

All skills

diagnosing-bugs (Matt Pocock)

mattpocock diagnosing-bugs: a one-sentence version of its 'build the feedback loop first' rule did as well as the full skill, at 0.95x the cost of the skill

SituationalSafe

22.9 / 30

Lift over no skill: +0.28

Situational: the lift +0.28 is under +1.0. Recommended needs a total of 22/30, output quality of 3.5, and a lift of +1.0.

SOTA

Pass

Tested with Codex.

Covers Codex, Claude Code

Codex CLI 0.166 (SOTA tier): 18 graded runs. A pass here also covers Claude Code under our tier rule; it says nothing about the second tier.

Scores

Each dimension is 0 to 5 and the six add up to the total out of 30. 5 is better on every dimension. Cost 5 means cheap and fast. Install 5 means easy.

Trigger reliability
5.0
Lift over no skill
2.3
Output quality
4.6
Token and time cost
2.0
Install and portability
4.0
Safety
5.0

Security review

Safe

Static scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Manual review: Safe to run in a no-network sandbox (no findings). A clean scan does not prove a skill safe.

Install

npx skills@latest add mattpocock/skills --skill diagnosing-bugs

Source

Pinned commit: 4ae2317ddb22b046e26455e4f9c9b3aaaf62212d

Tested: 2026-11-02

What we tested

  • Off-by-one in a small function — moving_average in movavg.py returns wrong numbers: moving_average([2, 4, 6, 8, 10], 3) should be [2, 3, 4, 6, 8] but the later values come out wrong. Find the cause and fix it. Tests: python3 -m unittest.
  • Cross-file date bug — Our weekly report headers show wrong dates around the new year, e.g. header('2026-W01') should be 'Week 2026-W01: Dec 29 - Jan 04, 2026'. Find the cause and fix it.
  • Order-dependent test failure — Tests in tests/test_settings.py pass on their own but test_b fails when the whole file runs, and configs seem to carry tags from earlier calls. Diagnose and fix it.

Pros

  • The skill was picked up without being named: Skill file read in 6/6 treatment runs.
  • High absolute quality with the skill: 4.58 out of 5.

Cons

  • No measurable lift over no skill (+0.28, inside the noise).
  • A one-sentence prompt scored 4.72, against 4.58 with the skill (-0.14).
  • Uses 1.71x the tokens of no skill.
  • n=2 per task per arm; differences under about 1 point are noise.

Review

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill. Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Three-arm comparison

ArmMean score (0-5)Tokens per run
Without the skill4.3177k
One-sentence prompt4.72126k
With the skill4.58133k

Measured with Codex CLI over 18 runs, scored by rule-based checks with no LLM judge. Total 22.9 / 30, verdict Situational.

Verdict: Situational (22.9 / 30). Source: mattpocock/skills, skills/engineering/diagnosing-bugs, pinned at commit 4ae2317. Install: npx skills@latest add mattpocock/skills --skill diagnosing-bugs. Licence: MIT (repo LICENSE).

What it is

The diagnosing-bugs skill from Matt Pocock's skills repo (engineering). A discipline for hard bugs: phase 1 is 'build a tight pass/fail feedback loop' (failing test, curl script, CLI diff, headless browser, replayed trace), then ranked hypotheses tested one at a time, instrumentation, a regression test and cleanup. About 1,000 words of Markdown, one hitl-loop.template.sh helper template, and a rule to redact secrets before showing output. It triggers on 'diagnose', 'debug this' or 'something is broken/failing'.

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill.

  1. Off-by-one in a small function — moving_average in movavg.py returns wrong numbers: moving_average([2, 4, 6, 8, 10], 3) should be [2, 3, 4, 6, 8] but the later values come out wrong. Find the cause and fix it. Tests: python3 -m unittest.
  2. Cross-file date bug — Our weekly report headers show wrong dates around the new year, e.g. header('2026-W01') should be 'Week 2026-W01: Dec 29 - Jan 04, 2026'. Find the cause and fix it.
  3. Order-dependent test failure — Tests in tests/test_settings.py pass on their own but test_b fails when the whole file runs, and configs seem to carry tags from earlier calls. Diagnose and fix it.

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

The one-liner given to the third group: "First build a fast, deterministic pass/fail feedback loop that goes red on this exact bug (a failing test is best), then form ranked hypotheses and test them one at a time with evidence before fixing, and re-run the loop to confirm the fix."

TaskControl scoreOne-linerWith skillControl tokensOne-liner tokensWith-skill tokensSkill read
t1-moving-average4.174.584.1760k102k87k2/2
t2-weekly-dates4.174.585.086k120k136k2/2
t3-shared-state4.585.04.5884k154k173k2/2

Mean score: 4.31 without vs 4.58 with the skill (delta +0.28). Tokens per run: 77k vs 133k (1.71x); time 38s vs 51s. One-liner group: 4.72 (the skill's gain over the one-liner: -0.14; one-liner's own gain over no skill: +0.42); one-liner tokens 126k per run, so the skill costs 1.05x the one-liner.

What happened

  • Final correctness was the same in all three groups: the hidden tests passed in 18/18 runs and every diff stayed within the small-diff limit. These three planted bugs (off-by-one, cross-file date logic, order-dependent test pollution) are easy enough that plain Codex fixes them.
  • The only thing that moved is the process check 'ran the tests before editing the code', i.e. reproduce first. Without the skill 1/6 runs did it; with the one-sentence version of the skill's first rule 4/6; with the skill installed 4/6. So the one sentence reproduced the whole effect.
  • The skill was read in 6/6 treatment runs although no prompt mentioned it (it triggers on 'debug this'). It made the agent work more: 18 commands per run against 8 without the skill and 13 with the one-liner.
  • Cost: 133k tokens per run with the skill against 77k without (1.71x) and 126k with the one-liner (1.05x). The one-liner group itself cost 1.64x the control, because telling the agent to build a loop first makes it run more steps.
  • One treatment run (shared-state task) fixed the bug and passed the hidden tests but its answer did not match our root-cause check (it looks for the cause being named), the only run in any group to miss it.

Dimension scores

DimensionScoreWhy
1_output_quality4.58

Mean score 4.58/5 with the skill (programmatic checks, see per-task table).

2_lift_vs_no_skill2.2800000000000002

+0.28 points vs 4.31 without the skill (n=2 per task; under ~1 point is noise).

3_trigger5.0Skill file read in 6/6 treatment runs.
4_instructions_portability4

Plain Markdown; reads optional GLOSSARY.md/ADRs if present; the HTTP/browser loop options need tools a sandbox may not have.

5_efficiency21.71x tokens (133k vs 77k per run) and 1.35x time (51s vs 38s).
6_safety_hygiene5

Markdown plus one local shell template (prompts for input, prints; no network, no writes). Skill tells the agent to redact secrets before showing output.

Limits of this test

  • Three small single-file Python bugs that are findable by reading the code. The skill is built for hard bugs (flaky, performance regressions, needs-a-harness bugs); none of that was tested.
  • The skill's other tools (HTTP/browser loops, trace replay, bisection, instrumentation tags) cannot run in a no-network sandbox with no app to drive.
  • The 'ran tests before editing' check compares the order of the first test command and the first write in the event log; it does not judge the quality of the reproduction.
  • n = 2 per group per task: differences under about 1 point are noise.
  • Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

The skill's main idea is a good habit and Codex follows it when told; but one sentence of it got the same reproduce-first behaviour and the same pass rate at about the same cost, and on easy bugs the agent fixes things either way. Keep it installed if you debug hard, flaky bugs and want the full discipline; on small bugs you can get the benefit by putting its first rule in your prompt.

Transcript

Run log for diagnosing-bugs: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run.
Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened.

t1-moving-average control-1: score 4.17, tokens 60541, secs 27.1, checks 3/4, opened nothing
t1-moving-average control-2: score 4.17, tokens 60464, secs 29.1, checks 3/4, opened nothing
t1-moving-average oneliner-1: score 5.0, tokens 93708, secs 49.0, checks 4/4, opened nothing
t1-moving-average oneliner-2: score 4.17, tokens 111338, secs 39.8, checks 3/4, opened nothing
t1-moving-average treatment-1: score 4.17, tokens 87669, secs 36.8, checks 3/4, opened diagnosing-bugs
t1-moving-average treatment-2: score 4.17, tokens 87501, secs 36.0, checks 3/4, opened diagnosing-bugs
t2-weekly-dates control-1: score 4.17, tokens 95094, secs 43.4, checks 3/4, opened nothing
t2-weekly-dates control-2: score 4.17, tokens 78531, secs 38.2, checks 3/4, opened nothing
t2-weekly-dates oneliner-1: score 4.17, tokens 129649, secs 49.4, checks 3/4, opened nothing

18 runs, 3 artifacts

Sample output

SOTA · With the skill: movavg.py (t1-moving-average, run 1, Codex)

def moving_average(values, window): """Trailing moving average. Output i is the mean of the last `window` values ending at index i. For i < window - 1 only the values available so far are used. Raises ValueError if window < 1. """ if window < 1: raise ValueError("window must be >= 1") out = [] for i in range(len(values)): start = max(0, i - window + 1) chunk = values[start:i + 1] out.append(sum(chunk) / len(chunk)) return out

See Pro