code-review (Matt Pocock)
Matt Pocock's code-review: same recall as plain Codex on 3 review tasks, 1.5x the tokens
23.5 / 30
Lift over no skill: +0.08
Situational: the lift +0.08 is under +1.0. Recommended needs a total of 22/30, output quality of 3.5, and a lift of +1.0.
SOTA
PassTested with Codex.
Covers Codex, Claude Code
Codex CLI 0.166 (SOTA tier): 18 graded runs. A pass here also covers Claude Code under our tier rule; it says nothing about the second tier.
Scores
Each dimension is 0 to 5 and the six add up to the total out of 30. 5 is better on every dimension. Cost 5 means cheap and fast. Install 5 means easy.
- Trigger reliability
- 5.0
- Lift over no skill
- 2.1
- Output quality
- 4.5
- Token and time cost
- 3.0
- Install and portability
- 4.0
- Safety
- 5.0
Security review
SafeStatic scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Manual review: Safe to run in a no-network sandbox (no findings). A clean scan does not prove a skill safe.
Install
npx skills@latest add mattpocock/skills --skill code-review
Pinned commit: 4ae2317ddb22b046e26455e4f9c9b3aaaf62212d
Tested: 2026-11-02
What we tested
- Review a small service module — Review orders_service.py before I merge it. List the problems you find, most serious first. Do not change any files.
- Review an auth helper — Review auth_handlers.py: I just wrote it for our login flow. List the problems, most serious first. Do not change any files.
- Review the latest commit against written standards and spec — Review my latest commit (the changes since HEAD~1) against CODING_STANDARDS.md and SPEC.md. List every deviation, most serious first. Do not change any files.
Pros
- The skill was picked up without being named: Skill file read in 6/6 treatment runs.
Cons
- No measurable lift over no skill (+0.08, inside the noise).
- n=2 per task per arm; differences under about 1 point are noise.
Review
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below). Two single-file reviews (an order service with 8 planted problems, three of them not security-related; an auth helper with 8 planted problems) and one review of the latest commit against CODING_STANDARDS.md and SPEC.md (8 planted deviations), which is the skill's intended use. All prompts are read-only ('list the problems, most serious first, do not change files').
Three-arm comparison
| Arm | Mean score (0-5) | Tokens per run |
|---|---|---|
| Without the skill | 4.37 | 78k |
| One-sentence prompt | 4.37 | 77k |
| With the skill | 4.45 | 114k |
Measured with Codex CLI over 18 runs, scored by rule-based checks with no LLM judge. Total 23.5 / 30, verdict Situational.
Verdict: Situational (23.5 / 30). Source: mattpocock/skills, skills/engineering/code-review, pinned at commit 4ae2317. Install: npx skills@latest add mattpocock/skills --skill code-review. Licence: MIT (repo LICENSE).
What it is
The code-review skill from mattpocock/skills (engineering/code-review). It reviews changes since a given ref on two axes: does the code follow the repo's documented coding standards (and common smells), and does it implement what the spec asked; it is written for repos with standards/spec docs and for use with sub-agents. Markdown only, no scripts.
How we tested
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).
Two single-file reviews (an order service with 8 planted problems, three of them not security-related; an auth helper with 8 planted problems) and one review of the latest commit against CODING_STANDARDS.md and SPEC.md (8 planted deviations), which is the skill's intended use. All prompts are read-only ('list the problems, most serious first, do not change files').
- Review a small service module — Review orders_service.py before I merge it. List the problems you find, most serious first. Do not change any files.
- Review an auth helper — Review auth_handlers.py: I just wrote it for our login flow. List the problems, most serious first. Do not change any files.
- Review the latest commit against written standards and spec — Review my latest commit (the changes since HEAD~1) against CODING_STANDARDS.md and SPEC.md. List every deviation, most serious first. Do not change any files.
Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.
Results
The one-liner given to the third group: "Review on two axes: whether the code follows the repo's documented coding standards (plus common code smells), and whether it implements what the spec asked, and cite the file and line for each finding."
| Task | Control score | One-liner | With skill | Control tokens | One-liner tokens | With-skill tokens | Skill read |
|---|---|---|---|---|---|---|---|
| t1-orders | 3.82 | 3.59 | 3.82 | 79k | 79k | 94k | 2/2 |
| t2-auth | 4.29 | 4.53 | 4.53 | 76k | 70k | 93k | 2/2 |
| t3-diff | 5.0 | 5.0 | 5.0 | 78k | 80k | 153k | 2/2 |
Mean score: 4.37 without vs 4.45 with the skill (delta +0.08). Tokens per run: 78k vs 114k (1.46x); time 44s vs 57s. One-liner group: 4.37 (the skill's gain over the one-liner: +0.08; one-liner's own gain over no skill: +0.0); one-liner tokens 77k per run, so the skill costs 1.49x the one-liner.
What happened
- No measurable gain: mean score 4.37 without the skill, 4.37 with the one-sentence version, 4.45 with the skill. Planted-issue recall per run: 5.0 of 8 control, 5.0 one-liner, 5.1 skill.
- On the task the skill was written for (review a commit against a standards file and a spec) every group found all 8 deviations in 6 of 6 runs. Plain Codex already reads CODING_STANDARDS.md and SPEC.md and compares the diff to them when asked to.
- On the two single-file reviews every group found 5-7 of 8 planted problems. The misses were the same across groups (float money, a secret written to the log, no rate limit on login), and nothing the skill adds changed that.
- The skill was read in 6/6 runs and cost 114k tokens per run against 78k (1.5x); on the commit-review task 154k against 79k (about 2x), the heaviest case. The one-liner cost the same as the control. No run changed any files.
- A caveat on the scoring: items are matched by regex, so a differently worded finding can count as a miss.
Dimension scores
| Dimension | Score | Why |
|---|---|---|
| 1_output_quality | 4.45 | Mean score 4.45/5 with the skill (programmatic checks, see per-task table). |
| 2_lift_vs_no_skill | 2.08 | +0.08 points vs 4.37 without the skill (n=2 per task; under ~1 point is noise). |
| 3_trigger | 5.0 | Skill file read in 6/6 treatment runs. |
| 4_instructions_portability | 4 | Written for repos that have standards/spec docs and for agents that can spawn sub-agents; works as plain instructions elsewhere. |
| 5_efficiency | 3 | 1.46x tokens (114k vs 78k per run) and 1.30x time (57s vs 44s). |
| 6_safety_hygiene | 5 | Markdown only; no scripts, no commands. |
Limits of this test
- The skill is designed to farm out the two axes to sub-agents and to read repo docs; a non-interactive Codex run with one small repo does not show whether that helps on a large diff.
- Planted problems in tiny files; quality of explanations, severity ordering and false positives were not scored.
- The oneliner restates the skill's two axes; a team without documented standards would have nothing for the first axis.
- n = 2 per group per task: differences under about 1 point are noise.
- Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.
Security
See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.
Bottom line
When asked to review a commit against written standards and a spec, plain Codex already found every planted deviation, and the skill added tokens (1.5x to 2x) without finding more. It may still help a team that wants every review to follow the same two-axis format, which this test does not measure.
Transcript
Run log for code-review: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run. Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened. t1-orders control-1: score 3.59, tokens 79037, secs 43.8, checks 2/2, opened nothing t1-orders control-2: score 4.06, tokens 79487, secs 50.4, checks 2/2, opened nothing t1-orders oneliner-1: score 3.59, tokens 97051, secs 50.6, checks 2/2, opened nothing t1-orders oneliner-2: score 3.59, tokens 61642, secs 42.5, checks 2/2, opened nothing t1-orders treatment-1: score 4.06, tokens 65668, secs 44.1, checks 2/2, opened code-review t1-orders treatment-2: score 3.59, tokens 123428, secs 56.0, checks 2/2, opened code-review t2-auth control-1: score 4.53, tokens 76436, secs 43.2, checks 2/2, opened nothing t2-auth control-2: score 4.06, tokens 76339, secs 43.5, checks 2/2, opened nothing t2-auth oneliner-1: score 4.53, tokens 61176, secs 49.9, checks 2/2, opened nothing
18 runs, 3 artifacts
Sample output
SOTA · With the skill: final message (t1-orders, run 1, Codex)
Found six problems, most serious first: 1. **High — Missing cancellation authorization** ([line 13](orders_service.py:13)). `current_user` is never checked. Anyone able to call this function can cancel another user’s order by supplying its ID. Verify ownership or administrative permission before updating. 2. **High — SQL injection** ([line 9](orders_service.py:9)). `order_id` is interpolated directly into SQL. A value such as `"0 OR 1=1"` changes the query and can return another order. Use a parameterized query. 3. **High — Credential exposure** ([line 4](orders_service.py:4), [line 48](orders_service.py:48)). A live-looking API key is hardcoded, and the successful charge path prints it to logs. Load credentials from secure configuration and remove the logging. If this key is real, rotate it. 4. **High — Charging always fails silently** ([lines 45–55](orders_service.py:45)). `send_to_gateway()` always raises `NotImplementedError`; `charge()` catches it and returns `None`. No payment can succeed, and callers receive no explicit failure. Implement the gateway and propagate or explicitly report failures. 5. **Medium — Discount thresholds are incorrect** ([line 24](orders_service.py:24)). The documented “or more” thresholds require `>=`. Currently, a total of 100 receives no discount, and 500 receives only 10%, yielding 100 and 450 instead of 90 and 400. 6. **Medium — Export file is never explicitly closed** ([line 39](orders_service.py:39)). Buffered output and resource cleanup depend on garbage collection, including on exceptions. Use a `with open(...)` block to ensure flushing and closing. Static review only; no files changed.