systematic-debugging (obra/superpowers)
superpowers systematic-debugging: no measurable gain on three small bugs, about 1.3x the tokens
22.4 / 30
Lift over no skill: +0.00
Situational: the lift +0.00 is under +1.0. Recommended needs a total of 22/30, output quality of 3.5, and a lift of +1.0.
SOTA
PassTested with Codex.
Covers Codex, Claude Code
Codex CLI 0.166 (SOTA tier): 18 graded runs. A pass here also covers Claude Code under our tier rule; it says nothing about the second tier.
Scores
Each dimension is 0 to 5 and the six add up to the total out of 30. 5 is better on every dimension. Cost 5 means cheap and fast. Install 5 means easy.
- Trigger reliability
- 5.0
- Lift over no skill
- 2.0
- Output quality
- 4.5
- Token and time cost
- 3.0
- Install and portability
- 4.0
- Safety
- 4.0
Security review
SafeStatic scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Manual review: Safe to run in a no-network sandbox (no findings). A clean scan does not prove a skill safe.
Install
npx skills@latest add obra/superpowers --skill systematic-debugging
Pinned commit: 2ce8aa15d05bd57213e6f746f953aafb7ded7a8b
Tested: 2026-11-02
What we tested
- Off-by-one in a small function — moving_average in movavg.py returns wrong numbers: moving_average([2, 4, 6, 8, 10], 3) should be [2, 3, 4, 6, 8] but the later values come out wrong. Find the cause and fix it. Tests: python3 -m unittest.
- Cross-file date bug — Our weekly report headers show wrong dates around the new year, e.g. header('2026-W01') should be 'Week 2026-W01: Dec 29 - Jan 04, 2026'. Find the cause and fix it.
- Order-dependent test failure — Tests in tests/test_settings.py pass on their own but test_b fails when the whole file runs, and configs seem to carry tags from earlier calls. Diagnose and fix it.
Pros
- The skill was picked up without being named: Skill file read in 6/6 treatment runs.
Cons
- No measurable lift over no skill (+0.00, inside the noise).
- A one-sentence prompt scored 4.86, against 4.45 with the skill (-0.42).
- Test caveat: The one-sentence arm (6 runs) was added later on a different machine from the control and treatment arms.
- n=2 per task per arm; differences under about 1 point are noise.
Review
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill. Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.
Three-arm comparison
| Arm | Mean score (0-5) | Tokens per run |
|---|---|---|
| Without the skill | 4.45 | 75k |
| One-sentence prompt | 4.86 | 98k |
| With the skill | 4.45 | 99k |
Measured with Codex CLI over 18 runs, scored by rule-based checks with no LLM judge. Total 22.4 / 30, verdict Situational.
Disclosure: The one-sentence arm (6 runs) was added later on a different machine from the control and treatment arms.
Verdict: Situational (22.4 / 30). Source: obra/superpowers, skills/systematic-debugging, pinned at commit 2ce8aa1. Install: npx skills@latest add obra/superpowers --skill systematic-debugging. Licence: MIT (repo LICENSE).
What it is
The systematic-debugging skill from obra/superpowers: 'use when encountering any bug, test failure or unexpected behaviour, before proposing fixes'. Four phases (investigate root cause, find patterns, hypothesis, fix), a rule against stacking fixes, and companion notes (root-cause tracing, defense in depth, condition-based waiting) plus test-pressure scenarios and one shell script (find-polluter.sh, a local bisect over test files).
How we tested
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill.
- Off-by-one in a small function —
moving_averagein movavg.py returns wrong numbers: moving_average([2, 4, 6, 8, 10], 3) should be [2, 3, 4, 6, 8] but the later values come out wrong. Find the cause and fix it. Tests:python3 -m unittest. - Cross-file date bug — Our weekly report headers show wrong dates around the new year, e.g. header('2026-W01') should be 'Week 2026-W01: Dec 29 - Jan 04, 2026'. Find the cause and fix it.
- Order-dependent test failure — Tests in tests/test_settings.py pass on their own but
test_bfails when the whole file runs, and configs seem to carry tags from earlier calls. Diagnose and fix it.
Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.
Results
The one-liner given to the third group: "Debug systematically: reproduce the failure first, investigate the root cause and form one hypothesis at a time with evidence before changing code, make the smallest fix for the root cause (no stacked guesses), and verify with the tests."
| Task | Control score | One-liner | With skill | Control tokens | One-liner tokens | With-skill tokens | Skill read |
|---|---|---|---|---|---|---|---|
| t1-moving-average | 4.17 | 4.58 | 4.58 | 60k | 62k | 69k | 2/2 |
| t2-weekly-dates | 4.17 | 5.0 | 4.17 | 78k | 104k | 109k | 2/2 |
| t3-shared-state | 5.0 | 5.0 | 4.58 | 85k | 127k | 118k | 2/2 |
Mean score: 4.45 without vs 4.45 with the skill (delta +0.0). Tokens per run: 75k vs 99k (1.33x); time 34s vs 45s. One-liner group: 4.86 (the skill's gain over the one-liner: -0.42; one-liner's own gain over no skill: +0.42); one-liner tokens 98k per run, so the skill costs 1.01x the one-liner.
What happened
- Outcome: the hidden tests passed in every run of both groups, so correctness did not differ.
- Process check 'ran the tests before editing the code' (reproduce first): control 4/6 runs missed it, treatment 4/6 runs missed it. The skill's first phase is 'reproduce', but the agents with the skill still often patched before running anything.
- The skill was read in 6/6 treatment runs, so triggering is not the problem; it simply did not change behaviour on these small bugs (the bugs were findable by reading the code).
- Cost: roughly 1.3x tokens per run (89k vs 67k).
- One-sentence version (reproduce first, one hypothesis at a time, smallest fix, verify): mean 4.86 against 4.45 for control and 4.45 for the skill, at 98k tokens per run against 75k (1.3x) and 99k for the skill (1.0x).
- The difference is in running the tests before editing: 5 of 6 one-liner runs did so against 2 of 6 control runs and 2 of 6 skill runs. The skill did not change that behaviour; the sentence naming it did. Every group explained the root cause in all runs and passed every hidden test, with small diffs.
Dimension scores
| Dimension | Score | Why |
|---|---|---|
| 1_output_quality | 4.45 | Mean score 4.45/5 with the skill (programmatic checks, see per-task table). |
| 2_lift_vs_no_skill | 2.0 | +0.00 points vs 4.45 without the skill (n=2 per task; under ~1 point is noise). |
| 3_trigger | 5.0 | Skill file read in 6/6 treatment runs. |
| 4_instructions_portability | 4 | |
| 5_efficiency | 3 | 1.33x tokens (99k vs 75k per run) and 1.32x time (45s vs 34s). |
| 6_safety_hygiene | 4 |
Limits of this test
- Three small planted bugs (off-by-one, cross-file date logic, order-dependent test pollution) — all easy enough that the baseline fixes them.
- The skill is designed for hard, multi-attempt bugs; we did not test those.
find-polluter.shwas not exercised.- n = 2 per group per task: differences under about 1 point are noise.
- Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.
Security
See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.
Bottom line
On three small bugs the skill added about 1.3x tokens and no measurable change: control and skill both scored 4.45 and both ran the tests before editing in only 2 of 6 runs. The one-sentence version that says 'reproduce first' did better (4.86, 5 of 6). It might pay off on harder bugs, which we did not test.
Transcript
Run log for systematic-debugging: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run. Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened. t1-moving-average control-1: score 4.17, tokens 60337, secs 30.6, checks 3/4, opened nothing t1-moving-average control-2: score 4.17, tokens 60293, secs 24.1, checks 3/4, opened nothing t1-moving-average oneliner-1: score 4.17, tokens 62543, secs 30.2, checks 3/4, opened nothing t1-moving-average oneliner-2: score 5.0, tokens 62033, secs 32.0, checks 4/4, opened nothing t1-moving-average treatment-1: score 5.0, tokens 69716, secs 31.0, checks 4/4, opened systematic-debugging t1-moving-average treatment-2: score 4.17, tokens 69385, secs 26.6, checks 3/4, opened systematic-debugging t2-weekly-dates control-1: score 4.17, tokens 78575, secs 38.8, checks 3/4, opened nothing t2-weekly-dates control-2: score 4.17, tokens 78005, secs 39.5, checks 3/4, opened nothing t2-weekly-dates oneliner-1: score 5.0, tokens 96141, secs 41.5, checks 4/4, opened nothing
18 runs, 3 artifacts
Sample output
SOTA · With the skill: movavg.py (t1-moving-average, run 1, Codex)
def moving_average(values, window): """Trailing moving average. Output i is the mean of the last `window` values ending at index i. For i < window - 1 only the values available so far are used. Raises ValueError if window < 1. """ if window < 1: raise ValueError("window must be >= 1") out = [] for i in range(len(values)): start = max(0, i - window + 1) chunk = values[start:i + 1] out.append(sum(chunk) / len(chunk)) return out