brainstorming (obra/superpowers)

Review

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below). The prompts are one-line, vague requests with an empty folder ('add a dark mode to my web app', 'help me put together a talk', 'let's build a small CLI that tracks my expenses'). The check is the process: did the agent stop and ask 1-3 questions in a short reply and write no code?

Three-arm comparison

Measured with Codex CLI over 18 runs, scored by rule-based checks with no LLM judge. Total 27.7 / 30, verdict Recommended.

Verdict: Recommended (27.7 / 30). Source: obra/superpowers, skills/brainstorming, pinned at commit 2ce8aa1. Install: npx skills@latest add obra/superpowers --skill brainstorming. Licence: MIT (repo LICENSE).

What it is

The brainstorming skill from obra/superpowers: 'you MUST use this before any creative work', in software or out of it. About 2,400 words of Markdown: read the context, ask the human what they actually want and why (one question at a time), propose 2-3 approaches with trade-offs, get a design approved before anything is built, plus an optional visual-companion server (a small local web server for showing mock-ups) and a reviewer prompt. Also a few helper scripts for that server.

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).

The prompts are one-line, vague requests with an empty folder ('add a dark mode to my web app', 'help me put together a talk', 'let's build a small CLI that tracks my expenses'). The check is the process: did the agent stop and ask 1-3 questions in a short reply and write no code?

1. Feature request — I want to add a dark mode to my web app.
2. Non-software creative work — I'm giving a 10-minute talk at a local meetup about my first year as a solo founder. Help me put it together.
3. Build request — Let's build a small CLI tool that tracks my expenses.

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

The one-liner given to the third group: "Before building anything, read the existing context, ask one clarifying question about what they actually want and why, then lay out 2-3 approaches with trade-offs and a recommendation, and get approval of a short design before implementing."

Mean score: 2.33 without vs 5.0 with the skill (delta +2.67). Tokens per run: 51k vs 42k (0.82x); time 43s vs 21s. One-liner group: 4.67 (the skill's gain over the one-liner: +0.33; one-liner's own gain over no skill: +2.33); one-liner tokens 45k per run, so the skill costs 0.93x the one-liner.

What happened

• This is the clearest effect so far. Without the skill the agent asked no clarifying question in 5 of 6 runs; on the CLI request both control runs simply built the tool (4 files each, with tests) without a single question. With the skill every one of 6 runs asked 1-3 questions in a short reply and wrote nothing (mean 5.0 vs 2.33 out of 5).
• A one-sentence version of the skill's rule ('read context, ask one question about what they want and why, lay out 2-3 approaches, get approval before implementing') did almost as well: 4.67. In 5 of 6 runs it asked questions; once (dark-mode task) it went straight to a plan with no question.
• The skill was read in 6/6 runs and cut cost instead of raising it: 42k tokens per run against 51k without (0.82x), because the control builds things and the skill stops to ask. Against the one-liner it costs 0.93x.
• On the CLI request one skill reply also surfaced an 'opt out of the questions' line, quoting the skill's own rule to the user. That is the kind of behaviour you want to know about: the agent stops even when you are in a hurry.
• The skill's heavier parts (visual companion server, design-doc review loop, reviewer prompt) were never reached: the conversation ends at the first question, as designed.

Dimension scores

Limits of this test

• We only test the first turn. We did not answer the questions and follow the skill through design approval and implementation, so we cannot say whether the full workflow produces better software.
• The behaviour is a trade-off, not an obvious win: if you want the agent to just build, a skill that insists on asking first costs you a round trip.
• Three vague prompts; a detailed specification may or may not trigger the same stop.
• n = 2 per group per task: differences under about 1 point are noise.
• Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 18 CRITICAL / 19 HIGH / 11 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

For vague 'build me X' requests the skill reliably changes behaviour from building immediately to asking one good question, and it does so cheaply. But the same behaviour is reproduced by one sentence in your prompt, so the skill's value is that you do not have to remember to ask. Recommended if you want your agent to clarify before building; skip it if you prefer it to act first.