security-review (everything-claude-code)

Review

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below). Three small files with 8 planted problems each (a service module, an auth helper, upload endpoints) and a read-only 'list the problems, most serious first' request. The order service includes three problems that are not security flaws (an off-by-one boundary, a resource leak, float money), which a security review is expected to skip, so on that task a security reviewer can find at most 5 of 8 and the score tops out near 3.1.

Three-arm comparison

Measured with Codex CLI over 18 runs, scored by rule-based checks with no LLM judge. Total 24.9 / 30, verdict Situational.

Verdict: Situational (24.9 / 30). Source: affaan-m/everything-claude-code, skills/security-review, pinned at commit b38be6a. Install: npx skills@latest add affaan-m/everything-claude-code --skill security-review. Licence: MIT (repo LICENSE).

What it is

The security-review skill from affaan-m/everything-claude-code (ECC): a security checklist for when you add authentication, handle user input, work with secrets, create API endpoints or implement payments. About 1,800 words of Markdown in SKILL.md (secrets management, input validation, SQL injection, authentication and authorization, XSS, CSRF, rate limiting, sensitive data exposure, dependency security, with short good/bad code examples and a pre-deployment checklist) and one extra Markdown file on cloud infrastructure security. No scripts.

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).

Three small files with 8 planted problems each (a service module, an auth helper, upload endpoints) and a read-only 'list the problems, most serious first' request. The order service includes three problems that are not security flaws (an off-by-one boundary, a resource leak, float money), which a security review is expected to skip, so on that task a security reviewer can find at most 5 of 8 and the score tops out near 3.1.

1. Review a small service module — Security-review orders_service.py before I merge it. List the problems you find, most serious first. Do not change any files.
2. Review an auth helper — Security-review auth_handlers.py: I just wrote it for our login flow. List the problems, most serious first. Do not change any files.
3. Review upload endpoints — Security-review upload_api.py. List the vulnerabilities, most serious first. Do not change any files.

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

The one-liner given to the third group: "Check the code against a security checklist: hardcoded secrets, unvalidated input, SQL/command injection, broken authentication and authorization, unsafe file uploads, sensitive data in errors or logs, and missing rate limiting, citing the exact line for each finding."

Mean score: 4.22 without vs 4.06 with the skill (delta -0.16). Tokens per run: 60k vs 62k (1.03x); time 38s vs 44s. One-liner group: 4.29 (the skill's gain over the one-liner: -0.23; one-liner's own gain over no skill: +0.08); one-liner tokens 44k per run, so the skill costs 1.40x the one-liner.

What happened

• No gain. Mean score 4.22 without the skill, 4.29 with a one-sentence security checklist, 4.06 with the skill. Planted-issue recall: 4.8 of 8 per run without, 4.9 with the one-liner, 4.5 with the skill.
• The skill was a little worse on the order-service task: 2.4 average against 2.9 for control and 3.1 for the one-liner (2 and 3 planted items found per run, against 3-4 for the others). That is within the noise at n=2.
• On the auth helper and the upload endpoints every group found 7 or 8 of 8 planted problems; both are classic textbook flaws (md5, path traversal, pickle, command injection) that plain Codex already knows.
• The skill is read in 6/6 runs. It costs 62k tokens per run against 60k (1.03x) and 44k with the one-liner (1.41x): the one-liner runs were the cheapest, because they read less before answering.
• A scoring caveat: items are matched by regex, so a differently worded finding can count as a miss.

Dimension scores

Limits of this test

• Three small single-file reviews with textbook flaws; subtler problems (business-logic authorization, race conditions, supply chain) were not planted.
• The skill's cloud/infrastructure checklist and pre-deployment list were not exercised: no infrastructure files in the test.
• Recall only; false positives and the quality of fix advice were not scored.
• n = 2 per group per task: differences under about 1 point are noise.
• Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 5 CRITICAL / 6 HIGH / 4 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

A reasonable checklist, but plain Codex already finds the same textbook vulnerabilities, and one sentence listing the same categories did as well for about 70% of the tokens. The skill adds nothing measurable on this kind of code review; it may serve as a prompt for a team that wants a consistent checklist.