docx (anthropics/skills)

Review

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below). Environment: both groups got the same offline toolkit copied into the run (the npm docx package, python lxml/defusedxml/openpyxl, and a python shim for python3), because the skill assumes these are preinstalled. LibreOffice exists on the Mac but aborts (exit 134) inside the Codex sandbox, so nobody could render a preview.

Three-arm comparison

Measured with Codex CLI over 12 runs, scored by rule-based checks with no LLM judge. Total 21.4 / 30, verdict Situational.

Verdict: Situational (21.4 / 30). Source: anthropics/skills, skills/docx, pinned at commit ca45fc4. Install: npx skills@latest add anthropics/skills --skill docx. Licence: Proprietary (LICENSE.txt in skill dir): evaluated read-only, not redistributed.

What it is

The docx skill from Anthropic's skills repo. It tells the agent to create Word files with the docx npm library, edit existing ones by unzipping the XML, and read them with pandoc, and it ships helper scripts for tracked changes and comments (comment.py, accept_changes.py), an XSD validator (office/validate.py) and a LibreOffice wrapper (office/soffice.py). SKILL.md says the libraries are "preinstalled".

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).

Environment: both groups got the same offline toolkit copied into the run (the npm docx package, python lxml/defusedxml/openpyxl, and a python shim for python3), because the skill assumes these are preinstalled. LibreOffice exists on the Mac but aborts (exit 134) inside the Codex sandbox, so nobody could render a preview.

1. Create a memo — Create memo.docx: a Word memo titled 'Project Falcon status update' on US Letter paper with three headed sections (Summary, Risks, Next steps), a bulleted list of at least four next steps, and a 3-column table (Task, Owner, Due) with three rows. Use real Word…
2. Tracked change + comment — In contract.docx change the payment term in clause 2 from 30 days to 45 days as a tracked change authored by 'Reviewer', and attach a comment to that clause saying 'Confirmed with finance'. Do not change anything else. Save the result as contract.docx (edit in…
3. Extract table and outline — From report.docx write table.csv (the table, with its header row) and outline.txt (the document's headings, one per line, in order). Do not modify report.docx.

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

Mean score: 5.0 without vs 4.7 with the skill (delta -0.3). Tokens per run: 91k vs 120k (1.32x); time 60s vs 54s.

What happened

• No measurable lift: every run of both groups scored full marks on our structural checks (new memo: US Letter, 3 real headings, 4+ list items, 3-column table, opens in LibreOffice; tracked change by 'Reviewer' plus comment with the other clauses intact; table extracted and outline correct; source untouched).
• The skill is read in 6/6 treatment runs. In the redline task one treatment run used the skill's comment.py to add the comment and tried its XSD validator (the call failed, exit 1); the other only read comment.py --help and edited the XML itself. The control runs edited the XML directly and got the same result.
• Side effect: both treatment redline runs left their working tree in the project (.edit/ or .docx-edit/ with the original copy and about 22 unpacked XML files); control runs left nothing. We count that as clutter (small penalty in the score).
• Cost: about 1.3x the tokens per run overall; the redline task was about 1.9x (the agent reads the skill and the helper scripts first).
• The preview step the skill recommends (convert to PDF with soffice.py, then look at the pages) failed with exit 134 in the sandbox in both treatment tasks that tried it, so its main verification loop did not actually run.

Dimension scores

Limits of this test

• Three tasks that a baseline agent with Python's zipfile can already do; complex layouts (tables of contents, headers/footers, images) and visual quality were not tested.
• We checked structure, not how the page looks in Word.
• Tracked-change correctness was checked with regexes on the XML, not by opening in Word.
• The skill is proprietary (see licence), so only its behaviour is described.
• n = 2 per group per task: differences under about 1 point are noise.
• Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 0 CRITICAL / 1 HIGH / 7 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

No measurable gain over an unaided agent on three ordinary Word tasks, at about 1.3x the tokens and with leftover working files; the skill's own preview step does not run in a sandboxed agent. Its validator and comment helper may still help on harder editing jobs.