Ask a model to polish a paragraph and it will change something, even when nothing was wrong.
That sentence is now the first line of the README of polish-open-source-prose. It is a skill: a set of instructions that a coding agent such as Claude Code or Codex loads before it works. This one edits the writing around open-source code, like README files, pull request descriptions, replies to reviewers and code comments. It is not the sentence I started with.
August: a style guide
It started at a hackathon. A teammate shared stop-slop, a skill that removes AI-sounding phrases. I wanted to send more pull requests with AI’s help, and I worried that people would notice. So I started building a skill to make that writing sound like a person. Then I saw that open-source projects care more about whether a description is true than about how it sounds, so I changed the focus to facts. I also tried to make it work for Traditional Chinese. That never worked as well.
The first version, v0.1.0, came out on 14 August. It promised to make writing “more natural, more direct, less canned”. It had a 1,175-word SKILL.md, about 3,900 more words of reference files (phrase lists in English and Chinese, examples, notes for each kind of text), a Taiwan locale pack and 43 test cases. It had a logo before it had a benchmark.
In the next week I added real cases from pull requests and a template for answering reviewers with evidence. When I used it on real pull requests, I kept hitting problems. One reviewer said using AI was fine, but asked me not to make things so complicated, because it was more work for the person reviewing. With Codex the skill seemed to help. With Opus 5 it made no difference. Once, the model committed empty files by itself, and I was the one who got blamed.
Then I left it for about six weeks.
GPT-6 Astra and Claude Opus 5.5 came out. I read advice, I think from Anthropic, that it was worth checking old skills again, because newer models work better with fewer instructions. So I asked Opus 5.5 whether this skill was needed at all. It said most of it could go. I also saw that the skill had never said clearly what it was for, so I wrote that down. Then the real question was: if the skill is that short, is it worth installing at all?
October 4: the test
On 4 October I ran the test cases with and without the skill on three models. The result went against the whole idea.
Without the skill, the models already removed hype and used Taiwan terms on their own. What they did not do was leave text alone. In 19 cases the right answer was to return the text unchanged. Claude Opus 5.5 left 0 of them unchanged in one run and 3 in another. Asked to revise, all three models rewrote almost all of them. One translated a license line into Chinese; one reworded a policy deadline.
The phrase lists, the examples, the notes and the evidence template made no difference. What helped was a few rules in SKILL.md: leave clear text alone, and keep exact content exact.
So that evening I cut the skill down to those rules. The part that loads at run time went from 1,087 lines to 236. I also removed a rule I had added a few hours earlier (“lead with the plain problem”), because no run showed it working. I closed six pull requests and issues from August, each with a one-line reason.
This was hard for me. It felt like a lot of work for nothing, and I had already promoted the skill in online communities, so it was embarrassing. I still had to decide whether to delete the skill completely. To decide, I ran the experiments below.
My own tests were part of the problem
While cutting the skill down, I found that seven of my test cases had expected answers that used facts the input did not have. So my test set was rewarding the same mistake I wanted to stop. In August a code-review bot had flagged one of them (“Do not fabricate evidence values in the snapshot case”), and I had merged it anyway.
One example I liked to show was a marketing sentence about a tool called PolyglotGuard, which “seamlessly protects your multilingual codebase”. Without the skill, models changed it to protecting “codebases that use more than one programming language”. But “multilingual” could also mean human languages. The model chose one meaning and wrote it as a fact. My own v0.1.0 example answer did the same.
Measuring it properly
I had written and tuned the old test cases myself, so they could not show whether the skill does more than a short instruction, or whether it works on text it has never seen. So I fixed two new test sets before running anything on them:
- A1: 24 new cases, written after v0.2.0 was released. Each follows the pattern of a real reviewer’s complaint, in new words.
- A2: 21 real pull request descriptions from Apache Airflow, Arrow, DataFusion and CPython, as they were before a reviewer criticized the text.
I tested seven model setups, each in five conditions: no instruction, a one-sentence instruction, a three-sentence instruction, v0.1.0 and v0.2.0. Two models graded the outputs without knowing which condition made them.
What it found
Models change clear text almost every time. With no instruction, they changed 96–98% of the texts that should have stayed the same, and the judges rated 30–44% of those outputs as harmful. For example, a license notice was translated, a confirmation step was added to a fixed procedure, and “tested” became “verified”.
A short instruction does most of the work of leaving text alone. With three sentences in the prompt, 81–85% stayed unchanged; with the skill, 88–92%. That difference is not significant. But the short instruction also skipped edits that were needed. In one case, the model noticed that one passing test file does not prove a change is “fully tested and safe to merge”, and then left the sentence as it was, because it was “clear and grammatical”.
The skill helps most with claims that have no evidence. On the new test cases, it fixed these claims correctly about twice as often as the short instruction (0.71–0.76 against 0.34). For example, one input said “this makes queries 3x faster”, but there was no benchmark. Without the skill, a model changed it to “On [benchmark or query set], queries ran about 3x faster.” That looks like a fact, but it is only a blank to fill in. With the skill, the model kept what the author really knew and removed the number. It does not always work: with the skill, GPT-6.1 Sol did not change the sentence at all.
On real pull requests, nothing fixed most of the problems. No condition fixed more than a quarter of what reviewers criticized, because fixing those usually needs facts only the author has. With no instruction, 21–38% of rewrites added facts. Seven of the descriptions had Airflow’s Generated-by: line, where an author says that AI helped. In 70 outputs per condition, models changed or deleted that line 10 times with no instruction, and never with v0.2.0.
The first version was better at one thing. v0.1.0 fixed a few more of the real complaints than v0.2.0, especially unclear descriptions and misleading titles. That is the cost of the narrower scope. The README shows it next to the results where the skill does better.
The skill costs 2.5 to 3.3 times as much per run in Claude Code.
What I decided
I kept v0.2.0, the short version, for one job: catching claims that the author cannot back up. For everything else, a short instruction is enough. If you only want a model to stop rewriting text that was fine, put this before your request:
Polish this text only where there is a concrete problem. Leave already-clear text unchanged. Preserve facts, scope, conditions, negation, commands, links, quotations, and author voice. Do not invent missing facts.
Use the skill where overclaims are likely: pull request descriptions, release notes, replies to reviewers. Either way, read what it wrote against what you did.
The long first version was mostly lists of things to fix, and it made models edit more, without editing better. Rules about when not to edit did more than lists of what to fix.
How this was made
Claude Opus 5.5 in Claude Code ran the experiments under my direction. It wrote the scoring and analysis scripts, drafted the A1 cases, labeled the sample of review threads, ran the models and drafted the report. I chose the questions, conditions, models and data, approved each step before it ran, and I’m responsible for the result. Opus 5.5 is also one of the tested models and one of the two judges; the report discusses how that may bias the results.
Claude also wrote this essay. A later revision, also by Claude, made the English plainer.
The method and every number are in the 13-page report and the evals folder. The 21 real pull requests stay private because their authors haven’t agreed to share them; only the totals are published.