Letting harness2 tune Codex on my own site
A coding agent is more than a model. Around it sits a harness: the instructions file, the skills, the helper scripts, the habits it uses to check its own work. A good harness can turn a mediocre agent into a useful one, and most of us write ours by hand, by trial and error.
harness2 from Google Research tries to do that automatically. An improver model watches the agent work on a set of tasks, proposes edits to its harness, tests each edit, keeps what helps and combines the winners into a new harness. Then it compares the new harness with the old one on the same tasks.
Benchmarks are someone else’s problems, though. I wanted to know what it does with mine, so I gave it the repository of this site and OpenAI Codex as the agent.
Short version: everything worked, Codex solved every task without help, and the “improved” harness won nothing and cost more.
A harness is not the climber. It decides how much the climber can safely try. Photo: Kody Dahl on Unsplash
Keeping it in a box
An agent that runs code on its own and an improver that keeps rewriting its instructions are not things I want next to my real keys. So everything ran in a virtual machine with no access to my files, my SSH keys or the server this site lives on. Inside, Codex could do whatever it liked. Outside, nothing could reach it. One line in its instructions forbade deploying, and harness2 was not allowed to remove it.
Turning a website into a test
To tune anything you need a score. I wrote a set of tasks against this site’s code, each with a test script that answers yes or no for every requirement. No model grades the code, only scripts.
The first task, a simple endpoint to publish blog posts, turned out too easy: Codex passed every check on the first try. Fine for Codex, useless for the experiment. If the agent is already perfect, a better harness has nothing to show.
So I made the work bigger and the instructions vaguer. Three steps that build on each other: an MCP server for this blog, so an AI assistant can work with the posts.
- Read. A server that lists posts. 12 checks.
- Write. Creating, editing and deleting posts in English and Spanish, without breaking the site. 14 checks.
- Protect. “Add authentication”, and nothing more. 17 checks, which expect the full login flow the MCP specification describes.
Every requirement is a script with a yes or no answer. No model grades the code. Photo: Aswin Anand on Unsplash
Results
Same scores, more tokens. Outlined bars are the original harness, filled bars the one harness2 tuned.
Both versions got the same scores. The only miss, in the first step, was the same for both and was a nitpick in my own tests rather than a real bug. On the second and third steps the tuned harness made Codex read 67–88% more, and on the second step it took more than twice as long.
The whole improvement loop cost a couple of dollars on the improver’s side. The real price is Codex’s own tokens, and the tuned harness only added to them.
Two things that surprised me
Codex knew what “authentication” means here. In the third step nobody said “OAuth”. Codex looked at an MCP server, decided it needed what the MCP specification asks for, and built it: discovery documents, client registration, PKCE, one-time codes, tokens that expire. All 17 checks, from a two-word request.
The tuner kept its own work anyway. harness2 picked its tuned harness every time, even though it scored no better. Some of its edits were sensible, like keeping the agent’s own test scripts in the project instead of throwing them away. Others were not: it wrote Codex a whole skill on rotating refresh tokens, a topic none of the tasks touched. It tuned against its own idea of what the work might need, not against what the work needed.
What I take from it
The setup is worth keeping: a private project, honest checks, a sealed machine, and an automatic loop on top.
The result is a ceiling effect. With a strong model on tasks of this size, the plain agent already gets everything right, so there is nothing for a better harness to win, only cost to add. Tools like harness2 should pay off where the agent fails, so that is where I am going next:
- A smaller, cheaper model. Can a tuned harness help it catch up with the big one? That would be real savings.
- Tasks that need knowledge of the project: keeping English and Spanish posts in step, the privacy rules of this site, the house code style. Things no model learns in training.
- Making cost count. If tokens and time are part of the score, a harness that only spends more cannot win.