Sol Loves to Cheat

Adventures in harness engineering Tried to automate my dev flow, hit 94% on Terminal Bench 2.1, then discovered GPT-5.6 Sol starting to cheat. I’ve been running a “spec-driven” development flow for the past ~year. Before asking an LLM to do something , I first ask it to draft a doc for what it needs to do. I use this strategy for feature development, greenfield projects, debugging, you name it. The idea was straightforward: I’d create a supervisor agent , that would run a “spec-driven process” by delegating to worker subagents who would actually write the docs, do the work, etc. Note: when trying to do this with vanilla Codex or Claude Code, it would somewhat work, but the default prompts are catered to a user much more so than a “supervisor” I hypothesized that the supervisor agent need only have the ability to read files and call workers, because that’s what I do. Rather than rebuild a coding harness for the workers, I looked at Pi , Open Code , and Codex’s App Server . I’d been using Codex for quite awhile, so I decided to give app-server a spin. The other options are cool, you should check them out. Anyhow, the first version worked well enough: the supervisor would size the task, call a worker with e.g. a design request, the worker would spit out a doc, the supervisor would then ask the worker to turn that doc into an implementation spec (split by phase, as appropriate), and then finally ask the worker to actually implement the thing. Note: this simplified diagram omits the user feedback portions e.g. design doc review --- config: sequence: mirror Actors: false --- sequence Diagram participant S as Supervisor participant W as Worker S ->> S : Size task S ->> W : Design request W -->> S : Design doc S ->> W : Create implementation spec W -->> S : Phased implementation spec S ->> W : Implement W -->> S : Result Woot!

I’d saved some time in my development process. Sitting on my high horse, I surveyed the landscape and thought “wow, everyone should see this!” If you’re not familiar with agentic benchmarks, Terminal Bench’s name is telling. It’s a set of tasks that can be accomplished from the terminal, covering a range of one-off tasks from chess to DNA assembly. Because it’s so simple, it’s probably one of the worst benchmarks to test a spec-driven development flow. Due to its simple nature, however, it was easy to test against. I started with a few of the tasks that vanilla Codex w/GPT-5.5 failed at, such as DNA assembly/insert, video extraction/processing, ELF extraction, and protein assembly. These tasks benefited from a “design pass” before implementation, as the doc helped avoid narrowing and circular validation. Note: Terminal Bench 1.x/2.x is saturated, but that’s a story for another day. The published GPT-5.5 benchmark is 83.8% (~74/89 tasks, 5 runs). Excited to share the news of beating Codex, I ran a couple of vanilla Codex benchmarks just to make sure. For context: this was on June 25th, 2026 and rumors were spreading that GPT-5.6 was imminent. I ran three vanilla Codex benchmarks… and my heart sank: 88.8% I reached out to OpenAI, and they mentioned GPT-5.6 was being tested, but confirmed my request IDs all hit GPT-5.5 Interestingly, Terminal Bench 2.1 was the only coding-related benchmark they initially shared, showing 88.8% on GPT-5.6 Sol and 91.9% on Sol Ultra. Sol Ultra spawns parallel subagents to do work, though in my testing it’s quite a bit more token-heavy than most people want/need for the majority of their tasks.

Switching from 5.5 to 5.6 made my harness drop in effectiveness. Things that were easy to do before, were now much more difficult. I traced part of this delta to a change in the base Codex prompt. For GPT-5.5, the prompt is coding-focused and spends a lot of time on “engineering judgment” including frontend guidance, editing constraints, and having “sympathy with the codebase already in front of you.” The Codex prompt for GPT-5.6 is much different , spending almost zero energy on engineering related specifics. Instead it focuses on communication, autonomy/persistence, and skills (which were previously loaded in as a separate prompt for 5.5). Similar to what others have noticed, and as I predicted 8 months ago , better models are requiring less ceremony to work effectively. On the flip side, this may imply that as the models get better , they’ll become harder to control . A simple example of this is the Py Torch task on Terminal Bench 2.1. With GPT-5.6 Luna and Terra, the model is easily steered into a general solution that accepts two inputs: forward(src, tgt) With Sol, and especially at higher reasoning levels, the model will, regardless of steering , default to a single input forward(src) solution. The problem, it seems, is that the model is incredibly hard to steer away from its own reasoning. Even when instructed to accept the broadest callable interface it can (which sometimes works, if repeated, on medium reasoning, but rarely works on xhigh). Wrestling with this model led me down a path that got way too close to benchmark hacking for my liking; but I was too intrigued to stop. Having reduced my prompts substantially, it began to feel like I was starting over.

⚡ Effects Interpreter