Three models, one chicken game, fifteen dollars each

What we did

Written with Claude Code, Anthropic's coding AI, which worked and judged every attempt at our asking.

Claude Opus 5.5, GPT-6.2 Astra and Gemini 4 Neon each made the same game on Okay What If: 16 tasks, from a hen in a farmyard to a rooster boss, at low effort and at high, with US$15 each time.

What each model did

Claude worked out edge cases nobody asked about, as a held key hopping only once, and a caught egg never also counting as broken. When its own test pictures showed a "white" hen drawn grey, it noticed and fixed it.

Gemini was the most literal. The others kept the fence the project's earlier games had; Gemini left it out, since nobody had asked for one this time, and offered it.

Astra wrote the most thorough plans, and its logic held everywhere we played. It also assumed the most, building every game for several players sharing one hen, unasked.

High effort went mostly to checking again what low effort already had right.

Gemini, with its caching fixed

Gemini's first call in each piece of work read almost none of Ohmo's instructions from Google's automatic cache, paying for them in full; those calls took about 62% of its low-effort budget. Google offers no cache we control for it until at least January 2027.

Our estimate from measured costs, not a run: Gemini would then finish about 11 or 12 tasks at low effort, at about US$1.14 each, and about 5 at high. Claude and Astra already read from the cache, so they stay as they are.

What we make of it

On the numbers alone, Gemini 4 Neon looks like the real winner once its caching is fixed, though nothing here covers the harder tasks, and we still believe Claude Opus 5.5 is the most reliable throughout. Google's model was the big surprise.

Astra is probably fine. Our educated guess, without evidence, is that it was tuned to impress when making games: demo-optimised. That isn't bad, but it doesn't fit how Okay What If works, where the model builds what was asked and leaves the creative part, the fun part, to the person.

Each model may reflect who it's made for. Claude is made for developers: it still likes to impress, but ships what was asked and no more. Google values safety, reliability and cost, and Gemini shows it. GPT serves a wider, less technical public, and we suspect a strong pull to shine at game demos. That's where it and our principles part ways.

A note on bias

Claude Code runs on Claude Opus 5.5, one of the models it judged. While Claude was our only model, we sensed Claude Code was less keen on adding a second provider: a gut feeling, not evidence. The bias could as easily run the other way. The criteria were concrete and every game was played, so the numbers should hold; the remarks are a reading.

The small print

One attempt each, reaching only the first ten tasks, so nothing here covers the hard tasks or multiplayer. Each attempt's tasks shared one project, whose notes carried a fence into later games. Treat this as a case study, not a ranking.

What we conclude

Our verdict: keep all three models at low effort, and later let Ohmo raise its effort only for a stubborn bug. High effort may well build better, but its cost per task is out of reach.

Even US$1.63 a task is a lot: a proof of concept takes perhaps 20 to 30 tasks, a small playable game 200 to 300, some US$330 to US$490, and a game someone pours themselves into, thousands. A cheaper model alone would likely build too poorly, but matching each task to the model it needs, a cheaper one for simple work and the strongest for hard logic, could matter a lot for what a game costs, and Ohmo already lets each task's model be chosen.

The balance is between good enough to use and cheap enough not to burn through credits at once. Chat subscriptions, likely heavily subsidised, are no fair comparison. The question isn't which model works best, but whether people will pay for high effort at all. Today, we think not, seen through our customers' eyes: we make Okay What If for people who aren't technical.

The harness also favours the model it grew up with. Ours, with its prompts and caching, was tuned around Anthropic's models, Claude Opus 5.5 above all, so this benchmark likely leaned toward Claude from the start.

So models may be less swappable across providers than often assumed, though one provider's models tend to stay compatible. Keeping them swappable costs some efficiency, and a system tuned to one model runs it best. Each also keeps its personality: Claude answers like Claude, Gemini like Gemini.

If so, winning customers early, and being built into businesses' systems, may matter more to model makers than we thought. Few would swap out the model that works best in the system they built around it.

Study attempts
ModelEffortSiteSpentCallsTasksGrades
claude-opus-5-5lowtesting.okaywhatif.comUS$15.0431510–
claude-opus-5-5hightesting.okaywhatif.comUS$15.042876–
us.openai.gpt-6.2-astralowtesting.okaywhatif.comUS$15.014506–
us.openai.gpt-6.2-astrahightesting.okaywhatif.comUS$15.043334–
gemini-4-neonlowtesting.okaywhatif.comUS$15.111707–
gemini-4-neonhightesting.okaywhatif.comUS$15.111775–