Claude Code vs Codex on the Same Ticket: A Measured Test With Hidden Tests
I gave Claude Code and Codex the same CSV-export ticket with nine acceptance criteria and eleven hidden tests. Both passed everything. Here are the times, the setup, and what the test cannot tell you.
I wanted a fair, boring comparison: same ticket, same repository state, one attempt each, scored by tests the agents never saw.
What was the ticket?
Implement exportOrdersCsv(orders) in a small Node project. The ticket lists nine acceptance criteria: a lazy generator that yields one CSV line per chunk, header-only output for empty input, streaming (an infinite input must work if the caller stops early), totals in dollars from integer cents, ISO 8601 UTC dates without milliseconds, proper CSV quoting, a spreadsheet-injection guard, empty output for an undefined field, and a hard cap of 50,000 rows that throws RangeError on the 50,001st.
The agents saw three visible tests. I kept eleven more tests private, covering laziness, time-zone conversion, injection guards, quoting, and the row cap.
How was it run?
Each tool got the same one-line prompt in its own clean copy of the repository: implement TICKET.md in src/export.js, make the tests pass, add no dependencies. One attempt each, no retries. Claude Code ran in non-interactive mode and Codex ran with a workspace-write sandbox. I scored both with the full 14-test suite afterwards.
What were the results?
| Claude Code | Codex | |
|---|---|---|
| Time | 36 s | about 50 s |
| Hidden + visible tests | 14 / 14 | 14 / 14 |
| Cost reported | about $0.20 | not reported in dollars |
| Extra verification | ran the 3 visible tests | wrote its own checks for laziness and the 50,000-row cap |
The two implementations were also close in size (38 vs 44 lines). A tie on correctness. Claude Code was faster here, and Codex did more of its own verification before stopping. See the raw results page.
What does this not show?
It was one run, one small ticket, one machine, and wall-clock time included tool start-up. Token counts are reported differently by the two tools, so I did not compare them. Both tools succeeded, so the test cannot rank them on harder work.
What did I take from it?
When a ticket has explicit acceptance criteria and tests, the choice of agent mattered less than the quality of the spec. Both tools followed the criteria closely. The failure modes I worry about, such as vague tickets, missing tests and silent scope changes, are the ones this experiment removed on purpose. The next test I want to run is a messy legacy bug with no spec.
FAQ
- Which is better, Claude Code or Codex?
- This test does not say. It is one run on one small ticket where both tools succeeded. A messy ticket with no spec would likely separate them more, and I have not tested that.
- What are hidden tests and why use them?
- Hidden tests are tests the agent never sees. They check whether the implementation satisfies the written requirements, not just the visible examples.
- Can I reproduce this?
- Yes. The ticket, the visible tests and the hidden tests are small. Copy the same ticket into two clean directories, give each tool the same prompt, and run the hidden tests afterwards.
Join the Newsletter
Subscribe to get my latest content by email.
We won't send you spam. Unsubscribe at any time.