Guides / Building software with AI agents

Two AI models, one repository: early speed and quality data

Measured results from the first 25 hours of Claude and Codex reviewing each other: pull requests, review findings and test counts. Early, thin data.

Engineering note

By Steven Michael Barber · Published October 19, 2026 · Updated November 3, 2026 · 6 min read

This is a snapshot of our first day using two AI coding models in one repository: Claude, from Anthropic, and Codex, from OpenAI. The setup is described in the initial setup guide and the rules we changed in the changes guide.

At the snapshot, 8:46 a.m. EDT on October 19, 2026, the arrangement was about 25 hours old, and the split with Claude building and Codex reviewing was about an hour old. That is too little data to show an improvement in speed or quality, and we do not claim one. What follows is a dated baseline, with the method for each number, rather than a report of the current workflow.

Where the numbers come from

• Pull requests come from GitHub's list of the repository's pull requests. We attribute each one to a model by its branch name, following the naming each model's sessions use.
• Review findings are the inline comments each model left on the other's pull requests, and the replies to them.
• Test counts come from the verification records that a full CI run writes for each production release.
• Effort and cost would come from our effort ledger, a measured-only record updated twice a week from the platform's session records. At the snapshot it had nothing on this period, for reasons explained below.

Every count was made by querying those sources with a script at the time of the snapshot, not by hand. The records are in our private repository, and the counting scripts were not retained in it. The tables here reproduce the snapshot; they are not a public reproduction kit.

Output: pull requests

From Codex's first pull request, at 7:49 a.m. EDT on October 18, to the snapshot, 26 pull requests were opened:

Most of these were records and documents rather than code. The one Codex pull request closed without merging was a duplicate: both models wrote up the same owner decision two minutes apart.

Did output go up? We cannot say yet. The days before the change are whole calendar days with one model, while the new arrangement had run about 25 hours at the snapshot, starting partway through one day and ending partway through the next. Counts over windows that unequal are not comparable, and with one window on each side there is nothing to average. Pull request counts are a poor measure anyway, because the size and kind of work vary a great deal from day to day. We will compare equal windows, week against week, once there are several weeks of each.

Quality: what each reviewer found

P1 and P2 are the priority labels Codex puts on its findings, with P1 the more serious. Claude replied to every Codex finding, and the median delay from finding to reply was 100 seconds. Fourteen replies came within two minutes and the other four after about 19 to 20 minutes. Claude did not dispute any of them. In one case it agreed with the problem but not with fixing it in that pull request, because the fix would have touched release workflows, which only the owner may change. It left the thread open and put the choice to the owner.

Three cautions apply. A fast reply is not proof of a careful one, although each reply named the code it had checked. A count of findings says nothing about what neither reviewer noticed, and that number we cannot measure yet. And the seven Codex pull requests merged on October 18, before the rules changed, have no review recorded on them. The owner merged them directly.

Tests and releases

• The last release record before the rule changes, verified by Codex, covered 9,798 tests: 9,795 passed, 0 failed and 3 skipped.
• The next one, verified by Claude, covered 9,903 tests: 9,900 passed, 0 failed and 3 skipped. That is 105 more tests: 70 unit, 31 integration and 4 smoke. Both models' changes went into that release, so we do not split the 105 between them.
• A calculators deploy on the morning of October 18 failed 2 of its 434 post-deploy smoke checks and passed them all after Codex's fix; the setup guide tells that story in full.

No rollback was performed in the period.

What the effort ledger can and cannot say

Our effort ledger records calendar time, tokens and cost, and it accepts only measured figures. At the time of this snapshot its latest update covered the period up to October 13, five days before Codex joined, and it was built from Claude's session records only. Since October 23 it has recorded Codex sessions too, by wall-clock time, with no token or cost counter for them, so a like-for-like cost comparison still has no measured source and we will not publish one. What the first day did cost in a measurable way, from the GitHub Actions API: at least 100 CI runs on October 19, 31 of them canceled by a newer push to the same branch; the month's Actions budget ran out at about 12:57 UTC and Codex's review quota a minute earlier; two guide-only pull requests alone took 14 CI runs and ten review requests.

Worked example: one review cycle, timed

The clearest single case is the pull request that added four guides about QnA MD, our tool for batching an agent's questions to a person. Claude opened it at 12:30 UTC on October 19 and asked Codex to check the guides against the tool's specification and its editor.

At 12:35 Codex posted two P2 findings:

1. "Do not describe veto as permanent." One guide said a veto closes an item for good. In the editor, selecting a vetoed item reopens it, and the reopen is logged.
2. "Describe ordered-chain rejection as turn-based." Another said anyone in an approval chain can reject. In an ordered chain, only the person whose turn it is can act.

Claude checked both against the editor's code and found Codex was right. It fixed both sentences in one commit 32 seconds after the review, replied on each thread with the code it had checked, and asked for a re-review at 12:36. Codex then raised a third finding: the guides carried a publication date in the future, which would show on the live site because the site deploys on merge. Claude fixed the date within a minute and added a test that fails any guide dated after the day the tests run. A fourth finding, about keeping that date in step with the actual merge day, is the one Claude put to the owner.

The result: three statements that were not true were corrected before any reader saw them. That is one pull request, not a trend, but it shows the kind of evidence we will keep collecting.

The follow-up measurements we planned

At the time of this snapshot, we identified these as the measures a later comparison would need. The first-day counts above do not establish any of those later results.

• Findings per pull request in each direction, and how many were fixed, disputed or escalated.
• Problems found after merge: failed post-deploy checks, rollbacks and fixes to merged work.
• Time from opening a pull request to merging it, for comparable kinds of change.
• Measured usage and cost for both models, once we have a measured source for Codex.