Asserting Control Over Code Review

Agents now produce code faster than our once human-centered processes can absorb. This is Goldratt’s Theory of Constraints in action. The overall process, delivering working code to production, is bound by the slowest gate.

As the volume of code agentically produced increased at WarmHub we were faced with the unenviable position of either redirecting more human attention to review or risking publishing agent-reviewed code directly.

We chose a third path: deploy an agentic review system that classifies risk, assembles the appropriate resources to establish certainty about the safety and correctness of change, and importantly, captures information at every step that allows the process to self-improve with every run.

Asserting control over your code review process lets you ship with confidence at the speed of agents.

We refer to our solution internally as The Council.

The end of an era

Our journey is a familiar one: when we started WarmHub, code was reviewed by the team, and every pull request (PR) was signed off by at least one other team member. Human review suffered from the widely shared challenges, including variable quality of review (time of day, too much pizza, competing demands), latency, and attention burden.

The introduction of agentically produced code amplified those challenges.

We ventured into agentic code review by evaluating several off-the-shelf solutions. Our initial efforts positioned agentic code review as an AND operation: everything was still reviewed by humans but augmented by agentic review.

Our experience with off-the-shelf review was widely variable: too much detail and pedantry in some cases and mile wide misses in others. Our confidence level in the quality of the code we shipped did not substantially increase, and team sentiment towards the AI reviews was squarely mixed. At no time did it decrease the human effort for code review or attempt to replace it entirely.

Our next approach was frontier model review. We adopted Codex as a GitHub Action and it ran as our primary reviewer for a few months. Initially this played the same role of augmenting human reviewers. We began to identify the next set of barriers to confident increased velocity.

Enter Risk Review

Our risk review model evolved from the obvious and simple observation that not every change requires the same level of scrutiny. A typo in a README.md has a quantitatively different risk profile from a 50 line change in core security code. The level of attention, human or otherwise, paid to the latter is attention better spent.

We introduced a deterministic risk reviewer that establishes a risk floor: the risk rating below which the PR cannot go. If the changes involve core domain logic or security code, the floor is set to high, and no subsequent analysis can lower it.

The calculated risk rating then became another axis of control for our code review inputs.

Initially we ran risk assessment at the beginning of the process, ran the code review, and then recalculated the exit risk rating based on both the initial rating (the floor) and the findings of the review. This effective risk rating then became a gate criterion with a recommended merge decision.

When the plan comes together

We continued to incrementally evolve our review process by adding deterministic packet preparation. The packet is the patch, PR metadata, prior review state, and policy context. A script prepares the review diff deterministically for the reviewer’s consumption. We also added structured output for the review findings. These findings were then applied to the PR review itself by adding inline comments and risk review outputs.

We got more delivery velocity without surrendering review discipline.

Structure emerged to encapsulate the agentic review core:

• Deterministic risk classification
• PR packet preparation
• Standardized findings structure

Opacity vs Control

Two things can be true at once. We rarely found argument with the single agent’s findings. On rare occasions some context was available to us that the agent lacked, so credible false positives could emerge. The review agent’s effectiveness was overshadowed by several frustrating traits:

• Latency: The time it took to do the reviews was not at all proportional to the size of the review
• Divergence: Each review turn finds another issue
• Alignment: We needed to steer from the edges about things we cared about, like specific performance tells, or workflow conventions

All three of these issues are bound together. Fixing an observed P1 (Priority 1, highest severity defect) in an initial review, followed by a new P1 in a follow-up run (each taking 4-7 minutes), is very frustrating. The second P1 existed at the time of the initial review; our read is that resolving the first freed the agent’s attention to reach the second. Expecting both in the initial turn is hopeful.

Lastly, as new strongly held beliefs about our system continued to emerge, the prompts used to drive the code review increased in complexity. While we believe in the value of models’ capability increasing exponentially, that trajectory wasn’t helping us in the here and now.

A desire for speed, cost, and overall effectiveness drove us to evaluate a system with greater control.

The remainder of the post covers what we evolved to.

The bets

Our plan was to capitalize on the structure we’d already put in place: deterministic review packets, calculated risk ratings, and structured outputs.

We folded into the plan several strongly held beliefs that we hold about working with models:

• Multi-agent, persona-based systems provide more diverse and steerable outcomes than single agent generalists
• Token spend is more valuable when the results of every run have the potential to convert to deterministic processes

Translated, we assembled a council of reviewers, each with a specific role, some hyper-specialized, each of which proposes defect hypotheses. The output of the council members is then collated, deduplicated, and then evaluated by a coordinator agent.

Deterministic inputs and orchestration surround the stochastic work. The council proposes findings; the coordinator decides; every run is recorded.

Recall & the digital twin

Before we assembled the council, we built a digital twin of our PR process that capitalized on the existing deterministically derived structure. This is simpler than it sounds: a new repo that can be manipulated to recreate the state of any code review at any point in its history. The twin environment allows us to run and rerun code reviews with a defined state.

We baselined the digital twin by replaying prior review states with the current harness. We validated that P1 & P2 (Priority 2, second-highest severity) issues continued to surface given the same input state. We refer to this as recall. Recall is only a test of the effectiveness of a new version of the PR review system against historical findings.

Enter the council

The council is a dynamically assembled set of agents, each of which is given a role defined as a prompt. The prompts for all council members contain general guidance on the nature of the review, where to access the review packet, and how to report results. Council members are given full access to the repository at the review SHA. Each council member then contains specific guidance on their area of focus.

Each council member run is executed with a specific model and thinking level.

Council member persona definitions vary from general (but application specific) to highly specialized.

We have a deterministic dispatcher (the orchestrator) that is responsible for assembling the right council based on diff content, calculated risk profile moderated by a set of heuristics. The orchestrator is responsible only for assembling, dispatching, and reporting council outputs. The last point is important. Every council member must report back even if they have no valid findings. A failed council dispatch is not conflated with silent acceptance.

The results of the council are dispatched to a coordinator agent, also with a defined role, to adjudicate the set of results and produce the final output packets.

The orchestrator stores each council run result and the final outputs in a durable ledger for post hoc analysis.

Each council run is fully instrumented with OpenTelemetry (OTEL) spans covering duration, token counts, and findings by severity and cost. We can easily visualize and query each run to find runtime and overall effectiveness issues.

State model

One specific abstraction that has proven to be valuable is modeling a code review as a state machine. We start with the initial state and then have states for author comments, fix verification, final pass, and importantly, no change. Each council run is parameterized by the review’s current state, and council assembly varies with it. An initial review containing SQL and security changes will have a bigger council (more specialists) than a simple web form change. State is a first class axis that factors into the council constitution.

State influences future trajectories.

What Control Bought Us

Our collected telemetry shows a median PR cost at ~$3 with a P95 less than $15. Duration is more meaningful per run: a median under 2 minutes, with a P95 under 5 minutes, because that’s how long a human or agent has to wait to get feedback.

Classifying risk allows more intelligent routing decisions.

We’ve recently switched from a Codex only harness where the only variability we had was model choice (within the GPT family) and thinking level to Pi, the most excellent and minimal coding harness from Mario Zechner. We’re now selectively swapping in alternate models including open weight models for selected council members.

The process for any change to the PR review system is the same. Identify a candidate test set of PRs that have the characteristics we’re testing for, a golden set of baseline PRs exhibiting a wide range of findings (machine-curated, 33 PRs in all: 25 for development, 8 held out, 69 findings), run the experiments, and keep or discard based on the outcome. Holdouts are not included in development but are used as a regression test dataset. We have a runbook defined for agents to design and then exercise the changed behavior in the digital twin environment. In many cases we hillclimb: iteratively adjust one control variable (say, thinking level) while holding recall fixed, until we hit a cost or latency target.

The digital twin continues to pay off. When GPT-5.6 dropped we looked at the cost/performance data relative to GPT-5.4. We started up an agent session and asked it to run the golden set against the new model. The agent summarized the results, including highlighting the change in per-run costs and time savings - recall checked out and we asked the agent to produce a PR and move it to production. We were live on it in our production reviewer within hours. No manual assessment was needed: the test, by design, showed no regression on recall against the historical golden set. Recall gates only against known regressions; that’s why field escapes still feed the ledger. We got both a 20% speed improvement and an 18% cost reduction per run with that change.

We do have a sampling mode for the digital twin where we can run random samples through the twin environment in parallel with the production code review process. This is useful for both vetting new approaches to council formation and model changes.

The twin allows replay of the PR process at any point.

Is the council better than the single agent? We no longer run them head to head. But by every trait that drove us to the council in the first place: yes. It’s significantly faster, findings are legitimate (we don’t measure false positives, but we should), reviews converge, and the shared sentiment internally is that when the council says “it’s good,” it is. Our CEO remarked “The review council is annoyingly good.”

"The review council is annoyingly good."

Recursive Self Improvement

One of the core tenets of our approach from the beginning was to capture data from each run, not just telemetry covering cost and duration but the issues identified during the code review. This lands in a durable ledger in WarmHub as a graph. Scheduled agents crawl the graph, labeling each found issue and then clustering those labels into reusable issue classes. Other agents process identified issue classes, create deterministic remedies, and integrate that back into our codebase. It’s the transformation of token spend from a cost into an investment. It’s how we turn stochastic processes into deterministic ones: tokens spent today, captured as evidence, become predictable, repeatable processes tomorrow. This is the subject of the next post.

It's the transformation of token spend from a cost into an investment.

Recursive self improvement with a deterministic bent.

Should You Build This?

Just because you can build this doesn’t mean you should. It’s an investment: engineering time, attention, evaluation data, and tokens. And it isn’t the right answer for every team.

For us the goal is clear: more delivery velocity without surrendering review discipline. Once you’ve watched one reviewer surface, by file and line, a bug that lived entirely outside the diff, it’s hard to accept that a single pass or reviewer perspective is enough.

The confidence comes from the system around the reviewers: routed attention, findings checked against the cited code, explicit coverage, and a record for every run.

If you’re not going to build all of this, three pieces travel well on their own:

• Route before you reason. Deterministic policy picks the review shape before any model spends a token. It’s the cheapest component here and it shapes all the spend downstream.
• Treat findings as hypotheses, not votes. In our experience one verifier that reads the cited code and confirms the mechanism is worth more than five specialists agreeing with each other.
• Report coverage. Report declared versus returned reviewers on every review. A partial pass that can’t pose as a clean one is worth more than a bigger council.

Even a good council will miss defects. Issues found in the field become the basis for more exploration: how did they escape? What process could have detected them beforehand?

The ledger lives in WarmHub, and it is queryable. In the next post we start querying it: turning a pile of review records into evidence-backed changes to the codebase and to the review process itself.

Code review isn’t a gate you pass through. It’s a loop you invest in.