S
Saqib Jan
Guest
Coding agents are graded on whether they can write code that passes tests, but nobody grades the review work that code creates once it lands in the queue.
The productivity case for AI-assisted coding has been made loudly ever since coding agents became capable of generating production-grade code at speed, and in the organizations that leaned into them hardest, the numbers genuinely hold up.
Developers who used to open two pull requests a week are now opening two to five a day, each carrying hundreds or even a thousand lines that an agent produced in the time an engineer would need to write a fraction of it.
There is a second number that gets far less attention, though, and the distance between it and the productivity figures is where most of the risk now sits. Even if AI-generated code carried no more bugs per line than human-written code, it arrives in volumes the review process was never built to absorb, across codebases that are growing faster than anyone can realistically read them.
So the bottleneck that coding agents removed from writing software simply moved to review, where it grows heavier with every agent a team adds.
Itamar Friedman, CEO and co-founder of Qodo, describes the review bottleneck as something engineering leaders are already living with, often before they have a name for it, and the arithmetic behind it is simple once it is laid out. A developer opening two pull requests a week at roughly a hundred lines each produced a volume one reviewer could read carefully without slowing the team down.
Coding agents changed both of those numbers at once, since a single pull request can now hold a thousand lines generated in five minutes and the same developer may open several in a day. That puts an order of magnitude more code in front of reviewers whose numbers and hours have not changed at all.
That volume also brings density with it. Because every line still carries some chance of a defect, ten times the code means ten times the defects reaching review, and AI-generated defects tend to look like clean code until someone with enough context reads them closely. A leader who lets generation scale without scaling review has already decided where that quality debt will land, even if nobody made the decision deliberately. The first practical move is to set a size limit on agent-generated pull requests, so that each review covers an amount of code a person can actually hold in their head.
SWE-bench has become the default way to compare coding agents, and Friedman is direct about what it actually measures. It is a code generation benchmark, where an agent starts from an issue or feature request, writes code, and scores well as long as that code implements the requirement and passes the associated tests, regardless of what it looks like inside.
“You can get a really good grade with quite a shitty code,” Friedman says. “Maybe the code is not maintainable. Maybe the code includes a security issue, but it will pass the test.” He frames the gap as yin and yang. The benchmark rewards the generation capability that turns a specification into working code, while it ignores the verification capability that decides whether working code is good enough for production. An agent can top the leaderboard and still add more to the review burden than it removes, and most teams adopting coding agents have no number that would tell them it is happening.
Measuring that other side calls for a different kind of test, and Friedman describes the one his team at Qodo built. It seeds AI-generated code with known issues that the review agents cannot see, then scores whether each agent finds all of them without raising false alarms that waste a reviewer’s time.
Teams evaluating their own tools can borrow the idea by planting known defects in a sample of real pull requests and checking how many their current review process catches. That gives them a verification number to sit beside the generation numbers they already track.
Mayank Bhola, Co-Founder and Head of Products at TestMu AI, points to a problem that sits in front of the technical one, because it decides whether leaders respond to the review bottleneck before something breaks or only after. The metrics most organizations rely on to track AI adoption all describe how much the generation side is producing, and none of them say anything about the health of the review process handling that output.
“Engineering leaders are looking at throughput metrics and seeing productivity gains, but they are not seeing the compounding quality debt that accumulates when review capacity does not scale with generation capacity,” Bhola explains, before describing what that looks like inside real teams.
“At TestMu AI, we see this pattern consistently across enterprise teams that have adopted coding agents aggressively. The PR volume increases dramatically. The review queue grows faster than the team can process it.” Because AI-generated code looks clean and passes basic tests, the issues that matter most get through simply because reviewers lack the bandwidth to look closely. Those include an architectural inconsistency or an edge case that only appears at scale.
That blind spot persists because throughput is easy to count and easy to credit to AI, while quality debt builds slowly and is easy to wave away until an incident or a failed audit forces the issue. Leaders who want to get ahead of it can put review metrics in the same report as velocity. A good start is how long pull requests wait for a first review alongside how much code each reviewer approves per day, so the cost becomes visible while it is still a trend and well before it becomes an outage.
Slowing generation down is no longer realistic, since teams that have seen real gains from coding agents are not going back to the output levels of two years ago, so whatever fixes review has to grow at the same pace. Friedman sees that as one of the central engineering challenges of this moment. “We want to prove that AI could actually do this really challenging work,” he says. “Doing code review, bringing it to high accuracy, is one of the biggest challenges in software development today.”
Getting there means building review agents on different terms from coding agents. A coding agent is rewarded for producing code that compiles and satisfies its requirements, while a review agent has to be rewarded for finding what the coding agent missed without flagging problems that are not there.
Those goals pull far enough apart that one model doing both jobs rarely does either of them well. I saw the same pattern when I wrote about AI coding tools a few months ago (https://hackernoon.com/ai-coding-tools-made-shipping-faster-but-testing-harder), where a delivery team at BairesDev stopped using AI to write tests for AI-generated code because the same blind spots showed up in both.
Bhola draws the product consequence directly. “The solution is not slower generation. It is smarter review, which means AI that is specifically designed to catch what human reviewers miss under pressure rather than AI that generates more of what already needs reviewing.” It echoes a point he made in my earlier piece on agent-to-agent testing (https://hackernoon.com/agent-to-agent-testing-is-now-the-baseline-for-enterprise-reliability), where a person approves the test plan before any test goes live. TestMu AI builds the same separation into Kane CLI, its testing agent. Kane CLI reports for every run what the tests actually proved and what coverage is still owed, so teams get a verification number in place of a simple count of passing tests.
Hiring more reviewers buys a little time against a structural shift before falling behind again, and the durable answer is review infrastructure that scales the way generation already does. The place to start is measurement, because a team that can see both sides of the bottleneck can tell where AI is genuinely saving work and where it is only moving that work onto someone else’s desk.
The productivity case for AI-assisted coding has been made loudly ever since coding agents became capable of generating production-grade code at speed, and in the organizations that leaned into them hardest, the numbers genuinely hold up.
Developers who used to open two pull requests a week are now opening two to five a day, each carrying hundreds or even a thousand lines that an agent produced in the time an engineer would need to write a fraction of it.
There is a second number that gets far less attention, though, and the distance between it and the productivity figures is where most of the risk now sits. Even if AI-generated code carried no more bugs per line than human-written code, it arrives in volumes the review process was never built to absorb, across codebases that are growing faster than anyone can realistically read them.
So the bottleneck that coding agents removed from writing software simply moved to review, where it grows heavier with every agent a team adds.
Review Capacity Stayed Flat While Code Volume Multiplied
Itamar Friedman, CEO and co-founder of Qodo, describes the review bottleneck as something engineering leaders are already living with, often before they have a name for it, and the arithmetic behind it is simple once it is laid out. A developer opening two pull requests a week at roughly a hundred lines each produced a volume one reviewer could read carefully without slowing the team down.
Coding agents changed both of those numbers at once, since a single pull request can now hold a thousand lines generated in five minutes and the same developer may open several in a day. That puts an order of magnitude more code in front of reviewers whose numbers and hours have not changed at all.
That volume also brings density with it. Because every line still carries some chance of a defect, ten times the code means ten times the defects reaching review, and AI-generated defects tend to look like clean code until someone with enough context reads them closely. A leader who lets generation scale without scaling review has already decided where that quality debt will land, even if nobody made the decision deliberately. The first practical move is to set a size limit on agent-generated pull requests, so that each review covers an amount of code a person can actually hold in their head.
SWE-bench Grades Only Half the Job
SWE-bench has become the default way to compare coding agents, and Friedman is direct about what it actually measures. It is a code generation benchmark, where an agent starts from an issue or feature request, writes code, and scores well as long as that code implements the requirement and passes the associated tests, regardless of what it looks like inside.
“You can get a really good grade with quite a shitty code,” Friedman says. “Maybe the code is not maintainable. Maybe the code includes a security issue, but it will pass the test.” He frames the gap as yin and yang. The benchmark rewards the generation capability that turns a specification into working code, while it ignores the verification capability that decides whether working code is good enough for production. An agent can top the leaderboard and still add more to the review burden than it removes, and most teams adopting coding agents have no number that would tell them it is happening.
Measuring that other side calls for a different kind of test, and Friedman describes the one his team at Qodo built. It seeds AI-generated code with known issues that the review agents cannot see, then scores whether each agent finds all of them without raising false alarms that waste a reviewer’s time.
Teams evaluating their own tools can borrow the idea by planting known defects in a sample of real pull requests and checking how many their current review process catches. That gives them a verification number to sit beside the generation numbers they already track.
Dashboards Count Output and Miss Review Debt
Mayank Bhola, Co-Founder and Head of Products at TestMu AI, points to a problem that sits in front of the technical one, because it decides whether leaders respond to the review bottleneck before something breaks or only after. The metrics most organizations rely on to track AI adoption all describe how much the generation side is producing, and none of them say anything about the health of the review process handling that output.
“Engineering leaders are looking at throughput metrics and seeing productivity gains, but they are not seeing the compounding quality debt that accumulates when review capacity does not scale with generation capacity,” Bhola explains, before describing what that looks like inside real teams.
“At TestMu AI, we see this pattern consistently across enterprise teams that have adopted coding agents aggressively. The PR volume increases dramatically. The review queue grows faster than the team can process it.” Because AI-generated code looks clean and passes basic tests, the issues that matter most get through simply because reviewers lack the bandwidth to look closely. Those include an architectural inconsistency or an edge case that only appears at scale.
That blind spot persists because throughput is easy to count and easy to credit to AI, while quality debt builds slowly and is easy to wave away until an incident or a failed audit forces the issue. Leaders who want to get ahead of it can put review metrics in the same report as velocity. A good start is how long pull requests wait for a first review alongside how much code each reviewer approves per day, so the cost becomes visible while it is still a trend and well before it becomes an outage.
Generation and Review Need Separate Systems
Slowing generation down is no longer realistic, since teams that have seen real gains from coding agents are not going back to the output levels of two years ago, so whatever fixes review has to grow at the same pace. Friedman sees that as one of the central engineering challenges of this moment. “We want to prove that AI could actually do this really challenging work,” he says. “Doing code review, bringing it to high accuracy, is one of the biggest challenges in software development today.”
Getting there means building review agents on different terms from coding agents. A coding agent is rewarded for producing code that compiles and satisfies its requirements, while a review agent has to be rewarded for finding what the coding agent missed without flagging problems that are not there.
Those goals pull far enough apart that one model doing both jobs rarely does either of them well. I saw the same pattern when I wrote about AI coding tools a few months ago (https://hackernoon.com/ai-coding-tools-made-shipping-faster-but-testing-harder), where a delivery team at BairesDev stopped using AI to write tests for AI-generated code because the same blind spots showed up in both.
Bhola draws the product consequence directly. “The solution is not slower generation. It is smarter review, which means AI that is specifically designed to catch what human reviewers miss under pressure rather than AI that generates more of what already needs reviewing.” It echoes a point he made in my earlier piece on agent-to-agent testing (https://hackernoon.com/agent-to-agent-testing-is-now-the-baseline-for-enterprise-reliability), where a person approves the test plan before any test goes live. TestMu AI builds the same separation into Kane CLI, its testing agent. Kane CLI reports for every run what the tests actually proved and what coverage is still owed, so teams get a verification number in place of a simple count of passing tests.
Hiring more reviewers buys a little time against a structural shift before falling behind again, and the durable answer is review infrastructure that scales the way generation already does. The place to start is measurement, because a team that can see both sides of the bottleneck can tell where AI is genuinely saving work and where it is only moving that work onto someone else’s desk.
Credits for Commentary
- Mayank Bhola, Co-Founder and Head of Products, TestMu AI - linkedin.com/in/mayankbhola/
- Itamar Friedman, Co-Founder and CEO, Qodo - linkedin.com/in/itamarf/