G
George Ukkuru
Guest
Eighty-eight percent of respondents say AI is a priority for their organization’s future testing strategy. Only 12.6% report using it across key testing activities today. Those findings, from a Leapwork and SD Times survey of 302 QA professionals, do not mean that AI in testing is failing. They reveal a considerable gap between enthusiasm for AI and its broad adoption in testing.
Here is where many QA teams go wrong. A team runs a small pilot, then judges it against the standards of a full organization-wide rollout. When the numbers do not look appealing, they conclude that the tool is not capable, even though the pilot was never built to answer that question. In some cases, the reverse happens. A clean pilot gets treated as proof the tool is ready for everything, and it gets deployed at scale into use cases it was never tested on.
Capgemini’s World Quality Report 2025 shows uneven GenAI adoption levels across the industry. Fifteen percent of organizations have reached organization-wide AI use. 43% are still experimental and 30% of companies are only running limited use cases. 11% are non-adopters.
One size does not fit all, as these groups are trying to achieve different goals. An AI testing tool pilot succeeds if the tool works reliably for the use case that was in scope. A scaling program succeeds if teams can continue using it without the maintenance effort wiping out the productivity gains. A successful organization-wide rollout of an AI tool should deliver high-quality outcomes and meaningful business value across the organization. Judge any of these against the wrong bar, and you’ll get a number that looks bad for reasons that have nothing to do with the tool itself.
A health-tech client I advised ran a six-week pilot on one well-documented intake flow. It went well. The generated tests required almost no rework as the coverage was good. The team also identified three Severity 2 defects that the existing manual approach had not identified.
Leadership was impressed with the results, skipped the scaling stage and approved a rollout across 12 flows within the next two weeks. Three weeks in, the maintenance effort on the complex and less-documented flows increased above 40%. QA engineers were spending more time fixing generated tests than writing new ones. Confidence in the new tool dropped because a pilot on one clean flow never answered the question leadership acted on.
The pilot was not wrong. It just answered a much smaller question than the one they used it to answer.
Pilot: One team, one flow, usually your most documented one. The goal is to prove that the tool can generate tests worth trusting with little or no rework. Track the acceptance rate, meaning how many generated tests go into the suite with minimal or no rework, and compare its defect detection effectiveness with the current manual approach. A pilot that only proves the tool can run tells you little about its testing effectiveness.
Scaling: Several teams, several flows, usually ones with moderate documentation and manageable risk. The question shifts from “does it work” to “can teams trust it at scale?” Keep tracking the override rate week over week and it should show a declining trend. If it is flat or increasing over the past few weeks, the tool has failed to earn the team’s trust yet. Focus on measuring coverage of the critical workflows that would affect the business or end users if they were to fail in production, instead of looking at the number of test cases produced.
Scaled: AI-generated tests run across the org with minimal per-test review, sitting inside CI, touching flows nobody watches closely anymore. Success needs to be measured in terms of the defect escape rate. The metric should remain flat or show improvement across the whole portfolio, without limiting the measurement to flows covered during the pilot. Track the flaky test rate on AI-generated tests separately from your human-written tests. A lower flaky test rate will tell you whether the tests are becoming more stable as the implementation scales.
Each stage has its own finish line. Moving to the next one means clearing the current one, not just running out of patience with it.
Stop asking whether AI in testing works. Ask these three questions before you move to the next stage of implementation.
A passing pilot is just a starting point. Do not treat it as a production readiness certificate.
Here is where many QA teams go wrong. A team runs a small pilot, then judges it against the standards of a full organization-wide rollout. When the numbers do not look appealing, they conclude that the tool is not capable, even though the pilot was never built to answer that question. In some cases, the reverse happens. A clean pilot gets treated as proof the tool is ready for everything, and it gets deployed at scale into use cases it was never tested on.
Different Stages, Different Finish Lines
Capgemini’s World Quality Report 2025 shows uneven GenAI adoption levels across the industry. Fifteen percent of organizations have reached organization-wide AI use. 43% are still experimental and 30% of companies are only running limited use cases. 11% are non-adopters.
One size does not fit all, as these groups are trying to achieve different goals. An AI testing tool pilot succeeds if the tool works reliably for the use case that was in scope. A scaling program succeeds if teams can continue using it without the maintenance effort wiping out the productivity gains. A successful organization-wide rollout of an AI tool should deliver high-quality outcomes and meaningful business value across the organization. Judge any of these against the wrong bar, and you’ll get a number that looks bad for reasons that have nothing to do with the tool itself.
A Client’s Costly Leap
A health-tech client I advised ran a six-week pilot on one well-documented intake flow. It went well. The generated tests required almost no rework as the coverage was good. The team also identified three Severity 2 defects that the existing manual approach had not identified.
Leadership was impressed with the results, skipped the scaling stage and approved a rollout across 12 flows within the next two weeks. Three weeks in, the maintenance effort on the complex and less-documented flows increased above 40%. QA engineers were spending more time fixing generated tests than writing new ones. Confidence in the new tool dropped because a pilot on one clean flow never answered the question leadership acted on.
The pilot was not wrong. It just answered a much smaller question than the one they used it to answer.
The Three-Stage Maturity Model
Pilot: One team, one flow, usually your most documented one. The goal is to prove that the tool can generate tests worth trusting with little or no rework. Track the acceptance rate, meaning how many generated tests go into the suite with minimal or no rework, and compare its defect detection effectiveness with the current manual approach. A pilot that only proves the tool can run tells you little about its testing effectiveness.
Scaling: Several teams, several flows, usually ones with moderate documentation and manageable risk. The question shifts from “does it work” to “can teams trust it at scale?” Keep tracking the override rate week over week and it should show a declining trend. If it is flat or increasing over the past few weeks, the tool has failed to earn the team’s trust yet. Focus on measuring coverage of the critical workflows that would affect the business or end users if they were to fail in production, instead of looking at the number of test cases produced.
Scaled: AI-generated tests run across the org with minimal per-test review, sitting inside CI, touching flows nobody watches closely anymore. Success needs to be measured in terms of the defect escape rate. The metric should remain flat or show improvement across the whole portfolio, without limiting the measurement to flows covered during the pilot. Track the flaky test rate on AI-generated tests separately from your human-written tests. A lower flaky test rate will tell you whether the tests are becoming more stable as the implementation scales.
Each stage has its own finish line. Moving to the next one means clearing the current one, not just running out of patience with it.
The Question That Actually Matters
Stop asking whether AI in testing works. Ask these three questions before you move to the next stage of implementation.
- What exactly did the pilot prove?
- Which scenarios and use cases were left out during the pilot?
- What additional data and insights must be gathered to make a scaling decision?
A passing pilot is just a starting point. Do not treat it as a production readiness certificate.