R
Your AI system passed every test. Accuracy looked good, response quality met the expected target, security checks passed, and the final demo went smoothly. Everyone agreed that the system was ready for production.
Then real users started using it.
A customer asked a question nobody had included in the test data. Another user asked a familiar question in an unfamiliar way. Someone provided incomplete information. A long conversation changed the context available to the model. Business data changed, user behavior shifted, and suddenly the AI system that looked reliable during testing started producing unexpected responses.
This does not necessarily mean the testing team failed. The bigger problem is that we are trying to apply traditional software testing ideas to systems that behave very differently.
Traditional software testing usually asks a straightforward question: Does the system behave as expected?
With AI, the question becomes more complicated: How reliably does the system behave when the inputs, context, data, and even the outputs can change?
That difference is important because passing a test suite does not automatically mean an AI system is ready for everything it will encounter in production.
Consider a traditional payment application. If a customer enters a valid account, selects a payment amount, and submits the transaction, we can usually define exactly what should happen. The payment should be processed according to established business rules, a confirmation should be generated, and the transaction should be recorded.
If we repeat the same test under the same conditions, we normally expect the same result. If the actual result does not match the expected result, the test fails.
Now consider an AI assistant. A customer asks, “Can you explain why my payment was declined?”
What exactly is the expected response?
There may be several acceptable answers. One response might explain common reasons for a declined payment. Another might ask the customer for additional information. A third might explain the relevant payment rules and recommend the next step.
All of these responses could potentially be correct.
This means AI testing cannot always depend on a simple comparison between an expected result and an actual result. We also need to determine whether the response is factually correct, relevant, grounded in available information, compliant with business policies, safe, complete, and appropriate for the user's request.
The definition of a passing test therefore becomes much more complicated.
Most AI systems are evaluated using controlled datasets before they are released.
Imagine that a customer-support AI is tested with 1,000 prompts and produces acceptable responses for 970 of them. The team reports a 97% success rate.
That number looks impressive, but it only tells us how the system performed against those particular 1,000 prompts under those particular conditions.
Production is different.
Real users do not communicate like carefully prepared test datasets. They misspell words, provide incomplete information, combine several questions into one request, use slang, switch languages, paste large amounts of text, and change subjects during conversations.
They also ask questions that developers and testers may never have anticipated.
Some users may even intentionally try to manipulate the AI into ignoring its instructions or producing information it should not provide.
A pre-production test suite can cover many scenarios, but it cannot perfectly reproduce every interaction that will occur after deployment.
That is one of the main reasons an AI system can pass its tests and still fail in production.
Another major difference between traditional applications and generative AI is nondeterminism.
With conventional software, repeating the same test under the same conditions normally produces the same result. With generative AI, the same question can sometimes produce different responses.
Imagine asking an AI system to summarize a customer's refund policy five times. The first four responses may be correct, while the fifth contains a subtle but important error.
If the testing team executed that scenario only once, the test might have passed.
But did the system really demonstrate reliable behavior?
This is why executing an AI test once may not provide enough information. Important scenarios may need to be evaluated repeatedly to understand how consistently the system produces acceptable responses.
Instead of asking only, “Did this test pass?”, AI quality engineers should also ask, “How consistently does this scenario produce an acceptable result?”
For AI systems, consistency can be just as important as correctness.
An AI application is rarely just a model connected directly to a user interface. Modern enterprise AI systems may depend on system instructions, retrieval services, knowledge bases, APIs, enterprise databases, business rules, security controls, and other components.
A failure in any part of that chain can affect the final response.
The AI model itself might be functioning correctly, but the retrieval system could provide an outdated document. A knowledge base might contain conflicting information. A prompt template might have changed during deployment. An API could return incomplete data. A document-processing system might incorrectly extract an important section of a file.
From the customer's perspective, these technical distinctions do not matter. The customer asked a question and received the wrong answer.
That is why AI quality cannot be treated only as model quality.
We need to validate the entire AI-enabled application and the systems surrounding the model.
Suppose an AI system processes 100,000 interactions and produces acceptable responses 98% of the time.
A 98% success rate sounds excellent. However, the remaining 2% represents 2,000 potentially problematic interactions.
More importantly, we need to understand what happened during those failures.
Imagine two AI systems with the same 98% success rate. In the first system, most failures involve minor formatting or wording problems. In the second system, the failures involve financial transactions, security decisions, or incorrect customer instructions.
The accuracy numbers are identical, but the business risks are completely different.
This is why AI quality should not be evaluated only by asking how frequently a system fails. Teams also need to understand where the system fails and what the consequences of those failures could be.
A useful way to think about this is that risk depends on both the probability of failure and the impact of failure.
A rare but critical failure can be far more important than hundreds of minor quality issues.
Another challenge is drift.
The AI system tested today may not operate under exactly the same conditions several months from now.
Customers change their behavior. Products change. Business policies change. Terminology changes. Enterprise knowledge bases are updated, and the types of questions users ask can shift over time.
The AI application itself may also change. A model version could be upgraded, a system prompt could be modified, retrieval settings could be adjusted, new documents could be added, or an external service could change.
Each individual modification might seem small, but together these changes can affect system behavior.
This means an AI system cannot simply receive a quality stamp on release day and be considered permanently tested. Validation has to continue as the environment evolves.
Some software failures are easy to recognize. An application crashes, an API returns an error, a page does not load, or a button stops working.
AI failures can be much harder to detect.
Imagine an AI assistant telling a customer, “According to your company's policy, customers have 90 days to request a refund.”
The sentence is clear. The grammar is correct. The answer sounds professional and confident.
But what if the actual policy allows only 30 days?
The system may not generate an error message. Nothing crashes. There may be no warning that anything went wrong.
Traditional monitoring could show that the service is available, response time is normal, and the API returned a successful status.
Technically, the system appears healthy.
Semantically, it failed.
This is one of the most important challenges in AI quality engineering. Monitoring whether an application is running is no longer enough. Teams also need ways to understand whether the AI's behavior remains reliable.
AI teams should also avoid treating every failure as if it has the same impact.
A slightly awkward sentence is not equivalent to incorrect financial information. An incomplete answer is not equivalent to an unsafe recommendation.
Failures can be categorized according to severity. Minor wording problems may simply need monitoring. Incomplete or irrelevant responses may require review. Incorrect business information may require investigation, while unsafe or prohibited behavior may need immediate blocking or escalation.
This changes the quality discussion from “What is our AI accuracy?” to a much more useful question: “Where does our AI fail, how frequently does it happen, and what is the impact when it does?”
Traditional testing relies heavily on two outcomes: pass or fail.
For AI systems, those two categories may not always tell the complete story.
One AI response might be completely correct, relevant, and grounded in trusted information. Another might be mostly correct but omit an important detail. A third might sound excellent while containing incorrect information.
Simply marking these responses as pass or fail can hide valuable information about their quality.
AI evaluation may need to consider multiple dimensions such as correctness, groundedness, relevance, consistency, completeness, policy compliance, and safety.
The exact measurements will depend on the application. The important point is that AI quality is often multidimensional.
A single pass/fail result may not be enough to describe whether an AI response is truly reliable.
AI testing should not focus only on clean and predictable requests.
Consider a user who simply says, “Cancel it.”
What should the AI cancel? A payment? An order? A subscription? An appointment?
A reliable system should recognize that it does not have enough information instead of confidently guessing.
Now imagine a user asking, “Why was it rejected?” Again, the AI may not have enough context to provide a reliable answer.
Teams should also test contradictory instructions, extremely long conversations, unexpected language changes, conflicting documents, missing information, unusual input combinations, and attempts to manipulate the system's instructions.
These scenarios may reveal more about production readiness than hundreds of predictable happy-path tests.
With traditional software, we often think of testing as an activity that primarily happens before release.
AI requires a different approach.
Production introduces real behaviors and input patterns that a test environment cannot completely reproduce. This means production monitoring should become part of the overall validation strategy.
When an important AI failure occurs in production, the team should investigate why it happened. When appropriate, that scenario should then become part of the evaluation or regression suite.
If a customer discovers an unusual input that causes an incorrect response, similar scenarios should be tested in future releases. If a retrieval problem causes the AI to use outdated information, the team should add validation around that failure mode.
In this way, AI testing becomes a continuous cycle of evaluation, deployment, observation, learning, and revalidation.
The test suite becomes stronger because it evolves based on actual production behavior rather than only assumptions made before release.
There is no single metric that can completely represent AI quality.
Depending on the application, teams may need to consider response correctness, groundedness, consistency, unsupported claims, policy compliance, retrieval quality, refusal behavior, latency, human escalation, production drift, and user complaints.
But teams should not collect metrics simply because they are available.
The important question is whether those measurements tell us something meaningful about the business risk of the AI system.
A customer-support assistant, coding assistant, fraud-detection system, and healthcare application should not necessarily use the same definition of quality.
Quality metrics should reflect what the AI is expected to do and what could happen when it gets something wrong.
Before releasing an AI feature, teams should ask more than whether the test suite passed.
Have we tested realistic and diverse user inputs? Have important scenarios been executed multiple times? Do we understand which failures could have serious business consequences? Are factual responses grounded in trusted information when necessary? Have we validated the entire application rather than only the model?
Teams should also know what happens when the AI is uncertain. Does it guess? Does it ask the user for clarification? Does it retrieve more information? Does it refuse to answer? Does it escalate the request to a human?
There should also be a plan for detecting quality degradation after deployment. If the AI becomes less reliable next month, the team needs a way to recognize the change.
Finally, high-impact actions should have appropriate safeguards. In some situations, deterministic validation, additional business rules, or human approval may be necessary before an AI-generated recommendation becomes a real-world action.
The biggest mistake we can make with AI testing is believing that the goal is to prove that an AI system will always work.
For generative AI, it is unrealistic to demonstrate that every possible future input will always produce the correct response.
The input space is enormous. User behavior changes. Business data changes. Context changes. Models and surrounding systems change. And generative AI can produce different responses even when the underlying request is similar.
What quality engineering can do is build evidence about reliability.
We can identify where the system performs well. We can discover where it struggles. We can measure how consistently it behaves. We can classify failures according to their impact. We can add safeguards around high-risk scenarios, monitor behavior after deployment, and continuously turn production failures into better future tests.
That changes what “passed testing” should mean for AI.
It should not mean:
“This AI will always work correctly.”
A more realistic statement is:
“This system demonstrated acceptable behavior across the conditions we evaluated, and we have mechanisms to detect and respond when production introduces conditions we did not anticipate.”
Because production always introduces something we did not anticipate.
Passing the test suite is still important.
But for an AI system, passing every test is not the end of testing.
It is where the next phase of validation begins.
Then real users started using it.
A customer asked a question nobody had included in the test data. Another user asked a familiar question in an unfamiliar way. Someone provided incomplete information. A long conversation changed the context available to the model. Business data changed, user behavior shifted, and suddenly the AI system that looked reliable during testing started producing unexpected responses.
This does not necessarily mean the testing team failed. The bigger problem is that we are trying to apply traditional software testing ideas to systems that behave very differently.
Traditional software testing usually asks a straightforward question: Does the system behave as expected?
With AI, the question becomes more complicated: How reliably does the system behave when the inputs, context, data, and even the outputs can change?
That difference is important because passing a test suite does not automatically mean an AI system is ready for everything it will encounter in production.
A Passing Test Is Not a Guarantee
Consider a traditional payment application. If a customer enters a valid account, selects a payment amount, and submits the transaction, we can usually define exactly what should happen. The payment should be processed according to established business rules, a confirmation should be generated, and the transaction should be recorded.
If we repeat the same test under the same conditions, we normally expect the same result. If the actual result does not match the expected result, the test fails.
Now consider an AI assistant. A customer asks, “Can you explain why my payment was declined?”
What exactly is the expected response?
There may be several acceptable answers. One response might explain common reasons for a declined payment. Another might ask the customer for additional information. A third might explain the relevant payment rules and recommend the next step.
All of these responses could potentially be correct.
This means AI testing cannot always depend on a simple comparison between an expected result and an actual result. We also need to determine whether the response is factually correct, relevant, grounded in available information, compliant with business policies, safe, complete, and appropriate for the user's request.
The definition of a passing test therefore becomes much more complicated.
The Test Environment Is Not Production
Most AI systems are evaluated using controlled datasets before they are released.
Imagine that a customer-support AI is tested with 1,000 prompts and produces acceptable responses for 970 of them. The team reports a 97% success rate.
That number looks impressive, but it only tells us how the system performed against those particular 1,000 prompts under those particular conditions.
Production is different.
Real users do not communicate like carefully prepared test datasets. They misspell words, provide incomplete information, combine several questions into one request, use slang, switch languages, paste large amounts of text, and change subjects during conversations.
They also ask questions that developers and testers may never have anticipated.
Some users may even intentionally try to manipulate the AI into ignoring its instructions or producing information it should not provide.
A pre-production test suite can cover many scenarios, but it cannot perfectly reproduce every interaction that will occur after deployment.
That is one of the main reasons an AI system can pass its tests and still fail in production.
The Same Question Can Produce Different Answers
Another major difference between traditional applications and generative AI is nondeterminism.
With conventional software, repeating the same test under the same conditions normally produces the same result. With generative AI, the same question can sometimes produce different responses.
Imagine asking an AI system to summarize a customer's refund policy five times. The first four responses may be correct, while the fifth contains a subtle but important error.
If the testing team executed that scenario only once, the test might have passed.
But did the system really demonstrate reliable behavior?
This is why executing an AI test once may not provide enough information. Important scenarios may need to be evaluated repeatedly to understand how consistently the system produces acceptable responses.
Instead of asking only, “Did this test pass?”, AI quality engineers should also ask, “How consistently does this scenario produce an acceptable result?”
For AI systems, consistency can be just as important as correctness.
Production Context Changes Everything
An AI application is rarely just a model connected directly to a user interface. Modern enterprise AI systems may depend on system instructions, retrieval services, knowledge bases, APIs, enterprise databases, business rules, security controls, and other components.
A failure in any part of that chain can affect the final response.
The AI model itself might be functioning correctly, but the retrieval system could provide an outdated document. A knowledge base might contain conflicting information. A prompt template might have changed during deployment. An API could return incomplete data. A document-processing system might incorrectly extract an important section of a file.
From the customer's perspective, these technical distinctions do not matter. The customer asked a question and received the wrong answer.
That is why AI quality cannot be treated only as model quality.
We need to validate the entire AI-enabled application and the systems surrounding the model.
High Accuracy Can Still Hide Serious Risk
Suppose an AI system processes 100,000 interactions and produces acceptable responses 98% of the time.
A 98% success rate sounds excellent. However, the remaining 2% represents 2,000 potentially problematic interactions.
More importantly, we need to understand what happened during those failures.
Imagine two AI systems with the same 98% success rate. In the first system, most failures involve minor formatting or wording problems. In the second system, the failures involve financial transactions, security decisions, or incorrect customer instructions.
The accuracy numbers are identical, but the business risks are completely different.
This is why AI quality should not be evaluated only by asking how frequently a system fails. Teams also need to understand where the system fails and what the consequences of those failures could be.
A useful way to think about this is that risk depends on both the probability of failure and the impact of failure.
A rare but critical failure can be far more important than hundreds of minor quality issues.
The System You Tested May Change Over Time
Another challenge is drift.
The AI system tested today may not operate under exactly the same conditions several months from now.
Customers change their behavior. Products change. Business policies change. Terminology changes. Enterprise knowledge bases are updated, and the types of questions users ask can shift over time.
The AI application itself may also change. A model version could be upgraded, a system prompt could be modified, retrieval settings could be adjusted, new documents could be added, or an external service could change.
Each individual modification might seem small, but together these changes can affect system behavior.
This means an AI system cannot simply receive a quality stamp on release day and be considered permanently tested. Validation has to continue as the environment evolves.
The Most Dangerous Failure May Look Completely Normal
Some software failures are easy to recognize. An application crashes, an API returns an error, a page does not load, or a button stops working.
AI failures can be much harder to detect.
Imagine an AI assistant telling a customer, “According to your company's policy, customers have 90 days to request a refund.”
The sentence is clear. The grammar is correct. The answer sounds professional and confident.
But what if the actual policy allows only 30 days?
The system may not generate an error message. Nothing crashes. There may be no warning that anything went wrong.
Traditional monitoring could show that the service is available, response time is normal, and the API returned a successful status.
Technically, the system appears healthy.
Semantically, it failed.
This is one of the most important challenges in AI quality engineering. Monitoring whether an application is running is no longer enough. Teams also need ways to understand whether the AI's behavior remains reliable.
Not Every AI Failure Is Equally Important
AI teams should also avoid treating every failure as if it has the same impact.
A slightly awkward sentence is not equivalent to incorrect financial information. An incomplete answer is not equivalent to an unsafe recommendation.
Failures can be categorized according to severity. Minor wording problems may simply need monitoring. Incomplete or irrelevant responses may require review. Incorrect business information may require investigation, while unsafe or prohibited behavior may need immediate blocking or escalation.
This changes the quality discussion from “What is our AI accuracy?” to a much more useful question: “Where does our AI fail, how frequently does it happen, and what is the impact when it does?”
AI Quality Needs More Than Pass or Fail
Traditional testing relies heavily on two outcomes: pass or fail.
For AI systems, those two categories may not always tell the complete story.
One AI response might be completely correct, relevant, and grounded in trusted information. Another might be mostly correct but omit an important detail. A third might sound excellent while containing incorrect information.
Simply marking these responses as pass or fail can hide valuable information about their quality.
AI evaluation may need to consider multiple dimensions such as correctness, groundedness, relevance, consistency, completeness, policy compliance, and safety.
The exact measurements will depend on the application. The important point is that AI quality is often multidimensional.
A single pass/fail result may not be enough to describe whether an AI response is truly reliable.
Test the Uncomfortable Scenarios
AI testing should not focus only on clean and predictable requests.
Consider a user who simply says, “Cancel it.”
What should the AI cancel? A payment? An order? A subscription? An appointment?
A reliable system should recognize that it does not have enough information instead of confidently guessing.
Now imagine a user asking, “Why was it rejected?” Again, the AI may not have enough context to provide a reliable answer.
Teams should also test contradictory instructions, extremely long conversations, unexpected language changes, conflicting documents, missing information, unusual input combinations, and attempts to manipulate the system's instructions.
These scenarios may reveal more about production readiness than hundreds of predictable happy-path tests.
Production Monitoring Should Become Part of Testing
With traditional software, we often think of testing as an activity that primarily happens before release.
AI requires a different approach.
Production introduces real behaviors and input patterns that a test environment cannot completely reproduce. This means production monitoring should become part of the overall validation strategy.
When an important AI failure occurs in production, the team should investigate why it happened. When appropriate, that scenario should then become part of the evaluation or regression suite.
If a customer discovers an unusual input that causes an incorrect response, similar scenarios should be tested in future releases. If a retrieval problem causes the AI to use outdated information, the team should add validation around that failure mode.
In this way, AI testing becomes a continuous cycle of evaluation, deployment, observation, learning, and revalidation.
The test suite becomes stronger because it evolves based on actual production behavior rather than only assumptions made before release.
What Should AI Teams Measure?
There is no single metric that can completely represent AI quality.
Depending on the application, teams may need to consider response correctness, groundedness, consistency, unsupported claims, policy compliance, retrieval quality, refusal behavior, latency, human escalation, production drift, and user complaints.
But teams should not collect metrics simply because they are available.
The important question is whether those measurements tell us something meaningful about the business risk of the AI system.
A customer-support assistant, coding assistant, fraud-detection system, and healthcare application should not necessarily use the same definition of quality.
Quality metrics should reflect what the AI is expected to do and what could happen when it gets something wrong.
Before Production, Ask Better Questions
Before releasing an AI feature, teams should ask more than whether the test suite passed.
Have we tested realistic and diverse user inputs? Have important scenarios been executed multiple times? Do we understand which failures could have serious business consequences? Are factual responses grounded in trusted information when necessary? Have we validated the entire application rather than only the model?
Teams should also know what happens when the AI is uncertain. Does it guess? Does it ask the user for clarification? Does it retrieve more information? Does it refuse to answer? Does it escalate the request to a human?
There should also be a plan for detecting quality degradation after deployment. If the AI becomes less reliable next month, the team needs a way to recognize the change.
Finally, high-impact actions should have appropriate safeguards. In some situations, deterministic validation, additional business rules, or human approval may be necessary before an AI-generated recommendation becomes a real-world action.
The Definition of “Done” Has Changed
The biggest mistake we can make with AI testing is believing that the goal is to prove that an AI system will always work.
For generative AI, it is unrealistic to demonstrate that every possible future input will always produce the correct response.
The input space is enormous. User behavior changes. Business data changes. Context changes. Models and surrounding systems change. And generative AI can produce different responses even when the underlying request is similar.
What quality engineering can do is build evidence about reliability.
We can identify where the system performs well. We can discover where it struggles. We can measure how consistently it behaves. We can classify failures according to their impact. We can add safeguards around high-risk scenarios, monitor behavior after deployment, and continuously turn production failures into better future tests.
That changes what “passed testing” should mean for AI.
It should not mean:
“This AI will always work correctly.”
A more realistic statement is:
“This system demonstrated acceptable behavior across the conditions we evaluated, and we have mechanisms to detect and respond when production introduces conditions we did not anticipate.”
Because production always introduces something we did not anticipate.
Passing the test suite is still important.
But for an AI system, passing every test is not the end of testing.
It is where the next phase of validation begins.