S
Spurthi Tallam
Guest
On September 3, 2026, OpenAI began rolling out GPT-6 Astra. Within days, some developers were reporting what one of them called a "post-launch lobotomy." An open issue in OpenAI's Codex repository, filed by one developer, described turns ending after about 30 seconds and completion reports for work that had never been done.
Others said they had gone back to the previous model, GPT-5.6 Sol. These were reports, not a diagnosis. As of mid-October, OpenAI had not publicly explained those reports, and nobody outside the company could say what, if anything, had changed.
Two months earlier, users had made the same complaint about Sol, and that time there was an answer of sorts. A widely circulated post attributed to OpenAI's Thibault Sottiaux denied Sol had been weakened but said the team had run and reverted experiments that changed its reasoning effort: roughly how much computation the model spends before it answers.
Stories like these get filed under "the model got dumber," which is true from the user's chair and not very useful. A router picks which model runs, an inference stack executes it, a system prompt constrains it, an SDK fills in the parameters you left out, and your application decides what the model gets to see. Any of those can move while the weights sit still.
When an office feels cold, someone checks the thermostat before blaming the furnace. A deployed model has many thermostats, and most are set by defaults nobody inspects.
The best-documented case came from Anthropic. On March 4, 2026, it moved Claude Code's default reasoning effort from high to medium to stop pauses that made the interface look frozen. A March 26 caching bug then made sessions that sat idle for more than an hour keep discarding their earlier reasoning on every later turn, and an April 16 system-prompt line capped text between tool calls at 25 words. Users noticed. AMD AI senior director Stella Laurenzo analyzed 6,852 sessions and found shallower reasoning and more premature stopping. Anthropic's April 23 postmortem said the API and inference layer were unaffected. It called the effort change "the wrong tradeoff."
Routers and names fail the same way. When GPT-5's model router broke on launch day in August 2025, Sam Altman said that "the result was GPT-5 seemed way dumber." Earlier that year, Google repointed a dated Gemini endpoint to a newer model, which developers on its forum called a breach of trust. OpenAI's documentation draws the line cleanly: a snapshot locks a model version, while an alias can be repointed. Pinning a dated name protects you only if the provider keeps that name attached to the same model.
Even when the model name stays the same, its defaults can change. When Claude Code's default moved to medium, most users stayed there, even after Anthropic added notices and a selector. A default isn't a suggestion. For most users, it is the product. Defaults move for reasons that look sensible on a dashboard: lower latency, lower token spend, a faster-feeling agent. The cost shows up as a vague sense that the product got less intelligent, and users rarely have a way to connect that feeling to a setting.
Below the product settings is the machinery, and it can fail without anyone deciding anything. Anthropic's September 2025 postmortem described three overlapping infrastructure bugs. One bug misrouted Claude Sonnet 4 requests to servers configured for a longer context window, starting at about 0.8% of traffic and reaching 16% at the worst hour after a routine load-balancing change. Another, a compiler bug, could drop the most likely next token from consideration. Anthropic's evaluations missed the degradation, partly because models often recover from a single bad token. The company also said it doesn't reduce quality in response to load. Customers can't audit that claim, but the incident didn't require a conspiracy. Bugs were enough.
Open-weight models make the problem easier to see, because many companies serve the same weights. vLLM's official results for OpenAI's gpt-oss-120b on the AIME 2025 math benchmark show 93.0 at high reasoning effort and 79.6 at medium. That gap matters when a provider mishandles the setting. When Artificial Analysis benchmarked the providers hosting the model at high effort, six scored a median of 93.3%, and Azure scored 80%, a gap of 13.3 points.
In Simon Willison's write-up, Microsoft's Lucas Pickup explained that older vLLM commits "didn't respect reasoning_effort, so all requests defaulted to medium." In that case, the gap came from a serving configuration that ignored the requested setting, not from the weights.
None of this is new, only easier to notice. In 2023, Lingjiao Chen, Matei Zaharia, and James Zou compared the March and June versions of GPT-4. In the revised paper, its accuracy on their prime-number task fell from 84.0% to 51.1%. Arvind Narayanan and Sayash Kapoor pointed out that the test contained only primes, so a shift in which answer the model leaned toward could look like a change in math ability. That's a fair correction, though a product built on those prompts would still have behaved differently in June.
So far the story has been "they changed it," which is comfortable because it puts the fault somewhere we can't reach. The less comfortable version is that the same failures live one layer up, in the code we own.
Start with sampling. For years, the standard advice was to use a low temperature for reliable output. Google's Gemini 3 guide now reverses that. It recommends leaving temperature at its default of 1.0 and warns that lower values may cause looping or degraded performance on complex reasoning. A temperature setting that suited one model family can hurt the next, with no change in your code.
Output budgets changed meaning when reasoning models arrived. For newer models, OpenAI deprecated max_tokens in favor of max_completion_tokens, which counts hidden reasoning tokens as well as the visible answer. A tight limit can be spent largely or entirely on thinking. The result can be an empty or cut-off answer from a model that was still working on the problem.
Context limits can change the input without raising an error. Since Ollama 0.15.5, its default context length depends on how much video memory the machine has, from 4,096 tokens on smaller GPUs to 262,144 on the largest. When a conversation overflows, Ollama prunes older messages from the front while trying to keep the system prompt and the latest message. Move the same application to a smaller cloud instance, and the model reads a different conversation. To the user, the model forgot an instruction. To the system, the instruction was never in the prompt.
Retrieval has quiet defaults too. Halve the chunk size without retrieving more chunks, and the model may get much less of the relevant evidence. Change the embedding model without rebuilding the index, and retrieval can break outright. It may also keep working while quietly becoming less accurate. Either way, the model is answering with different evidence from the evidence you thought you gave it.
And the evaluation set usually doesn't notice. Anthropic's April system-prompt change passed several weeks of internal testing. Only broader line-by-line ablations later found a 3% drop on one evaluation. That's a reason not to treat a small smoke test as evidence that a configuration change is harmless.
What Anthropic did afterwards is the useful part. After September 2025, it committed to running quality evaluations continuously against production systems. After April, it added line-by-line ablations for system-prompt changes, review tooling, and gradual rollouts for anything that might trade away intelligence. None of it required retraining the model.
The same discipline works outside the labs, and most of it is record-keeping. Pin dated snapshots rather than aliases, and store the model identifier returned with every response, where the provider exposes one, not just the name you requested. A commenter on that Astra bug report pointed out that their client logged only the requested model, so there was no way to tell what had actually answered. Log the parameters that actually reached the provider after your framework filled in its defaults.
Log the prompt, tool definitions and retrieval settings that were active, the reasoning and output token counts, and the reason each response stopped. Version the SDK, the inference engine, the embedding model, and the index like your own code. These records won't reveal a provider's internal routing error or an undisclosed experiment. They will tell you whether your application sent the same request with the same recorded configuration.
Then test against those records. Keep a golden evaluation set drawn from real traffic, long conversations and awkward edge cases included, and run it repeatedly. One good or bad answer proves little. What matters is whether the difference holds across the test set. At ICLR 2026, Jonas Kübler and colleagues showed that comparing two model versions sample by sample, with a classic paired test called McNemar's, can separate real degradation from noise.
In their case study, it caught an accuracy drop as small as 0.3%. How small a drop you can catch depends on the size of your test set. The test itself is cheap, though generating enough paired outputs may not be. Shadow a new snapshot against the old one before switching, and diff configuration the way you diff code. For open-weight models, run the checks their creators publish, such as OpenAI's compatibility test for gpt-oss and Moonshot's vendor verifier for Kimi K2.
The idea is a decade old. In 2015, D. Sculley and colleagues at Google described configuration debt in "Hidden Technical Debt in Machine Learning Systems." Their warning still holds: "Hidden debt is dangerous because it compounds silently." The only update is that the configuration now includes someone else's router.
This argument has a boundary, and leaving it out would be selling something.
Sometimes the model really did change. Drift between versions of a service that carries the same name is documented, and logging on the customer's side can't prevent it. It can only catch it. Provider-side faults are harder still: Anthropic's own engineers didn't immediately connect user complaints to the load-balancing change, and customer-side logs are unlikely to find that kind of fault first.
That's where Astra sits as of this writing. The cause is unknown, and from the outside, a quieter default, a fallback route, and post-launch disappointment look the same. What bookkeeping buys is narrower than an answer, and still worth having. It turns "the model got dumber" from a feeling into a short list of things that did or didn't change on your side. If the list comes back clean, you have a stronger case for asking the provider to investigate, and the evidence to support it.
When quality drops, skip the question of whether the model got dumber and ask for the diff. Which snapshot answered, which provider served it, which parameters reached the API, what context survived truncation, which library versions were running, and why did the response stop? If your records can answer, you can rule out the usual suspects quickly and hand the provider a report it can reproduce. If they can't, the first failure isn't the model. Nobody wrote the decisions down, and a default is a decision nobody remembers making.
Others said they had gone back to the previous model, GPT-5.6 Sol. These were reports, not a diagnosis. As of mid-October, OpenAI had not publicly explained those reports, and nobody outside the company could say what, if anything, had changed.
Two months earlier, users had made the same complaint about Sol, and that time there was an answer of sorts. A widely circulated post attributed to OpenAI's Thibault Sottiaux denied Sol had been weakened but said the team had run and reverted experiments that changed its reasoning effort: roughly how much computation the model spends before it answers.
Stories like these get filed under "the model got dumber," which is true from the user's chair and not very useful. A router picks which model runs, an inference stack executes it, a system prompt constrains it, an SDK fills in the parameters you left out, and your application decides what the model gets to see. Any of those can move while the weights sit still.
When an office feels cold, someone checks the thermostat before blaming the furnace. A deployed model has many thermostats, and most are set by defaults nobody inspects.
The name is not the model.
The best-documented case came from Anthropic. On March 4, 2026, it moved Claude Code's default reasoning effort from high to medium to stop pauses that made the interface look frozen. A March 26 caching bug then made sessions that sat idle for more than an hour keep discarding their earlier reasoning on every later turn, and an April 16 system-prompt line capped text between tool calls at 25 words. Users noticed. AMD AI senior director Stella Laurenzo analyzed 6,852 sessions and found shallower reasoning and more premature stopping. Anthropic's April 23 postmortem said the API and inference layer were unaffected. It called the effort change "the wrong tradeoff."
Routers and names fail the same way. When GPT-5's model router broke on launch day in August 2025, Sam Altman said that "the result was GPT-5 seemed way dumber." Earlier that year, Google repointed a dated Gemini endpoint to a newer model, which developers on its forum called a breach of trust. OpenAI's documentation draws the line cleanly: a snapshot locks a model version, while an alias can be repointed. Pinning a dated name protects you only if the provider keeps that name attached to the same model.
Even when the model name stays the same, its defaults can change. When Claude Code's default moved to medium, most users stayed there, even after Anthropic added notices and a selector. A default isn't a suggestion. For most users, it is the product. Defaults move for reasons that look sensible on a dashboard: lower latency, lower token spend, a faster-feeling agent. The cost shows up as a vague sense that the product got less intelligent, and users rarely have a way to connect that feeling to a setting.
The serving stack has opinions.
Below the product settings is the machinery, and it can fail without anyone deciding anything. Anthropic's September 2025 postmortem described three overlapping infrastructure bugs. One bug misrouted Claude Sonnet 4 requests to servers configured for a longer context window, starting at about 0.8% of traffic and reaching 16% at the worst hour after a routine load-balancing change. Another, a compiler bug, could drop the most likely next token from consideration. Anthropic's evaluations missed the degradation, partly because models often recover from a single bad token. The company also said it doesn't reduce quality in response to load. Customers can't audit that claim, but the incident didn't require a conspiracy. Bugs were enough.
Open-weight models make the problem easier to see, because many companies serve the same weights. vLLM's official results for OpenAI's gpt-oss-120b on the AIME 2025 math benchmark show 93.0 at high reasoning effort and 79.6 at medium. That gap matters when a provider mishandles the setting. When Artificial Analysis benchmarked the providers hosting the model at high effort, six scored a median of 93.3%, and Azure scored 80%, a gap of 13.3 points.
In Simon Willison's write-up, Microsoft's Lucas Pickup explained that older vLLM commits "didn't respect reasoning_effort, so all requests defaulted to medium." In that case, the gap came from a serving configuration that ignored the requested setting, not from the weights.
None of this is new, only easier to notice. In 2023, Lingjiao Chen, Matei Zaharia, and James Zou compared the March and June versions of GPT-4. In the revised paper, its accuracy on their prime-number task fell from 84.0% to 51.1%. Arvind Narayanan and Sayash Kapoor pointed out that the test contained only primes, so a shift in which answer the model leaned toward could look like a change in math ability. That's a fair correction, though a product built on those prompts would still have behaved differently in June.
Now check your own code.
So far the story has been "they changed it," which is comfortable because it puts the fault somewhere we can't reach. The less comfortable version is that the same failures live one layer up, in the code we own.
Start with sampling. For years, the standard advice was to use a low temperature for reliable output. Google's Gemini 3 guide now reverses that. It recommends leaving temperature at its default of 1.0 and warns that lower values may cause looping or degraded performance on complex reasoning. A temperature setting that suited one model family can hurt the next, with no change in your code.
Output budgets changed meaning when reasoning models arrived. For newer models, OpenAI deprecated max_tokens in favor of max_completion_tokens, which counts hidden reasoning tokens as well as the visible answer. A tight limit can be spent largely or entirely on thinking. The result can be an empty or cut-off answer from a model that was still working on the problem.
Context limits can change the input without raising an error. Since Ollama 0.15.5, its default context length depends on how much video memory the machine has, from 4,096 tokens on smaller GPUs to 262,144 on the largest. When a conversation overflows, Ollama prunes older messages from the front while trying to keep the system prompt and the latest message. Move the same application to a smaller cloud instance, and the model reads a different conversation. To the user, the model forgot an instruction. To the system, the instruction was never in the prompt.
Retrieval has quiet defaults too. Halve the chunk size without retrieving more chunks, and the model may get much less of the relevant evidence. Change the embedding model without rebuilding the index, and retrieval can break outright. It may also keep working while quietly becoming less accurate. Either way, the model is answering with different evidence from the evidence you thought you gave it.
And the evaluation set usually doesn't notice. Anthropic's April system-prompt change passed several weeks of internal testing. Only broader line-by-line ablations later found a 3% drop on one evaluation. That's a reason not to treat a small smoke test as evidence that a configuration change is harmless.
The fix is bookkeeping.
What Anthropic did afterwards is the useful part. After September 2025, it committed to running quality evaluations continuously against production systems. After April, it added line-by-line ablations for system-prompt changes, review tooling, and gradual rollouts for anything that might trade away intelligence. None of it required retraining the model.
The same discipline works outside the labs, and most of it is record-keeping. Pin dated snapshots rather than aliases, and store the model identifier returned with every response, where the provider exposes one, not just the name you requested. A commenter on that Astra bug report pointed out that their client logged only the requested model, so there was no way to tell what had actually answered. Log the parameters that actually reached the provider after your framework filled in its defaults.
Log the prompt, tool definitions and retrieval settings that were active, the reasoning and output token counts, and the reason each response stopped. Version the SDK, the inference engine, the embedding model, and the index like your own code. These records won't reveal a provider's internal routing error or an undisclosed experiment. They will tell you whether your application sent the same request with the same recorded configuration.
Then test against those records. Keep a golden evaluation set drawn from real traffic, long conversations and awkward edge cases included, and run it repeatedly. One good or bad answer proves little. What matters is whether the difference holds across the test set. At ICLR 2026, Jonas Kübler and colleagues showed that comparing two model versions sample by sample, with a classic paired test called McNemar's, can separate real degradation from noise.
In their case study, it caught an accuracy drop as small as 0.3%. How small a drop you can catch depends on the size of your test set. The test itself is cheap, though generating enough paired outputs may not be. Shadow a new snapshot against the old one before switching, and diff configuration the way you diff code. For open-weight models, run the checks their creators publish, such as OpenAI's compatibility test for gpt-oss and Moonshot's vendor verifier for Kimi K2.
The idea is a decade old. In 2015, D. Sculley and colleagues at Google described configuration debt in "Hidden Technical Debt in Machine Learning Systems." Their warning still holds: "Hidden debt is dangerous because it compounds silently." The only update is that the configuration now includes someone else's router.
What the logs can't see.
This argument has a boundary, and leaving it out would be selling something.
Sometimes the model really did change. Drift between versions of a service that carries the same name is documented, and logging on the customer's side can't prevent it. It can only catch it. Provider-side faults are harder still: Anthropic's own engineers didn't immediately connect user complaints to the load-balancing change, and customer-side logs are unlikely to find that kind of fault first.
That's where Astra sits as of this writing. The cause is unknown, and from the outside, a quieter default, a fallback route, and post-launch disappointment look the same. What bookkeeping buys is narrower than an answer, and still worth having. It turns "the model got dumber" from a feeling into a short list of things that did or didn't change on your side. If the list comes back clean, you have a stronger case for asking the provider to investigate, and the evidence to support it.
The question to ask.
When quality drops, skip the question of whether the model got dumber and ask for the diff. Which snapshot answered, which provider served it, which parameters reached the API, what context survived truncation, which library versions were running, and why did the response stop? If your records can answer, you can rule out the usual suspects quickly and hand the provider a report it can reproduce. If they can't, the first failure isn't the model. Nobody wrote the decisions down, and a default is a decision nobody remembers making.