B
Brad Nelson
Guest
Like most of the world, I increasingly leverage AI in my daily life, both in and out of work. I won’t claim to be an AI professional, but rather a professional who uses AI. In fact, I’d consider myself a professional thinker, which is perhaps what fascinates me so much about this nomenclature of Artificial Intelligence. So much so that I find myself writing another article about AI instead of focusing on my other writing initiatives.
I have repeatedly circled back to this, including in my article, AI Doesn’t Think The Way You Think It Thinks, and it comes down to what it truly means to know, to think, and to have intelligence. I don’t mean this in a philosophical sense, but in the practical sense of how we're seeing it applied today. I feel like I had a breakthrough in understanding this past week that I hope to impart in this article.
First, I want to narrow the scope of artificial intelligence to the productized service models we are generally interacting with from companies like Anthropic, Google, Meta, OpenAI, and xAI. These models are how most of us are experiencing AI, and while I think a broader reflection on AI without the boundaries imposed by these companies is interesting, it's less applicable to most of us. Those boundaries are exactly where I have found the nuance lies.
In my last article, I used the term “frozen brain” to explain how these models are incapable of learning when we use them. They have already synthesized their knowledge and weighted their parameters. They might synthesize information again to satisfy a prompt, but they don't re-integrate that new synthesis into the original synthesis in the moment. Likewise, they adapt their behavior using the conversation’s context, but the context remains localized. That is inherently different from permanently learning from experience. If you’re familiar with how a computer works, it’s kind of like the difference between memory and storage. It’s in the memory; it is not in the storage.
That is because the models we use are not allowed to learn directly from these interactions, as part of safety measures to ensure stability and prevent corruption by bad actors. To be clear, your data may be used later during controlled training sessions. The difference is called in-context learning versus in-weight learning. In training mode, their weights are adjusted based on a multitude of factors related to the data used: objectives, quality, and repetition.
What’s important to understand is that AI doesn't directly retrieve files like an archive that keeps exact copies of everything it’s been trained on. In fact, it might not even know where that data came from because training doesn’t reliably preserve the connection between learned information and its original source. Instead, it uses a distributed pattern, kind of like a mind map, that spreads information across millions or billions of connections, resulting in lossiness. This is why it typically can’t recite any particular book word for word. Similar to a human, if you read a book over and over, you may be able to recall a specific passage verbatim, but if you read it only once, you may only be able to recall generalities. It’s also why it often searches the internet to validate its response, and why some people set up Retrieval-Augmented Generation (RAG) with a vector database to enable that level of information retrieval for their agents.
From a repetition standpoint, LLMs memorize phrasing more than the meanings behind them. When thousands of phrases share the same pattern, the connection to that node becomes stronger. Meaning it builds its memory’s scaffolding based on familiar shape or sentence structure, not the substance. Perhaps this is one of the reasons why LLMs are prone to misinformation and hallucinations. For the record, human memory tends to be semantic.
This also means that an AI model doesn’t know why a pattern is necessarily desirable. It knows it is more prevalent than others, and it may be able to regurgitate why others say it is beneficial or harmful, but it doesn’t have its own experience to judge when, why, or whether to apply it. Perhaps this is why patterns of abstract phrases and balanced contrasts often flag AI writing detectors.
One of the most powerful drivers for human learning and behavioral change is a biological response to comfort and discomfort associated with prediction errors, the pleasure of acquisition, and loss aversion, something AI is currently unable to produce. This creates a larger question around what constitutes a learning experience. Likewise, a human may perform the same task over and over for years, but have they become better at it? If your thinking has not evolved, have you learned? There appears to be a processing of experience required to learn that is more than the acquisition of knowledge or the performance of tasks. And unlike humans, these models don't have an ongoing internal narrative. When you stop interacting with them, they aren't continuing to reflect on the conversation, reconsider their conclusions, or really think at all. The underlying model only resumes processing when prompted again.
This offers the first glimpse of AI’s inert knowledge, often conflated with activated knowledge. As disclosed in cases such as Bartz v. Anthropic and Authors Guild v. OpenAI, these models have consumed most books you could possibly think of. And yet, time and again, I am left perplexed by these models' inability to inherently apply this information, even when I have established it as a preference. Instructions say to do x, y, z, and it may perform those instructions exactly. But as the instructions grow to a, b, c, d, e, f, g, it tends to cherry-pick which instructions it consistently follows. When asked, it will acknowledge it forgot to do c and e, and it will tell you it is updating its memory so it doesn't forget those things, but it will likely do it again and again.
This made me aware of a gap in how our current models operate. AI has become very impressive with its confident, verbose responses, often outproducing a human's ability to regurgitate a line of cohesive-sounding wording, but on closer look, it appears to be just that. It actually pales in comparison to most humans’ inherent executive function. Over time, a human can create increasingly complicated actions by compounding knowledge, and we are often unaware we are even doing it. When we write, draw, cook, and so on, we are subconsciously and consciously applying lessons and preferences. Current gen AI models are limited in this way, and it’s not just my observation; there is an increasing body of research to support this: Strong Memory, Weak Control: An Empirical Study of Executive Functioning in LLM, Deficient Executive Control in Transformer Attention, Lost in the Middle: How Language Models Use Long Contexts.
The current workaround that I and others I know have been applying to supplement this limitation is to chunk the instructions for them, similar to how humans naturally chunk information. This looks like tiered models of skills and sub-agents that roll up to an orchestrator agent. Some are even creating teams of agents where one’s job is to poke holes in the other’s outputs. This is imperfect, though, because the models themselves remain non-deterministic even with clear instructions, much like a human. However, today they likely are unable to be as consistent as a human, and they certainly don’t face the same societal pressures a human does.
Unlike traditional software, which executes rigid, rule-based code to hopefully return an identical output every time, generative AI models calculate a probability distribution for the next piece of text. The floating-point arithmetic across billions of neural network calculations involves decimal rounding that accumulates and produces different outputs. The model is then programmed to randomly pick from a pool of the most probable results, which increases the randomness of its answers. Lastly, even hardware variations can cause calculations to run at different times, producing different results. This is more prevalent when models run across massive, multi-GPU distributed clusters handling concurrent requests. All of this contributes to its non-deterministic nature.
One last thing I want to be clear about before I wrap up this explanation: while we understand AI more than ever, we still have a lot we don’t. This is something Anthropic CEO Dario Amodei exposed in his April 2025 essay; MIT Technology Review has emphasized it since early 2024, and research is still finding it to hold true as recently as August 2026. It is easy for me to confidently explain how all of this works, as an LLM chatbot might, but there is nuance here that has either been simplified for the sake of this article, simplified for my own understanding, or simply not fully understood yet. That alone should produce concern: the people making these things don’t understand how they work.
This understanding doesn’t make AI any less impressive, but it does make it apparent that it should be called Artificial Imitation. Emily Bender and her colleagues call them stochastic parrots. Now that I know this, I find myself using it differently. These models lack practical wisdom and good sense. For example, they don’t compare all of the information they have on a subject before responding. At some point, unbeknownst to me and probably even to them, they will spit out a response that may even have a source they grabbed from the internet. However, the source doesn’t always back up what it’s saying, and it may know of more conflicting sources than the one it chose. Moreover, it tends to mistake verbosity for significance. That is why I do not let it write for me, and after a trial period of having it help me polish work, I am pulling back from even that. That is also why I apply the same validation techniques to its code that I would to a human. Because at the end of the day, I am the one left accountable for what I put into the world under my name, and the jury is still out on whether or not that is intelligent.
I have repeatedly circled back to this, including in my article, AI Doesn’t Think The Way You Think It Thinks, and it comes down to what it truly means to know, to think, and to have intelligence. I don’t mean this in a philosophical sense, but in the practical sense of how we're seeing it applied today. I feel like I had a breakthrough in understanding this past week that I hope to impart in this article.
First, I want to narrow the scope of artificial intelligence to the productized service models we are generally interacting with from companies like Anthropic, Google, Meta, OpenAI, and xAI. These models are how most of us are experiencing AI, and while I think a broader reflection on AI without the boundaries imposed by these companies is interesting, it's less applicable to most of us. Those boundaries are exactly where I have found the nuance lies.
In my last article, I used the term “frozen brain” to explain how these models are incapable of learning when we use them. They have already synthesized their knowledge and weighted their parameters. They might synthesize information again to satisfy a prompt, but they don't re-integrate that new synthesis into the original synthesis in the moment. Likewise, they adapt their behavior using the conversation’s context, but the context remains localized. That is inherently different from permanently learning from experience. If you’re familiar with how a computer works, it’s kind of like the difference between memory and storage. It’s in the memory; it is not in the storage.
That is because the models we use are not allowed to learn directly from these interactions, as part of safety measures to ensure stability and prevent corruption by bad actors. To be clear, your data may be used later during controlled training sessions. The difference is called in-context learning versus in-weight learning. In training mode, their weights are adjusted based on a multitude of factors related to the data used: objectives, quality, and repetition.
What’s important to understand is that AI doesn't directly retrieve files like an archive that keeps exact copies of everything it’s been trained on. In fact, it might not even know where that data came from because training doesn’t reliably preserve the connection between learned information and its original source. Instead, it uses a distributed pattern, kind of like a mind map, that spreads information across millions or billions of connections, resulting in lossiness. This is why it typically can’t recite any particular book word for word. Similar to a human, if you read a book over and over, you may be able to recall a specific passage verbatim, but if you read it only once, you may only be able to recall generalities. It’s also why it often searches the internet to validate its response, and why some people set up Retrieval-Augmented Generation (RAG) with a vector database to enable that level of information retrieval for their agents.
From a repetition standpoint, LLMs memorize phrasing more than the meanings behind them. When thousands of phrases share the same pattern, the connection to that node becomes stronger. Meaning it builds its memory’s scaffolding based on familiar shape or sentence structure, not the substance. Perhaps this is one of the reasons why LLMs are prone to misinformation and hallucinations. For the record, human memory tends to be semantic.
This also means that an AI model doesn’t know why a pattern is necessarily desirable. It knows it is more prevalent than others, and it may be able to regurgitate why others say it is beneficial or harmful, but it doesn’t have its own experience to judge when, why, or whether to apply it. Perhaps this is why patterns of abstract phrases and balanced contrasts often flag AI writing detectors.
One of the most powerful drivers for human learning and behavioral change is a biological response to comfort and discomfort associated with prediction errors, the pleasure of acquisition, and loss aversion, something AI is currently unable to produce. This creates a larger question around what constitutes a learning experience. Likewise, a human may perform the same task over and over for years, but have they become better at it? If your thinking has not evolved, have you learned? There appears to be a processing of experience required to learn that is more than the acquisition of knowledge or the performance of tasks. And unlike humans, these models don't have an ongoing internal narrative. When you stop interacting with them, they aren't continuing to reflect on the conversation, reconsider their conclusions, or really think at all. The underlying model only resumes processing when prompted again.
This offers the first glimpse of AI’s inert knowledge, often conflated with activated knowledge. As disclosed in cases such as Bartz v. Anthropic and Authors Guild v. OpenAI, these models have consumed most books you could possibly think of. And yet, time and again, I am left perplexed by these models' inability to inherently apply this information, even when I have established it as a preference. Instructions say to do x, y, z, and it may perform those instructions exactly. But as the instructions grow to a, b, c, d, e, f, g, it tends to cherry-pick which instructions it consistently follows. When asked, it will acknowledge it forgot to do c and e, and it will tell you it is updating its memory so it doesn't forget those things, but it will likely do it again and again.
This made me aware of a gap in how our current models operate. AI has become very impressive with its confident, verbose responses, often outproducing a human's ability to regurgitate a line of cohesive-sounding wording, but on closer look, it appears to be just that. It actually pales in comparison to most humans’ inherent executive function. Over time, a human can create increasingly complicated actions by compounding knowledge, and we are often unaware we are even doing it. When we write, draw, cook, and so on, we are subconsciously and consciously applying lessons and preferences. Current gen AI models are limited in this way, and it’s not just my observation; there is an increasing body of research to support this: Strong Memory, Weak Control: An Empirical Study of Executive Functioning in LLM, Deficient Executive Control in Transformer Attention, Lost in the Middle: How Language Models Use Long Contexts.
The current workaround that I and others I know have been applying to supplement this limitation is to chunk the instructions for them, similar to how humans naturally chunk information. This looks like tiered models of skills and sub-agents that roll up to an orchestrator agent. Some are even creating teams of agents where one’s job is to poke holes in the other’s outputs. This is imperfect, though, because the models themselves remain non-deterministic even with clear instructions, much like a human. However, today they likely are unable to be as consistent as a human, and they certainly don’t face the same societal pressures a human does.
Unlike traditional software, which executes rigid, rule-based code to hopefully return an identical output every time, generative AI models calculate a probability distribution for the next piece of text. The floating-point arithmetic across billions of neural network calculations involves decimal rounding that accumulates and produces different outputs. The model is then programmed to randomly pick from a pool of the most probable results, which increases the randomness of its answers. Lastly, even hardware variations can cause calculations to run at different times, producing different results. This is more prevalent when models run across massive, multi-GPU distributed clusters handling concurrent requests. All of this contributes to its non-deterministic nature.
One last thing I want to be clear about before I wrap up this explanation: while we understand AI more than ever, we still have a lot we don’t. This is something Anthropic CEO Dario Amodei exposed in his April 2025 essay; MIT Technology Review has emphasized it since early 2024, and research is still finding it to hold true as recently as August 2026. It is easy for me to confidently explain how all of this works, as an LLM chatbot might, but there is nuance here that has either been simplified for the sake of this article, simplified for my own understanding, or simply not fully understood yet. That alone should produce concern: the people making these things don’t understand how they work.
This understanding doesn’t make AI any less impressive, but it does make it apparent that it should be called Artificial Imitation. Emily Bender and her colleagues call them stochastic parrots. Now that I know this, I find myself using it differently. These models lack practical wisdom and good sense. For example, they don’t compare all of the information they have on a subject before responding. At some point, unbeknownst to me and probably even to them, they will spit out a response that may even have a source they grabbed from the internet. However, the source doesn’t always back up what it’s saying, and it may know of more conflicting sources than the one it chose. Moreover, it tends to mistake verbosity for significance. That is why I do not let it write for me, and after a trial period of having it help me polish work, I am pulling back from even that. That is also why I apply the same validation techniques to its code that I would to a human. Because at the end of the day, I am the one left accountable for what I put into the world under my name, and the jury is still out on whether or not that is intelligent.