Can artificial intelligence really think—and do we care?
Even before artificial intelligence burst into public consciousness, there had been an ongoing debate about whether such systems—large language models (LLMs) in particular—could really think.
A 2021 research paper suggested that LLMs do no more than probabilistically linking words and sentences together without considering meaning, and suggested they were mere ‘stochastic parrots’. This paper also asked whether language models can be too big, but their massive growth suggests the answer is ‘not yet’. As their use has taken off, many are convinced of AI’s almost sentient capabilities, finding AI models to be useful as coaches, therapists and more.
However, there has been much debate about whether AI can actually ‘think’. At this point, it is worth taking a step back and reflecting on what we mean by thinking and consciousness. Although we all think we know what these concepts are, at least as they relate to our own experience, there is no rigorous definition nor understanding of where these concepts physically arise from.
If neurologists can forgive my over-simplification, at a biomolecular level our brains consist of multiple layers of neurons with linkages that change dynamically with experience. The field of machine learning started with trying to build mathematical models of these structures, seeking to explain what we observe as human thinking and intelligence. Even smaller scale models turned out to be surprisingly good, even outperforming most humans, but only on some well-defined, narrowly scoped tasks such as image classification or language translation.
More recently, new algorithms, massive increases in computing power and the availability of large volumes of training data have enabled the development of AI products such as ChatGPT, Claude and Gemini. While they are still orders of magnitude less complex (in terms of number of connections) than the average human brain, these products can perform a wide range of tasks to process text-based inputs and generate relevant and useful content output.
It can be argued that if these models do so by simply pattern matching, it may not be so different to the way we learn many things: by observing a large number of samples and coming up with implicit rules to help us predict what will happen in a new scenario. You probably didn’t learn to throw and catch a ball by learning Newton’s laws of mechanics and solving the resulting differential equations, but instead by observing the work and inferring our own heuristics about how the trajectory will vary according to different variables.
Even in those cases where we think we do solve problems by creating our own worldview or model—for example, playing chess by learning the pieces and their characteristics—this is not a physically provable phenomenon. You can’t point to a set of neurons in your brain as forming this model. Who is to say a neural network doesn’t have a similar construct within its internal pattern of billions of connections?
However, we do know that LLMs have major limitations, but then so do people. To begin with, yes, they always make mistakes, but so do the most intelligent humans. When it comes to AI’s phenomenon of hallucinations, we probably all know at least one person in our lives with a similar approach of inventing a plausible-sounding answer rather than admitting their ignorance in response to some questions.
A study by Apple released in June appeared to show models hitting a fundamental limit, not trying to solve a problem even though they still had available resources. However, further investigation showed that actually this appeared to be a case of the model deciding that the path to solving it would take a long time and have a high chance of an error along the way, so it was better not to try. This could again be a personality trait found in some people around us.
The upshot is that asking if AI models can think isn’t really meaningful, because we don’t really have a way to distinguish thinking from not thinking. This doesn’t mean that AI models are some sort of artificial general intelligence. In fact, they are not even close. They may seem capable of a broad range of tasks, but it is still a limited set, and they perform those tasks in a very specific way that is sometimes useful and sometimes not.
Therefore, to build trust and confidence in their use, it is important to understand their limitations. Some of this can come from considering how they are built: what data is used to train them, and what does the training routine actually optimise for? However, such theoretical analysis can only get us so far, especially when using proprietary commercial pre-trained models. Therefore, we also need to do our own experiments, testing their performance in different situations, observing the results and using this to infer what guardrails we need to put around them.
Maybe we should treat working with LLMs just like working with new employees: first look at their qualifications and experience, then see how they go with small tasks to build the confidence to use them more broadly.
However, we need to be aware that they will fail in some cases, and potentially catastrophically—so proceed with caution!
This article was written by Dr Rajiv Shah, Advisory Board member at CAN.B Group and Fellow at the Australian Strategic Policy Institute (ASPI). It was originally published on The Strategist, the commentary and analysis site of ASPI, and is republished here in full with permission. The original article can be found at: https://www.aspistrategist.org.au/can-artificial-intelligence-really-think-and-do-we-care/.