The Evolution of Imagination

What makes mammals different is our ability to simulate

My last two essays described the evolution and workings of associative learning, and I outlined Simona Ginsburg and Eva Jablonka’s argument that consciousness evolved to make complex or unlimited associative learning possible. They conclude that consciousness is quite common among animals: almost all vertebrates, as well as some mollusks and arthropods, are probably conscious. However, large differences exist among the types of consciousness experienced by different animals. Mammals have many cognitive and behavioral abilities, which I will describe below, that their reptile and fish ancestors do not: mammals are better at learning to navigate spaces and are capable of more finely tuned movements than their ancestors are. They are also capable of goal-directed behavior and counterfactual reasoning. Some mammals, the primates, can form detailed theories about the intentions and perspectives of other animals.

In his 2023 book A Brief History of Intelligence: Evolution, AI, and the Five Breakthroughs That Made Our Brains, AI researcher Max Bennett proposes that it is their capacity for imagination that makes mammals different. While non-imagining brains can experience only their present reality, imagining brains can simulate experiences that took place in the past and that might take place in the future. Imagining brains can also form cognitive models of reality to guide their actions. Bennett demonstrates that imagination is responsible for a wide range of novel mammalian abilities and behaviors. While he focuses on mammals, he also notes that imagination may have evolved independently in birds and octopus.

Bennett locates the source of imagination in the neocortex, a brain region that first evolved in mammals. The neocortex is a sheet of neurons that occupies much of the surface of mammalian brains. While this brain region constituted only a small fraction of the brains of early mammals, it exploded in size in the human lineage to become 70% of the brain. In humans, the neocortex is divided into three basic regions: the sensory neocortex, the prefrontal cortex, and the motor cortex. Each of these regions is responsible for different types of imagination.

The human neocortex

The sensory neocortex models external reality

When you look at the world, are you seeing what’s there, or are you seeing a theory of what’s there? Consider this image:

Rather than seeing just three circles with missing pieces, the human brain interprets the white space as a triangle. There is no triangle there, but we see one because our brains not only take in sensory data, but constantly work to infer what causes those data. Such inferences are useful in interpreting incomplete and confusing sensory information. More examples of how the brain imposes interpretations on sense data can be seen here.

It is the role of the sensory neocortex to develop a model or simulation of external reality. It constantly generates predictions about what its external reality consists of. This generative function of the sensory neocortex explains many human attributes. It is because you possess such a model of reality that you are able to find your way around a familiar space like your home in the dark. Also, when brain damage causes the neocortex to stop receiving visual input, people don’t stop seeing. Rather, they begin to hallucinate. The neocortex goes on generating predictions about what is in the world, which are now unconstrained by sensory data. According to neuroscientist Anil Seth, our perception of the external world is a “constrained hallucination,” caused by the interaction of our inner model with the information coming in through our senses. Dreaming is another effect of neocortical simulations.

The simulating power of the sensory neocortex explains much about the distinctive nature of mammalian cognition. As a rat learns to negotiate a maze in which food is stored, it will sometimes pause and look back and forth along different possible routes. This behavior was inexplicable to early neuroscience researchers who thought that rats learn only behaviors that are positively reinforced. The researchers had never rewarded rats for looking back and forth, so why were they doing it? The mystery was not cleared up until 2007, when neuroscientists became able to record the activity in a rat’s hippocampus, which is responsible for spatial location. These recordings showed that the rat was imagining different places along possible future paths that it had learned in previous experiences with the maze.

Simulation gives mammals an ability that reptiles and fish don’t have: navigation from memory. Imagine an experiment: place a fish on one side of a tank with a transparent barrier in the middle. The barrier has a hole in one corner. The fish swims around, discovers the hole, and further explores the tank. Now place food on the opposite side of the tank from where the fish is. The fish swims directly towards the food, but it keeps on running into the barrier. Because the fish has no internal model of its external reality, it cannot figure out to go through the hole to the food. Fish who have explored a tank set up like this reach the food no faster than those who are experiencing the tank for the first time. A rat, however, is much better able to learn to solve problems like this because its sensory neocortex builds a model of its environment as it explores. Therefore, it can quickly figure out to run through the hole.

The capacity for simulation also undergirds episodic memory, another ability that first appeared in mammals. Episodic memory is memory of a particular situation from one’s past. You may remember what happened and how you felt at your wedding or high school graduation. Episodic memories are notoriously faulty, which is why people remember events differently and argue about what happened in the past. The reason for this fallibility is that our past, just as much as our present and future, is a simulation molded by the neocortex. A memory is constituted not only by sensory traces, but also by your theory of what was happening at the time.

Episodic memory permits animals to learn from counterfactuals. If a fish hunts for invertebrate prey and catches only one, whereas another fish, hunting in a different location, finds four, the fish will not change its behavior, even if it can see its fellow’s greater success. The fish is not capable of understanding that if it had done something else, a counterfactual, it might have gotten more food. However, a rat who makes a non-optimal choice during an experiment shows signs of regretting its error. If it chooses to go into one chamber of a maze when better food would have been available if it had stayed in the chamber where it was, the rat looks back at its path, and its neurons simulate the taste of the food that it might have eaten. Also, the rat’s behavior changes based on its experience.

Counterfactual learning is the basis for more precise causal learning than fish and reptile brains are capable of. The ability to simulate its past and imagine different outcomes helps an animal figure out why a course of action turned out well or badly.

The prefrontal cortex triggers simulations

Simulation is obviously a major innovation that enables more intelligent interactions with the world. But it isn’t worth much if the animal doesn’t have a guide that tells it what to simulate. As I noted elsewhere, all innovations in learning are a Pandora’s box that create just as many problems as they solve. The rise of associative learning brought with it the problem of how to prevent unproductive overlearning, and the same type of problem bedevils simulating cognitive systems. Unconstrained simulation, or “oversimulating,” would be likely to paralyze an animal rather than helping it. Here Bennett draws a useful analogy with the problems faced by AIs. The AI programs behind high-level chess playing also simulate what will happen if a given move is played. However, they cannot simulate the outcome of all possible moves: the computational problems are simply too great. Rather, these programs use a separate process to decide on which moves are the most plausibly winning, and then the AI simulates the outcomes only of those moves.

Bennett proposes that a very elegant mechanism solves the problem of oversimulating. Whereas the sensory cortex gets most of its input from the senses, another part of the neocortex, called the agranular prefrontal cortex (aPFC) gets input from processes in the older regions of the brain, like the hippocampus, hypothalamus, and amygdala. These generate sequences of places, desires, and emotions. The input of the aPFC is, then, the brain’s internal state. Bennett thinks that the aPFC is responsible for simulating the animal’s own intentions. It interprets brain activity and concludes that the animal wants water or food or sleep. Such information is probably quite noisy, just as sensory input is, so the interpretative tasks of the sensory neocortex and the aPFC are quite similar.

The aPFC solves the oversimulating problem by triggering a simulation only when brain activity suggests that the animal has contradictory goals. If a rat comes to a fork in the maze and one impulse causes it to want to turn right for food and another left for water, the aPFC begins generating simulations of each course to help it decide.

Goal-directed behavior

It is from the brain’s ability to simulate its own intentions that planned, goal-directed behavior arises. The aPFC permits an animal to recognize its overall goals and facilitates planning to achieve these goals.

Bennett draws a contrast between goal-directed behavior and the automatic, habitual behavior that happens without the representation of a goal. Such habitual behavior is the outcome of associative learning. The pigeon who pecks a button three times when it hears a tune learned this behavior through trial and error, not through representing a goal to itself and planning how to reach it.

We can recognize the distinction between habitual and goal-directed behavior in ourselves. It is the difference between driving to a new job for the first time and driving to the same place after having done it every workday for a year. The first time, a person will likely show goal-directed behavior. They represent to themselves where they want to go, and they plan the steps to reach that destination. But with time, the route becomes so habitual that people stop paying attention to it. People often don’t even remember habitual actions like driving to work. This distinction between habitual and goal-directed learning is similar to Daniel Kahnemann’s famous distinction between System 1 and 2 cognition, or thinking fast and slow.

The pre-frontal cortex is designed to handle this goal-directed behavior through the mechanisms of attention, working memory, executive control, and planning. Simulations are inexact so the animal needs further guidance in goal-directed behavior, which is provided by attention. Attention also makes the animal focus on goal-relevant stimuli so that it doesn’t forget its goal. Working, or short-term, memory is necessary so that the animal can keep the plan in its mind for long enough to accomplish it. Executive control is responsible for inhibition, which is necessary to block behaviors that contradict the organism’s goal. Inhibition is the origin of willpower and self-control.

This capacity for inhibition is another major difference between mammalian and reptile behavior. Many lizards have a hard-wired preference for green over red because vegetation is their natural habitat. If you try to train them to associate the color red with good food and green with worse food, it will take them hundreds of repetitions to override their hard-wired behavior and learn the association. But mammals can inhibit hard-wired as well as learned associations much more effectively than reptiles, so they can learn new behaviors more easily.

Other types of simulation

The capacity for simulation is also the explanation of mammals’ fine motor skills and large repertoire of movements. There is no reptile equivalent of the dexterity of a cat as it noiselessly leaps exactly as high and far as it needs to to reach its perch. The cat’s ability is due to a part of the neocortex called the motor cortex. Not all mammals possess a motor cortex, only those that descended from rodents, like cats, dogs, monkeys, and so forth. When the motor cortex is damaged, mammals become less capable of moving with precision and learning new movements. Bennett believes that the motor cortex works by simulating a mammal’s future movements. When an animal must do something physically difficult, the motor cortex helps them visualize and plan their movements.

We’ve seen that one aspect of the mammalian boost in intelligence comes from the brain’s ability to inspect its own neural activity and model its intentions. The evolution of primates took this self-interpretative ability to a new level. While mammals simulate situations and intentions, primates can also simulate themselves. They can develop theories about their own thought processes and critique them, an ability known as metacognition. This ability comes from a new brain region, the granular pre-frontal cortex (gPFC), which takes input from aPFC, the region that simulates intentions. If the mammalian brain simulates a reality in which it turns left and finds water, the primate brain can think about the reasons why it formulates intentions and holds beliefs. It can reason, “I want water because I’m thirsty and water relieves thirst,” and “I know that there is water behind the rock because I saw a zookeeper put it there.”

It is this metacognitive awareness of their own thought processes that enabled primates to understand the thought processes of others. One of the major topics in research on primate cognition is theory of mind, or primates’ ability to understand the mental states of other animals. Apes can form accurate theories about the what other animals intend and know. If chimpanzee 1 has seen a zookeeper place food somewhere, but knows that chimpanzee 2 has not seen that, chimpanzee 1 knows that chimpanzee 2 doesn’t know where the food is. Apes can also distinguish between intentional and accidental actions. Bennett believes that apes can understand others’ minds because they can project the self-understanding created by the gPFC onto other animals.

Bennett goes on to describe other aspects of primate and human cognition, and imagination is crucial to all of them. For example, linguistic communication is a way of imagining of the state of mind of another person.

Animals capable of associative learning thrived because they were more behaviorally flexible than non-learning animals. An animal that can learn a novel link between a stimulus and a behavior has more options for behavior than an animal that cannot. The evolution of imagination greatly expanded animals’ ability to respond with flexibility to the trials of life. Imagining animals have models of reality that permit them more options than their ancestors had when they navigate through space, and their capacity for goal-directed behavior means that they are no longer the slaves of habit. The period since the extinction of the dinosaurs 66 million years ago is known as the Age of Mammals because the small rodent-like creatures that lived under the shadow of the dinosaurs evolved into such a wide array of new forms and expanded into so many new habitats. Imagination is one of the major reasons why mammals have thrived.