........................ TitleIntro    

Ever since the earliest days of the cognitive revolution in psychology, philosophers and psychologists have tended to think of thoughts as analogous to sentences.  This single-minded focus on sentence-like representations has led to the neglect of mental imagery.  In the 1970’s and 1980’s, there was a lively debate over the existence of mental imagery (Pylyshyn 1973, Kosslyn 1975).  But researchers have still not given imagery its due in theories of how the mind represents states of affairs in the world, and they have almost completely neglected the question of how we manage to solve practical problems by means of manipulating mental images.

We often use mental imagery to solve practical problems.  If I am an experienced builder of wooden bird houses, I can build a bird house without a plan on paper.  I can imagine what the finished bird house should look like.  I can imagine what the dimensions of the various pieces should be.  I can cut the pieces of wood to the dimensions I have imagined.  I can then put the pieces together to form a bird house that looks just like what I had imagined.

In using mental imagery in such ways, we exercise a distinctive kind of knowledge – call it imagistic knowledge.  The necessity for such knowledge stands out clearly in cases where we lack the necessary knowledge or fail to apply the knowledge we possess.  Here are some examples:


The wrapping paper.  I want to wrap a certain box in gift wrap.  I imagine how big the piece of paper needs to be in order to cover the whole box and cut a piece of that size from the roll.  But when I try to use it to wrap the box, I discover that it is not big enough.

The tray of glasses.  As an employee in a bar, I carry a tray of glasses from the dishwashing machine to the counter in front of the cabinet where the glasses are stored.  However, I place the tray on the counter in such a way that almost half of it does not make contact with the counter.  Then I start to remove glasses from the back of the tray in order to put them into the cabinet. When enough glasses have been removed from the back, the weight of the glasses in front causes the tray to tip, so that the remaining glasses fall to the floor and shatter.

The bank shot.  We are at the end of a game of pool.  I need to sink the 4 ball into the nearest corner pocket.  However, it is resting against the cushion of the side of the table.  I imagine that if I shoot the cue ball so that it just grazes the side of the 4 ball, the 4 ball will roll into the corner pocket.  But this is a mistake, because however lightly the cue ball grazes the 4 ball, the 4 ball will ricochet somewhat from the cushion. 

The screw.  Having screwed a metal screw into a piece of wood by turning it clockwise with a screwdriver, I may imagine that I can unscrew the screw by again turning it clockwise with the screwdriver. 


We do not always make mistakes like these.  We typically avoid them because we possess some kind of knowledge that enables us to avoid them. Accordingly, the puzzle of imagistic cognition can be formulated thus:  What kind of knowledge is it that is lacking in these examples and which is present in cases where we use our imaginations more successfully? 

The puzzle can be deepened by considering the traditional conception of knowledge.  Traditionally, thoughts have been conceived of as mental states having propositional content.  For instance, the thought that some mammals lay eggs has the propositional content that some mammals lay eggs.  Propositional contents can be conceived as also being the meanings of spoken sentences (or more precisely, of utterances of sentences).  The meaning of the German sentence, “Es gibt Säugetiere, die Eier legen” is the proposition that some mammal lay eggs.  So thoughts have been conceived of as mental states with contents of a sort that can typically be expressed in sentences.  For instance, the sentence, “Es gibt Säugetiere, die Eier legen” expresses the thought that some mammals lay eggs.

Traditionally, knowledge has been thought of as consisting of proposition-bearing thoughts that have a special status.  Roughly, knowledge consists of thoughts that are both true and in some sense justified.  Call such knowledge propositional knowledge.

The puzzle of imagistic cognition runs deep, and is actually puzzling, because it does not seem that the knowledge that is employed in imagistic cognition can be conceived of as propositional knowledge.  In some cases, the requisite knowledge may indeed be knowledge of facts — same kind of facts that can be expressed in sentences.  Perhaps I have simply learned that screws that screw in clockwise unscrew counterclockwise, and I do not have to use my imagination to solve this problem at all. 

In other cases, though, it seems clear that what we know is not a proposition that could be expressed in a sentence, and the problem does not seem to be just that we lack an appropriate vocabulary.  In the wrapping paper example, we could try saying that the knowledge that I lack is the knowledge that this piece of paper isn’t big enough.  But that is at most a conclusion we can state as a consequence of successful imagining.  If we imagine successfully, then our imagining involves picturing the paper being wrapped around the box and “seeing” in our mind’s eye that the paper does not go all the way around.  If we are skilled at gift wrap, then our capacity to judge correctly how big the piece of paper has to be will be the consequence of some kind of general knowledge.  It is quite unclear what kind of propositional thought could do this work of the imagination. 

Whatever words we might use to try to express what we know, those words seem to be not an accurate reflection of what we know in using our imaginations.  In the example of the tray of glasses, one might say that what I fail to know is that if the weight of the glasses is not supported by the table, then the tray will tip over and the glasses will fall.  But consider my use of the word “weight” in this formulation.  It does not seem obvious that a person who avoids the barman’s error would have to have any particular conception of weight, for instance, as a quantity that is measured on a scale or as quality such that more of it makes a thing harder to lift.  

Furthermore, if the knowledge employed in imagination really does consist in propositional thoughts, then, contrary to our initial assumption, mental imagery plays no crucial role in problem solving.  In the example of the bird house, that which counts as knowledge in the use of the imagination would consist only of propositional knowledge.  The cognitive problem of building the bird house would just be a matter of deducing a solution from premises constituting the contents of some propositional knowledge.  The mental images that we thought we were using to solve the problem would turn out to be nothing more than illustrations accompanying a thought process that makes no essential use of them. 

It would be tempting to say that the knowledge that we employ in imagistic cognition is the product of trial and error.  While that is true, it is not especially helpful.  What do we try and how do we learn from our errors?  Imagine that a child has observed a dancer lift his left leg and his right arm while remaining upright.  On this basis, the child might realistically imagine a dancer lifting his right leg and his left arm while remaining upright.  But the child would not imagine realistically if she imagined the dancer lifting both his left leg and his right leg while remaining upright.  What enables the mind to imagine what is realistic while avoiding imagining what is unrealistic?

As far as we are aware, philosophers and cognitive scientists have never squarely confronted this question:  What kind of knowledge is employed in imagistic cognition?  However, there is currently a great deal of research being conducted into closely related questions concerning perception and mental imagery.  So the time is ripe to begin an investigation into imagistic cognition. 


Despite a dearth of research directly focused on imagistic cognition, there is a great deal of contemporary research that is relevant to the topic.  Precisely because there is so much current research that borders on the topic of this project, without directly addressing it, we contend that the time is ripe for closer attention to the problem of imagistic cognition.  Here we categorize the cognate research into four categories corresponding to the subprojects that will shape the research we propose to do.


In the 20th century, philosophy rediscovered the problem of perceptual representation.  In the early 20th century, philosophers such as Bertrand Russell (1912) and C. D. Broad (1925) tended to see in perceptual representation nothing more than a collection of sense data.  In the last two decades of the 20th century, due to a variety of influences (prominently Sellars 1956, but also relevant work in psychology) the nature of perceptual representation again became a topic of philosophical debate.  This topic is a foundational concern for our investigation into the nature of imagistic cognition because the principles of imagistic cognition are likely to be formulated in terms of the contents of imagistic representations.

Do perceptions have propositional content?

On one side of this issue are philosophers who hold that perceptual representations have propositional contents.  For instance, if I see two trees in the distance, my perception might represent the fact that the tree on my left is further away than the tree on my right.  Philosophers who defend this view include McDowell (1996), Siegel (2006, 2014) and Glüer (2009). 

On the other side of this issue are philosophers who hold that, while perceptual representations do represent, they do not have propositional contents.  These include (the later) Brewer (2006), Crane (2009). Gauker (2012) and Echeverri (2017).

A question that arises for those who deny that perceptions and mental images have propositional content is:  What kind of content might they have?  On this question there are authors who conceive of images as representing by means of some kind of map-like isomorphism between the representation and the represented (Peacock 1992 could be cited here, with some qualifications; also Camp 2007, Rescorla 2009 and Nanay 2010b).  Others suppose that the mind contains a perceptual similarity space and that perceptions, and more generally, images, represent by virtue of their location in that similarity space and a mapping of similarity space into an objective quality space (Edelman 1995; Gauker 2011, 2012; Berger 2015). 

This issue is directly relevant to the puzzle of imagistic cognition inasmuch as the principles of imagistic cognition are likely to be formulated in terms of the representational contents of perceptual and imagistic representations.  If perceptions and images do have propositional contents, then that would be a reason to expect that the principles of imagistic cognition are general propositions of some kind.  Otherwise, if the contents are of a different kind, then we should expect to formulate the principles of imagistic cognition in terms of those other kinds (e.g., the locations of objects and scenarios in an objective quality space).

Top-down influences on perceptual representation

A closely related issue, in both the philosophical and the psychological literatures, is the extent to which, and the ways in which, our perceptual and imagistic representations can be shaped by our propositional thoughts.  The psychological literature is replete with purported demonstrations that our expectations and prejudices can literally affect our perceptions (Levin and Banaji 2006, Lupyan et al. 2010).  On the other side, optical illusions seem to demonstrate that our perceptions are not affected by what we know (Nanay 2010a).  A recent review by Firestone and Scholl (2017) challenges the interpretation of many of the experimental results purporting to demonstrate such top-down effects.  The issue is further adjudicated in Teufel and Nanay (2017).

On the side of the philosophical literature, some authors have argued that the acquisition of expertise can change the perceptual appearance of objects and on this basis have claimed that perceptual representations must have propositional contents (e.g. Siegel 2009), or have argued that the phenomenon of seeing-as (e.g., the duck-rabbit, the Necker cube) lead to the same conclusion.  Against this, Gauker (2017a) argues that the effect of expertise on perceptual representation can be accommodated in terms of a perceptual similarity space approach to perceptual/imagistic representation.  See also Briscoe (2015).

Correctness conditions for imagistic representation

The question of the conditions under which an imagistic representation is correct goes beyond the question of what it represents and also concerns the cognitive role of the representation.  If an image is a memory, then it has to represent what has actually happened.  If it represents an option in a decision problem, then it has to represent an attainable possibility or what would happen if certain actions were taken, and the one we choose has to accurately represent what will happen.  As these examples, illustrate, we need a taxonomy of various kinds of mental images according to the relation they are expected to stand in to reality. Very little has been written on this topic.  Some basic distinctions are drawn in Gauker (2011).  The topic has been explored in a preliminary way by Langland-Hassan (2015), but his attempt can be criticized for relying too heavily on analogies to propositional attitudes, such as believing and desiring.

Amodal completion

It is supposed by many that in perception we in some sense represent the occluded parts of the objects we perceive.  In an example from Nanay (2010a), if we see a cat behind a picket fence, and the cat’s tail is occluded by one of the slats of the fence, we nonetheless in some sense perceive a whole cat, including the tail.  The problem is to make sense of this idea that we in some sense perceptually represent parts of an object that we do not literally perceive.  Nanay, like Sellars (1978) before him, has suggested that in amodal completion we integrate with our perception what we merely imagine.  This solution has been criticized by Briscoe (2011, 2018) and by Gregory (2017). 

This issue of amodal completion is directly relevant to the puzzle of imagistic cognition inasmuch it is plausible that the knowledge that enables us to “complete” our representation of the occluded parts of the object of perception is the same in kind as the knowledge that constrains our imagistic problem solving.

We intend this project to be interdisciplinary, of interest to psychologists and possibly even computer scientists. Our own theories will be informed by the theoretical tools that researchers outside of philosophy have already invented.  Our aim will be to characterize imagistic cognition in ways that researchers outside of philosophy find useful as well.  In principle, neuroscientific research could also be brought to bear, and the psychological research relevant to our project is often guided by neuroscientific hypotheses, but in order to keep the project within manageable bounds and within the competencies of the principle investigators, neuroscience will not be one of the target areas of this project.

Psychological theories of scene parsing

In the field of psychology there is a lot of interest in developing theories of perceptual object recognition.  Much of this work is aimed at explaining how perceived objects are classified, that is, verbally labeled.  Labeling as such may not be of first importance for the present project.  What is certainly of interest, however, is the manner in which the visual system identifies the boundaries between individual objects and tracks individual objects across changes in their position and configuration. 

One approach is that of Biederman’s recognition by components theory (Biederman 1987, 1995).  On the basis of configurations of edges that are more or less invariant across changes in point of view, the system recognizes the elements of a limited alphabet of shapes called geons. Types of object (such as watering can or dog or even desk-in-office) are identified based on the configuration of the geons. 

Another approach is that of the Wageman group (in Leuven) (e.g., Kubilius et al. 2014).  Here the aim is to develop a collection of heuristics based on principles of good form that allow the system to determine which color edges in a two-dimensional scene represent the boundaries of a single object.

Work on computer vision (e.g., Szeliski 2010) may also be a useful source of ideas, although the techniques employed in this field are not always plausible as models for human vision.

Perceptual tracking and navigation

One aspect of imagistic cognition is the ability to keep track of individual objects as they move through space, behind other objects, changing their form over time.  The extent and limits of this ability have been carefully studied under the label multiple object tracking (Scholl 2001, Pylyshyn 2001).

Another aspect of imagistic cognition, closely related to object tracking, is the ability to keep track of one’s location as one travels over a terrain.  This too has been the subject of significant research in psychology (Gallistel 2008).

Motor control

Another area of research that makes a connection to the puzzle of imagistic cognition is intelligent movement of the body.  Even if our initiating decisions are episodes of conceptual thought, the fine structure of the execution cannot plausibly be determined by such decisions.  So this is another area in which we seem to have to recognize a kind of intelligence that is not well conceived as a process of propositional thought.

Take the following example: in order to pick up a cup, one needs to represent various properties of this object that are relevant for the successful execution of this action: its size (in order to approach it with the appropriate grip size), its weight (in order to lift it with the appropriate force) and its spatial location (in order to reach out in the appropriate direction). All of these properties are represented in a way that could be used in guiding one’s action (Nanay 2013). The size of the cup is not represented in centimeters or inches but rather in a way that could directly influence one’s grip-size. And the spatial location is represented in a way that could guide one’s reaching movement: in an imagistic manner in one’s egocentric space (Jeannerod 1997, 2006).


In a series of papers, Ellen Fridland (2014, 2015, 2017) has argued that the exercise of skills should be thought of as permeated with intelligence.  That is, we need to reject the cognitive model of skilled action as consisting a cognitive decision followed by an automatic execution.  Rather, skilled action exhibits intelligence throughout its exercise.  However, in order to theoretically characterize this intelligence, we need to abandon the assumption that all intelligence must be modeled on discursive reasoning.


The defining topic of this project is the nature of the knowledge that allows us to imagine successfully.  A limited amount of literature in philosophy has recently taken up questions that are closely related to, or even directly about, this question.

The role of constraints

Two philosophers who have taken up the question of how by means of mental imagery we solve problems are Amy Kind (2016, 2018) and Peter Langland-Hassan (2016).  Both of them answer that imagination can be useful because it is subject to what they call “constraints”.  Unfortunately, they do not tell us much about how these constraints operate.  If they are conceived as general knowledge, in the form of propositional thoughts, then the answer risks taking the images out of the work of imagination, by reducing the work of the imagination to the application of propositional thoughts and relegating mental imagery to the status of inessential illustration.  If there is an alternative, Kind and Langland-Hassan do not tell us what it is.  The present project aims at identifying an alternative.

Realistic versus fantastic imagination

One of us, Gauker, contends that perceptual representations of processes that take place over time can be modeled as trajectories of representations across a perceptual similarity space.  Novel imagistic representations of events that are regarded as realistic can be formed by uniformly translating such perceptual representations across the dimensions of perceptual similarity space. Additional novel but realistic events can then be imagined by linking together such translated trajectories.

Imagistic decision-making

It is common-place, at least in philosophy, to think of decision-making as a matter of an agent’s bringing his or her beliefs and desires to bear on a question.  But there does not seem to be any good theoretical account of the relation between beliefs and desires and actions (Gauker 2005), and this approach does not seem to account for experimental results (Nanay 2016).  One of us, Nanay, has argued that we might produce a better account of human decision-making by acknowledging a role for mental imagery (Nanay 2016).

Imagery in modal discourse

In philosophy, the concept of a possible world plays a role in various semantic and metaphysical projects.  However, it is notoriously difficult to draw a line between the worlds that are possible and those that are not.  Several philosophers (Kung 2010, Williamson 2016, Gregory forthcoming) have claimed that imagination, in the sense of sensory imagery, can play a role here, supplemented with semantic constraints.


A great deal of work in the developmental psychology of human children and a lesser amount of work on the psychology of monkeys and apes may also contribute to a solution to the puzzle of imagistic cognition.

The child’s concept of an object

An important research program in developmental psychology consists in studying pre-linguistic infants’ understanding of the motions of and individuation of objects.  This is studied using a preferential looking time paradigm, in which the amount of time a child looks at something is taken to measure violation of expectations.  For instance, it is known that young children up to 8 months old do not understand that a rolling ball will not spontaneously change course (Spelke et al. 1994). This work is often described as an investigation into the child’s concept of objects (Xu 1997, Carey and Xu, 2001). However, we think this description is misleading.  Much of this work can be better characterized as investigation into the child’s imagistic understanding of how things move.

Monkeys and apes

A great deal of research on the intelligence of monkeys and apes has focused on their ability to use tools to obtain pieces of food.  For instance, in the trap tube task a chimpanzee has to use a stick to push or pull a piece of food out of transparent tube without causing the food to fall into a trap at the bottom of the tube.  It turns out that great apes are not very good at this.  Even when they succeed, further investigation demonstrates that they do not understand the pertinent causal relations (Povinelli and Reaux 2000).  However, various researchers, using various experimental paradigms, have explored the boundary between the sorts of tool-using tasks that monkeys and apes can solve and those they cannot.  Recently Gauker (2017b) has argued that, with some limitations, the results of experiments by Fujita and colleagues and those by Martin-Ordas and colleagues (Fujita et al. 2003, Fujita et al. 2011, Martin-Ordas et al. 2008, Martin-Ordas et al. 2012) can be explained on the assumption that monkeys and apes attempt to solve novel problems by visually morphing their mental images of solutions to old problems into solutions to new problems.


Another area of active research in developmental psychology is the child’s acquisition and understanding of pretend play (e.g. Rakoczy 2008).  For the most part, this topic has not been approached as a topic in mental imagery (but see the philosophical discussion in van Leeuwen 2011 and Nanay 2013).  But it would be interesting to see whether the developmental facts could be explained on the assumption that children acquire a capacity to pretend in part by acquiring a capacity to form and utilize mental images.


The main result one would hope for from this project would be an empirically supported and theoretically foundational account of imagistic cognition.  This would include a general account of the conditions under which imagistic cognition succeeds and of how experience produces the ability to engage in effective imagistic cognition.

We cannot say in advance what the theory we develop will say.  However, it seems clear that a solution will rest heavily on an account of the contents of imagistic representations and of the relation between mental images and their contents.  The principles of imagistic cognition will presumably be formulated in terms of these contents and relations.

If we are right in our supposition that imagistic cognition makes up a significant share of human-problem solving ability, and that it has been overlooked by science and philosophy, then, by initiating a serious scientific study of imagistic cognition, this project promises to open up new directions in many fields – in the study of human reasoning, in the study of language acquisition, and in comparative psychology. 


Historically, many major advances in psychology have begun with philosophical thought about the nature of knowledge and mental representation of the world.  It is fair to say that the philosophers Aristotle, Descartes, Hume, Kant, William James, Wittgenstein, Dennett, and Fodor all sparked innovations in peculiarly scientific thinking about the nature of human cognition (sometimes for the better, sometimes for the worse).  As a consequence of their training in intellectual history, their training in painstaking argumentation and their having their eye on the big picture, students of philosophy are sometimes in a position to nudge the scientific community in new directions, open up new problem spaces and construct hypotheses that are subject to test and development through empirical research.  That is the kind of project we are proposing here.  In two words:  hypothesis development.

Our methods are analytic and integrative.  On the analytic side we seek clear, precise definitions.  We need a clear statement of the problem, which clearly distinguishes imagistic cognition from discursive cognition.  We need a conception of imagistic representation that distinguishes it from conceptual representation.  We need a conceptual framework in which principles of imagistic cognition can be formulated so that, for instance, we can distinguish between realistic and fantastic imagination.  On the integrative side, we need to study closely the work that has already been done in the psychology and neuroscience of perception and in computer vision to see what insights can be drawn from that work and to learn what the problems the scientific community regards as most pressing.  Moreover, the conceptual tools that we employ will often be drawn from empirical work in psychology.

In light of these aims and hopes, we are acutely aware that we must understand, and not just superficially, the latest relevant work in cognitive psychology.  In particular, we must attend to contemporary work in vision science, perceptual psychology, developmental and comparative psychology and cognitive neuroscience.  Although, we hope that our work will result in novel empirical research, and we are prepared to participate in that where the opportunity arises, we do not directly aim to conduct experiments on human subjects ourselves.

  For the references, go to the Literature section.    

Video Clip: Jerry Lewis in "The Patsy"