Thinking in Pictures

The Thinker by Rodin, Wikimedia

I have been thinking for a while about how humans think and, since it is the only experience I have, about how I think. This has to do with my desire to eventually create a real Artificial Intelligence (AI). This AI would not try to phenomenologically emulate human thought processes for particular tasks or statistically analyze the channels of communication between cognitive observers. Instead, it would try to build from scratch the elements of thought and the relationships between them. These elements have yet to be accurately identified and the mathematics to describe the relationships don’t exist yet, but that does not preclude us from working on lesser problems in the meantime.

First I realized that the way I think has changed over time and started thinking about how there must be diversity of thought among humans. When I started searching the Internet for information about the topic I found out that there is some psychological research, mainly in the context of multiple intelligences, but nothing that has been studied at a fundamental level and only a few studies at the functional level. Maybe I am just bad at Duckduckgo-ing. Although not scientific, and based on the experience of only one person, the best resource that I have found so far are the writings of Temple Grandin. Here is her chapter on the topic: Autism and Visual Thought.

Grandin identifies three basic types of thinking: visual, pattern and verbal. She mentions that some individuals might think in combinations of these categories. These three axes seem to be reasonable and I believe that most people do think in some range in this three-dimensional space. There might be other axes as well. Nowadays I think about half of the time visually and the other half verbally. But this has not always been the case.

I have been thinking about my thinking (meta-thinking?) for about six months now and although at the beginning it was difficult, I am pretty proficient now. I do not have the prowess of Gradin to think visually, the malleable shapes and pictures in my mind are not as detailed as hers. This malleability is useful. For example, when I am reading a paper which includes mathematical equations (integrals, derivatives, algebra) I see them as slow-motion videos in my mind (after understanding them, of course). The putty-like figures created by the mathematical relationships are usually light blue, but the voices of people might have different colors. Recently I heard a story on NPR about Kartik Sawhney, a Stanford graduate student who has been blind from birth. He invented an apparatus that converts graphs to sounds of different pitches and so he can see them in his mind. “See” might be inaccurate, he might think acoustically for all we know.

Human communication in verbal language, natural or artificial, often contains beautiful patterns such as hierarchical repetitions and recursions. Patterns such as the ones found in songs, poems or computer programs for me take physical shapes. Absurdity is the description of ideas that reach conclusions that do not make sense in the real world or are paradoxical. I see these patterns as pictures. I feel that by discussing abstract ideas with people, or playing games such as chess or connect four I get to see into their minds. I ponder about why they chose one option over another or hold an opinion over another. In the end I usually have something that can look like a classical Greek building with pink, semitransparent walls or perhaps a dim cavern with brown, capricious, but sometimes reflective surfaces.

I have not always thought primarily in images. When I was a kid and throughout my college years I thought primarily verbally and in patterns. Chess was a sequence of patterns. Algebra and calculus were patterns as well. I was really fast at arithmetic at the time because I could add and subtract from imaginary lines at powers and multiples of 10 and then put them together. Now that skill is gone and I do arithmetic very slowly. I probably started thinking primarily in pictures as a result of my graduate work in which I had to visualize atoms in crystalline structures and their motions, and translate them to graphs with different axes, or vice versa, and between real and reciprocal space. After that it took a life of its own.

I will end by putting forward the hypothesis that the user interface that we use to think shapes how we think and what can be thought of. Gradin mentions that she had a hard time understanding abstract concepts such as honesty. Although images come to my mind while having a conversation or reading about something, this made me realize that I do not have an image for abstract concepts either. No image for honesty or freedom. I do believe I understand them though, so perhaps I am using a different kind of thinking in these cases, perhaps verbal. I do have images of how some quantum mechanical processes look like or what is going on in a black hole, but these images are based on my everyday experience and must necessarily be incorrect.

How do you think?