Through A Portal to the Mind
One morning at Harvard Square in Cambridge, I opened a book that had a micrograph of the hair cells inside the cochlea, down deep in the inner ear, and I was stunned with the natural forms and shapes that laid out so beautifully along the curves. It was one of several images I would later collect and study. I wondered what it would be like to float down inside the pillars of Corti, rising up like arches in an ancient cathedral beside me, and feel them go into motion. Standing on the basilar membrane, I imagined the elastic floor lifting me as if I was on a trampoline, bouncing me upwards as the fluids rolled under me, surrounded by thousands of these majestic pillars lined up as far as I could see, in either direction. In my imagination, I added the rhythmic fireworks at the cilia above me. The surreal landscape was a light show of the first order.
I wondered if we could animate this phenomenon, and how valuable it would be as a teaching and learning platform, where, as both science and art, multiple learning areas in the brain intersect. After all, hearing is one of the few areas that involve motion that we can easily understand through our everyday sense of pushing and pulling, stretching and twisting, lifting and hauling—an intuitive entryway into teaching and learning. Even rarer, we can apprehend hearing with this intuitive access through all levels: The hearing’s rhythmic motions at the microscopic level correlate easily to the macroscopic level of our perception, our rhythmic sense of music and cadence. In other words, the meaning of how it works extrapolates easily to the meaning we make of what it is.
I thought, too, that using hearing—a literal portal to the mind—could open to a whole set of ideas related to teaching and learning, like physics and biology, using the film as a portal to the mind.
Further, I wondered whether we could use actual computational neuroscience data to drive Hollywood-quality animation and special effects. Students wouldn’t just get graphs and numerical data to learn the results of computational data and the researchers wouldn’t just get plots and charts: Both would see the extraordinary beauty and elegance of our hearing system from the very start of their adventures in learning.
To explore the idea, I recruited Ben Clopton, one of the major players in inventing the first cochlea implant, and David Mountain, a bioengineer at Boston University who headed a cadre of hearing researchers in the EarLab Project. Together, we storyboarded the entire process, a surprisingly difficult task because no one had ever done it before. We found that we understood hearing in disconnected fragments of mathematical expressions and abstractions from physics, understandings that had never been scrutinized under the stringent, concrete demands of being drawn and animated, one frame to the next. We couldn’t stop at mere understanding: we had to push through the details until we could see where one part started and another began, and draw exactly how it moved.
I expanded my animation film studio, hiring programmers, medical illustrators, artists, modelers, and animators. Ben and David supplied numerical data from computational research. And we set to work.
From the director’s chair, I wanted to explore several qualities. First, I wanted a look that didn’t resemble 3D animation graphics but something other worldly, something that had a touch of realism and a touch of surrealism but at all times accurate in terms of a visual representation.
I was also keenly interested in exploring what would emerge if we made a film of several layers, each designed to communicate with the other layers—not too unlike the concept of Gesamtkunstwerk that interested the composer Wagner. Wagner composed his operas so that, independently and together, the singers, orchestra, sets, and the words each tell the story. For instance, in the opera Tristan and Isolde, Wagner composed themes in the orchestral layer for Isolde’s longing for Tristan and another theme for her angst when she’s afraid he’s gone and another for her anticipation of his return, bringing out these themes as the story unfolds. Following this pattern for each character, across a wide range of emotions and ideas, he has the orchestral layer tell its own version of the story, independently of the words.
I wanted to compose a score that was rich in telling the story. I wanted to have camera motions that told the story. I wanted to have color that told the story. I wanted the animation itself to tell the story. I wondered how many layers we could coordinate, and which ways. We keyed up the motion down to the frame with many, many layers interacting.
I also wondered whether the marriage of science and art and the multiple layers communicating to each other would make it easier to learn and absorb and appreciate our hearing system. I was interested in the question of whether or not beauty—the experience of beauty—enhances learning.
In the score, I wanted to hear a wide range of sounds because, after all, the film was about hearing. I wanted to hear the structure clearly and simply, because in hearing the structure is simple and clear. I wanted to hear markers that tipped off the viewer about changes in the story. Most importantly, I wanted the music to move with the animation precisely which meant, because our nervous system skews incoming signals, I had to adjust the music so that it could slide either slightly before or slightly after each visual event, depending on the way our minds perceive visual and aural events, which is not always in a one-to-one relationship.
I also wanted the music to invite a sense of wonder, to open up the perceptual portals so that learning was easy. I wanted to tell a story with evocative, yet clear words that resembles poetry. With the colors, I wanted a sequence of patterns that play along with the music and with the words to create memorable images. Structurally, I wanted all to create a sense of anticipation from one scene to the next, to propel the viewer through the material.
Finally, I wanted to draw connections to the astonishing patterns. For instance, I wanted to compose highlights in the music that correspond to highlights in the motion; I wanted to use a wide range of timbres in the score, with rhythmic, yet varying patterns, mirroring the patterns of motion and forms; I wanted to repeat, with elaboration, key analogies and metaphors in the script. I wanted to resonate common patterns, but among different senses and forms, tapping multiple parts of our nervous system.
What I didn’t want was a typical film full of explanatory words with explanatory illustrations thrown together. Such a film requires the viewer to bring prior knowledge to understand what they were seeing, as the words might be disconnected from the images or the meanings might slide between the words and images without any real thought of connecting them. Instead, I was keenly interested in coordinating each word with the image, with the learning, with the music, with the motion, so that the whole would emerge with qualities that aren’t necessarily in the individual parts.
We wrote software conduits from the research data to Pixar’s animation software, which we ran on top-of-the-line SGI workstations and rendered on a state-of-the-art render farm, which we routinely overwhelmed. For instance, David and Ben gave us a million-point dataset for the fluids in the cochlea, data from simulations that would typically only yield curves and percentages for their research needs, but which we had to filter down to motions we could see. Some frames took days to render; some took weeks: at thirty frames a second, we painstakingly produced the film.
To complete the platform, I recruited Michael Geisen, the National Teacher of the Year, who, lit up, eagerly joined the team and brought his magical teaching skills to build upon the film. He further recruited Seth Berg, Colorado Teacher of the Year, and Bob Williams, Alaska Teacher of the Year. Combined, their fertile imaginations and deep teaching experience wrote materials for teachers of biology and physics. Using the film as the centerpiece, they could elaborate on a wide range of off-shoot issues, engaging students at multiple levels.
In the end, we completed the Through a Portal to the Mind platform for middle and high schools and we are in the process of building the platform at the university level, where we would extend the ideas from biology and physics to include math, physiology, philosophy, and neuroscience.
