Say What? Robert Remez untangles how our brains decipher speech out of the jumble of sounds around us

From the Spring 2016 issue of Barnard Magazine

Say What?

For most people, the twin processes of listening and speaking are so natural that we never pause to think about how they work. But in the moment when a word or phrase hits the ear, the brain springs into action. “How do we know a sound is speech at all?” asks professor of psychology Robert Remez. “You can think about this from the perspective of a baby, lying in the nursery and hearing dogs barking, birds chirping, doorbells buzzing—and sometimes, a voice speaking. Out of that welter of sound, the baby somehow orients toward speech and doesn’t worry about the dogs and the doorbells. But how does the cognitive system recognize that speech is there?”

Since 1980, Remez has been conducting experiments on speech perception in his Barnard laboratory with six to eight undergraduates, who are his partners in what he calls “an adventure in the direction of the unknown.” Over the years, he’s relied heavily on his students to shape the course of the research, which focuses primarily on the way we perceive speech and how we follow it in quiet or acoustically busy environments.

In the lab, the students work with Remez to build “sine-wave speech,” an artificial form of speech stripped of familiar acoustic markers like tone and timbre. The result is a series of robotic-sounding beeps and whistles that convey a linguistic message despite their unnatural quality. By manipulating the samples and playing them for test subjects in experiments, Remez and his students can learn about how we track and identify speech. These findings can be applied in a wide range of fields, from voice-recognition technology to the creation of assistive devices for use in impaired hearing. The work of the lab is primarily supported by a grant from the National Institute on Deafness, part of the National Institutes of Health, which Remez was first awarded 30 years ago.

Some of Remez’s discoveries about the way we perceive speech can seem counterintuitive. Instead of listening for sounds, we are actually listening for modes of variation. It’s not unlike in music, where a melody comprises a series of intervals, rather than a specific set of notes. “That’s why we’re able to understand each other over the telephone,” he says. “If it were all about the sound, the corruptions and distortions imposed by the telephone connection would make it impossible to recognize the linguistic message. What we’re listening for are the relationships in an evolving pattern of change.”

Remez talks about his research almost poetically, as a quest for knowledge rather than a solution to a specific problem. He is a musician—before he went to graduate school to study speech, he played the bass fiddle in a band—and sees a natural bridge in his work between the humanities and sciences. Because of his appreciation for the arts, he helps students who want to continue their study of graphic arts, theatre, dance, and music by integrating those interests into their research projects. And to encourage his students to venture into the city, he organizes several yearly field trips to restaurants in different parts of Manhattan.

His collaborative approach draws students with backgrounds in many disciplines. Jessica Nowinski ’92 was planning to major in theatre when she began working with Remez as a first-year. By the time she was a senior, she had switched to psychology. Remez helped her design a senior thesis that incorporated her love of theatre—she examined whether speech performed by actors is perceived differently than spontaneous speech.

Working in Remez’s laboratory gave her the confidence to consider a career in science, says Nowinski, who is now a research psychologist at NASA: “Before I worked in Robert’s lab, I never thought of myself as a scientist. But Robert also never treated me as only a student in his lab—he wanted me to grow as a whole person. It was important for him that we continued to appreciate the things we loved outside the world of science.”

Jennifer Fellowes ’93, who worked on her senior thesis with Remez, says the professor sets the bar high from the first day. “The expectation is that everyone should be able to research and conduct an experiment that can be published in a well-regarded, peer-reviewed journal,” says Fellowes, who is now a psychiatrist in New York City. “But there’s no fear that you won’t measure up. The sense is that we’re all doing high-quality research together, and we’re having fun.”

Many of the students who work in his lab begin by meeting Remez in the classroom. He offers a lecture class each fall that introduces students to the ways we perceive the world through the five senses, as well as through our sense of balance. His course “Perception and Language” is a scientific survey of language from the listener’s perspective, everything from the physical acoustics of speech to the way we process metaphors.

To conclude the class, he gives students a challenge. “It’s fun to finish out the semester with a bit of Celtic transcendentalism,” he says. “I take a passage from a poem by Yeats, and I ask them, how would you understand someone who came up to you in a bus shelter and said, ‘Turning and turning in the widening gyre/ The falcon cannot hear the falconer’?” The point, he explains, is that poetry is a tricky kind of speech that isn’t exactly directed to the listener, so we listen differently. He creates these unusual exercises to allow the students to grapple with scholarly questions and to feel that they can make their own contributions to the scientific conversation.

To hone their professional abilities, Remez encourages his students to present their research at academic conferences and works with them on skills such as interviewing. “My best teachers taught me how to be taken seriously,” he says. “I want to give the students practice in explaining and arguing and thinking through questions, but also in how to collaborate, how to work generously on a team, when to bear down, when to wear your knowledge lightly.”

Many of the women who work in his laboratory stay in touch with each other—and with Remez—for decades. One of Remez’s aims is to create an environment where mentoring and cooperation are the norm. “My goal is to get the students to see me as a peer, someone they can speak their mind to,” Remez says. “Often they’re right and I’m wrong, but I’m happy to lose those arguments because it means we make progress together.”

Every Friday, Remez and the students in the lab gather—over food, of course—to discuss their research projects. For Samantha Caballero ’17, it’s one of the highlights of her week. “At first I thought it would be scary to give my perspective on what we’re doing, but now it’s just a fun, freewheeling conversation,” she says. “He just really wants to hear what we have to say.”

Lauren Beltrone ’17, who works in Remez’s lab, appreciates his welcoming approach: “He wants to capture your imagination and give you something to chew on intellectually. You learn a lot, but you have to work for it.”

Visit Remez's website to listen to examples of sine-wave speech and learn more about his research.