Christopher Caulfield was born profoundly deaf. Devon Bain’s mother has hearing loss.

Credit: Provided

As students in the Technion-Cornell Dual Master’s Degrees in Connective Media program at Cornell Tech, Caulfield and Bain were both motivated by personal experience to find a solution to the problem of hearing loss, which affects millions of people worldwide.

Their motivation has turned into creation: Caulfield and Bain developed a prototype for augmented reality headsets to help people who are deaf or hard of hearing navigate one-on-one conversations. And their research has inspired fellow Cornell Tech students to seek solutions to accessibility problems.

“We were surprised by how each of them [was] interested in accessibility because they were related to someone with a disability or good friends with someone with a disability,” said Caulfield. And Bain said that even people without a personal connection to someone with a disability cared, because they are focused on social good.

To develop their prototype, Caulfield and Bain interviewed seven people with hearing difficulty to learn about the tools they currently use, the challenges they face and the technology that would be most helpful. They are among approximately 466 million people worldwide – more than 5 percent of the world’s population – with “disabling hearing loss,” according to the World Health Organization.

The participants, ranging from students to senior citizens, reported difficulties participating in conversations in noisy locations like restaurants and subways. This made them frustrated, angry and less motivated to socialize – even though they wanted to interact with others. They also felt self-conscious; one woman said she feared her employer would realize she was hard of hearing.

Most of the people they interviewed wanted a wearable augmented reality captioning system that could be used for lengthier conversations and ones in loud places. That inspired Caulfield and Bain to show captions in augmented reality glasses, because most of the existing technology shows captions on computers or handheld devices.

“Our research focuses on how this is a barrier in one-on-one conversations,” said Caulfield. “We want to make conversations more seamless.”

Bain and Caulfield created the software as part of their specialization project, a requirement for students in Jacobs Technion-Cornell Institute master’s programs. They met with company advisers over two semesters to create a presentation and demo, which they then pitched to stakeholders at Verizon, their mentoring company. 

They are working with a Verizon executive who is interested in augmented reality. “Hopefully that will lead to a fully functional commercial project that people will use,” Bain said.

After receiving feedback on different iterations, Caulfield and Bain built a prototype that uses augmented reality headsets. They used ARKit, the iOS platform for augmented reality, and a text box for captions. Users insert their smartphones into a MERGE augmented reality headset. Currently, they’re using scripted dialogue to demonstrate the overall experience; future plans include integrating automated speech recognition.

They are using computer vision and face recognition to locate a person’s face and to position the captions directly under the speaker’s chin. The text size and color change based on the speaker’s tone and emphasis.

“Connective media is really unique because it doesn’t just teach us about software development, it also teaches about user research and design,” Bain said.

The pair is working with Shiri Azenkot, an assistant professor at the Jacobs Technion-Cornell Institute at Cornell Tech whose research focuses on accessible technology. Azenkot specializes in technology for people who are blind or have low vision, as she does. The fact that she is not deaf or hard of hearing has given them an additional perspective, Caulfield said.

A group of Cornell Tech students now meets regularly to discuss accessibility projects. Caulfield and Bain believe one of the ideas will be refined in the Startup Studio. Said Bain: “Our project has really inspired a lot of people to think about accessibility at Cornell Tech.”

Elana Gross is a freelance writer for Cornell Tech.