The lights dimmed and Dimitri Kanevsky took the stage at HLAA’s Convention 2022. He was there to accept the HLAA Innovation award on Google’s behalf. There was momentary confusion in the audience as the CART captioning disappeared. But it was quickly replaced by a screen share of the small phone in Dimitri’s hand. Dimitri, who has heavily accented speech, was using Google’s new Relate app to caption his acceptance speech with incredible accuracy. It was an exciting example of leading-edge technology and highlighted a new communication tool for people whose speech is difficult to understand. A perfect way to accept an Innovation Award!
Dimitri Kanevsky is the Google research scientist who conceptualized and created Google’s Live Transcribe, the handy app that turns spoken language into text so people can read what others are saying. It is a game changer for people with hearing loss, particularly during the pandemic where masks blocked much of the lipreading cues we use to decipher speech. I use it often.
I had a chance to chat with Dimitri at the conference. He shared important updates about Live Transcribe and introduced me to Relate, the app he used during his acceptance speech. Here is what I learned.
Live Transcribe Now Works Offline
I often recommend Live Transcribe as a tool to use in business meetings, but sometimes employers are cautious about its use. Until recently, Live Transcribe’s algorithm resided in the cloud, meaning that transcribed speech needed to be sent to the cloud for processing. This happened almost instantaneously, but the fact that private internal meetings were even briefly uploaded to Google’s internal server alarmed many corporate legal departments.
Good news! Live Transcribe now works without an internet or data connection. Simply toggle on “Transcribe Offline” in the app’s settings and you are good to go. Processing now occurs directly on the device itself. This not only makes decoding faster, but it may also assuage your workplace’s nervous legal department.
Or simply use it when you are out an about without incurring data charges.
There are limits however. When used online, Live Transcribe is available in more than 80 languages, but offline it is limited to English, French, Italian, Germany, Japanese, and Spanish. Also, while Live Transcribe is available on every Android phone when online, it is available on all Pixel phones, but only selected Android phones when used offline.
Relate: A New App for Heavily Accented Speech
Speech-to-text tools are terrific and getting better every day, but they often struggle with heavily accented speech. This often causes problems for Dimitri who speaks with a combined Russian and “deaf accent” that is difficult for speech-to-text programs to decode.
But it inspired Dimitri to find a solution. His goal: to provide people with difficult-to-understand speech an easier and confidence-boosting way to communicate through spoken language. The new tool is called Relate and is currently available as an Android Beta app.
Relate was originally developed as personalized speech recognition for people with dysarthric speech. According to the Mayo Clinic, dysarthria occurs when the muscles used for speech are weak or difficult to control. The condition often causes slurred or slow speech that can be difficult to understand.
Google is trying to expand Relate’s use case to people with hearing loss whose speech may be difficult for others to understand. Google is not presenting Relate as a replacement for sign language, but as an additional tool for people with heavy deaf accents who chose to use their voice to communicate.
Relate is tailored to each user individually. The algorithm uses machine learning to train on an individual’s speech patterns, eventually learning to decode the sounds into words and printed text.
Relate is currently being beta tested at Google. If you would like a personalized model for your speech, please express interest at g.co/ProjectRelate. Learn more about its features in this Google blog post.
Readers, what technology innovations do you use to communicate well?