The lights dimmed and Dimitri Kanevsky took the stage at HLAA’s Convention 2022. He was there to accept the HLAA Innovation award on Google’s behalf. There was momentary confusion in the audience as the CART captioning disappeared. But it was quickly replaced by a screen share of the small phone in Dimitri’s hand. Dimitri, who has heavily accented speech, was using Google’s new Relate app to caption his acceptance speech with incredible accuracy. It was an exciting example of leading-edge technology and highlighted a new communication tool for people whose speech is difficult to understand. A perfect way to accept an Innovation Award!
Dimitri Kanevsky is the Google research scientist who conceptualized and created Google’s Live Transcribe, the handy app that turns spoken language into text so people can read what others are saying. It is a game changer for people with hearing loss, particularly during the pandemic where masks blocked much of the lipreading cues we use to decipher speech. I use it often.
I had a chance to chat with Dimitri at the conference. He shared important updates about Live Transcribe and introduced me to Relate, the app he used during his acceptance speech. Here is what I learned.
Live Transcribe Now Works Offline
I often recommend Live Transcribe as a tool to use in business meetings, but sometimes employers are cautious about its use. Until recently, Live Transcribe’s algorithm resided in the cloud, meaning that transcribed speech needed to be sent to the cloud for processing. This happened almost instantaneously, but the fact that private internal meetings were even briefly uploaded to Google’s internal server alarmed many corporate legal departments.
Good news! Live Transcribe now works without an internet or data connection. Simply toggle on “Transcribe Offline” in the app’s settings and you are good to go. Processing now occurs directly on the device itself. This not only makes decoding faster, but it may also assuage your workplace’s nervous legal department.
Or simply use it when you are out an about without incurring data charges.
There are limits however. When used online, Live Transcribe is available in more than 80 languages, but offline it is limited to English, French, Italian, Germany, Japanese, and Spanish. Also, while Live Transcribe is available on every Android phone when online, it is available on all Pixel phones, but only selected Android phones when used offline.
Relate: A New App for Heavily Accented Speech
Speech-to-text tools are terrific and getting better every day, but they often struggle with heavily accented speech. This often causes problems for Dimitri who speaks with a combined Russian and “deaf accent” that is difficult for speech-to-text programs to decode.
But it inspired Dimitri to find a solution. His goal: to provide people with difficult-to-understand speech an easier and confidence-boosting way to communicate through spoken language. The new tool is called Relate and is currently available as an Android Beta app.
Relate was originally developed as personalized speech recognition for people with dysarthric speech. According to the Mayo Clinic, dysarthria occurs when the muscles used for speech are weak or difficult to control. The condition often causes slurred or slow speech that can be difficult to understand.
Google is trying to expand Relate’s use case to people with hearing loss whose speech may be difficult for others to understand. Google is not presenting Relate as a replacement for sign language, but as an additional tool for people with heavy deaf accents who chose to use their voice to communicate.
Relate is tailored to each user individually. The algorithm uses machine learning to train on an individual’s speech patterns, eventually learning to decode the sounds into words and printed text.
Relate is currently being beta tested at Google. If you would like a personalized model for your speech, please express interest at g.co/ProjectRelate. Learn more about its features in this Google blog post.
Readers, what technology innovations do you use to communicate well?
Connect with us on Facebook and Twitter!
Unfortunately, it does not work with an iPhone which is what I use.
Maybe one day. Please let them know if that is of interest to you. Thank you for your comment.
How do I access Live Transcribe? Is it available on an IPhone?
Live Transcribe is only on Android phones. For a similar product that works on iPhone try Otter.ai. Thank you for your question.
I use transcribe all the time. Have a Google pixel and it does work offline. It’s useful in work meetings with just a few people sat around a table. Only trouble is it does not pick up some softly spoken voices maybe mic not powerful enough. A useful accessory I guess would be a plug in mic which I could sit in middle of table, not sure if this exists. Interesting article. Captioning keeps on getting better. Thank you.
There are plug in mics that can be used to enhance the signal. Thank you for mentioning it and for sharing your experience with the app.
Almost all my doctors wear masks, which is a problem because I am a lip reader. I use Live Transcribe on my Samsung Galaxy S21. I have an Artone bluetooth neckloop which serves as a microphone for Live Transcribe. I use the bluetooth feature on my phone to pair the Artone neckloop with my phone, and I make sure that the Artone is selected as the microphone in Live Transcribe’s settings. Once I am connected I can hand over my Artone to the doctor. I ask the doctor, who is sitting several feet away, to hold it near her mouth and speak into it. I can read what she’s saying on my phone. One doctor clipped it to her lab coat and talked to me while she surgically removed a precancerous mole from my chest. I wish all doctors used clear masks for lip reading patients, but the Artone-Live Transcribe combination works well. Shari, a video of this set-up should be made and made widely available. Let me know if I can help. Thanks.
That sounds like a great set up. Thanks for sharing what works for you. If you have a video please feel free to post it.