Captions are an essential communication tool. They help us understand what words people are saying, environmental sounds like a door slamming, or if there is music playing. But captions could do more to express the emotional context behind what is being said.
Launched on Android phones in December, Google’s Expressive Captions aims to do just that.
What are Expressive Captions?
Expressive Captions uses AI to communicate not only the spoken word but also the tone and volume used. For example, using All CAPS to show excitement or yelling, like when a sports commentator announces a touchdown or your friend exclaims, “Happy Birthday!”
Environmental sounds like applause and cheers and vocal bursts like sighs or gasping are also represented, so you’ll not only know when your team scores but how the crowd reacts, too.
These small details help express the emotional intent behind the words, especially for live events and social content that doesn’t have pre-loaded or high-quality captions.
Pros and Cons of Expressive Captions
I’m excited about the possibilities of Expressive Captions! People with hearing loss sometimes misread emotions in spoken language due to listening fatigue or other factors, so spelling out the intent behind the words in black and white can be helpful. Written emotive cues may also help us to tap into the communal energy of a shared viewing experience, which could help us feel more confident cheering loudly for our favorite team because we know others are as well.
But not at the expense of transcription accuracy or legibility. It is important to remember that the primary purpose of captioning is for communication access. Adding emotional cues is great, as long as they are not distracting or impeding their use as a comprehension tool.
What’s Next?
It will be interesting to see how the technology evolves. Will programs with pre-populated captions (i.e., movies or scripted television shows) start to include more emotive cues? Can users toggle them on and off for different use cases? Will Apple get into the game?
Expressive Captions are currently available in the U.S. in English on Android devices running Android 14 and above that have Live Caption. They can be used for livestreams and video messages, and since the captions occur on the device, no Internet connection is needed.
Thank you, Google for continuing to push the accessibility envelope forward!
Readers, would you use Expressive Captions?
Related Articles
Connect with us on Facebook and Twitter!
Interesting article. It prompts another thought regarding captions and current technology advancements.
AI generated captions are now much more accurate and significantly faster with the ability to match in realtime a speakers voice. Is it time to change FCC standards around caption quality which now favor a live person typing in the belief that itâs more accurate which is not always the case? Certainly there are some situations (ie some medical, not all) where live captioner may still be needed.
We have lived with Zoom and other AI provided captions now for 4+ years… Time to look at FCC standards again?
Kerry Sullivan, HLAA Boston
>
Great points! Thank you for raising this.
I hope they succeed and lead to more innovations in written communication. Especially texting. I can be a bit sarcastic…which can cause problems when texting with people who don’t know me well!
Tone in texting can be a challenge for everyone, but yes, it would be helpful there too! Thank you for sharing your thoughts.