Sitting in the dimly lit restaurant, I struggled to read the menu. Lucky for me, I keep readers in my purse, which I quickly pulled out and the problem was solved. But when the waiter came to tell us about the specials, I had trouble hearing him, even though I was wearing my hearing aids and lipreading furiously.
Wouldn’t it be wonderful if I could solve this problem just as easily — reaching into my purse for my captioning glasses — ones that would turn speech into text in real-time and with exceptional accuracy? Interestingly, a graduate student team at Cornell Tech is working on something just like this.
The product is in its early stages, but it is exciting to see hearing loss taking center stage in this innovative and important technology project. Part of the reason is due to the project’s leader, Christopher Caulfield, a second-year graduate student at Cornell Tech who is deaf.
Diagnosed with profound hearing loss at thirteen months, Christopher received a cochlear implant at 18 months and has attended mainstream school ever since. His academic and professional accomplishments are numerous and inspiring and he has decided to focus his career on technology, with an emphasis on how technology can be used to enhance accessibility. I imagine this is only the first of many projects he will lead throughout his career.
Captioned Glasses Prototype In Development
The student team describes the product as “an application that could improve deaf and hard of hearing people’s experiences in one-on-one conversations. The app will use augmented reality and automatic speech recognition to display captions of what your conversation partner is saying.” The captions would be seen by wearing special glasses that would impose the captions near the speaker’s face so you could maintain eye contact like in a typical conversation while benefiting from the captions. Wow!
The team recently conducted interviews with potential end users for feedback on their current prototype, specifically the look and feel of the captions themselves. I attended one a few weeks ago where I was asked to evaluate the placement of the captions relative to the person’s face, the color and sizing of the captions and a number of special effects like shading or bolding that were used to demonstrate emphasis or high emotion.
In the exercise, the captions were displayed on a screen while a video played, so it was different from what I imagine the end product could look like and there was no issue with the caption accuracy since they were pre-programmed. I enjoyed having the captions since the video was purposefully set on a very low volume. I don’t think I would have caught much of the content without them.
I use captions frequently when watching TV, attending the movies, and whenever else they are available, but these captions were different. The size of the words varied to indicate the emphasis the speaker placed on each word. Sometimes the words were also shaded red or green to indicate the mood or affect of the speaker. There was a lot more information contained in the captions than is typical. It was an interesting concept, but took some practice to fully appreciate.
The work is ongoing as much still needs to be determined in terms of the end product configuration and the speech to text technology itself. I applaud the team for involving the potential end users in the early stages of the design and throughout the process. This will go a long way towards making the final product a success.
Readers, would you use captioning glasses?
Wow, this is a wonderful project! With all the progress of technology and Artificial Intelligence, I think deaf and HoH persons will see great improvement in our quality of life. I also look forward to the time when all movie theatres will offer this technology.
The advancements are amazing! Thanks for sharing your thoughts.
Shari,
Waiters announcing specials rather than providing written copy is one of my pet peeves. It always feels like pretentious performance art, not intended as actual communication. In noisy restaurants, even those without hearing loss are challenged by this practice.
This is a fabulous idea! I am sure there are many problems to solve in addition to the obvious one of recognizing different voices. In a noisy environment, I wonder how the captioning glasses would screen out speech other than from the speaker I want to hear. Still, I join you in applauding the effort and in admiring Christopher Caulfield. I would love to participate in evaluating products like this. How does someone get on the list?
Jon
There are many details still to be clarified and I’m sure Christopher will be looking for more feedback as the project progresses. The best way to reach him is through his website http://www.caulfield.io. Thanks for sharing your thoughts.
Wow. I most certainly would use captioned glasses. But they would need to be an awfully lot better than the ones provided at my local movie theater…which don’t work half the time. It’s great to see this work being done.
These would be for normal conversations, not just at the movies. It is exciting to see these creative ideas in process. Thanks for sharing your thoughts.
This would be awesome to have. I use captioned glasses at Regal Theatres which are awesome and have always felt they would be good to have at Restaurants (to hear conversations in groups or TV’s while watching regular season Bears games (when the place have multiple games going) . I would love to help with this.
It is an exciting idea! Thanks for sharing your thoughts.
People think I am joking when I say I would love to have captioned radio, but this might be a way to accomplish it. Even without the glasses, I don’t see any reason radio can’t be captioned and sent to a computer or phone, or even your tv set.
I would wear these around the house,not just to restaurants, or public places.
I miss out on the key points as my hearing is getting worse. Be less aggravation and humiliation as I talk to family.
It would be nice! So glad technology students are taking an interest. Thank you for sharing your thoughts.
This sounds wonderful. Tell Christoper we are waiting for him to help all of us and give him a big thank you!
Rosemary
BTW I love InnoCaption. It has made a big change in my life!
Thank you for sharing your thoughts.
This would be awesome. I wear glasses, and my hearing aid is the type that goes behind the ear. Sometimes wearing both is a bit much. I’d love to wear just the glasses. This would also be great if it could be used at slightly greater distances. I have jury duty coming up, and I’m not at all comfortable with the fact that I could easily miss a lot of what is said. A note from my audiologist and a copy of my hearing test wasn’t enough for me to be excused. Something like this could make a huge difference in my confidence in these types of situations.
It would be great to have such a technology. While it is likely several years away, I am happy the team is working on it. Thanks for sharing your ideas.
wow, I’m amazed Juie! a copy of just my audiogram was enough to excuse me from jury duty. It wouldn’t be fair to the defendent either that you wouldn’t be able to hear everything.
Since there will be a court reporter you should ASK for CART. This would provide the court reporters work to transcribe onto a tablet or computer for you.
Make people aware that you cannot do this unless you have CART. This will hopefully allow them to consider a different decision. We need to be advocates for ourselves while also making people aware of accessible needs. Look up the ADA info to help your argument. Good luck!
Wanda
Thanks Wanda! I figured I’d have more luck in person. I’ll ask about CART
I don’t think CART is available yet in the courts. A group of us from HLAA went down to evaluate a system in one of the courthouses, and to give advice. The system was actually very good, and they really listened to our comments. However, I don’t think they are actually in use yet.
I have been excused since 1992 based on a letter from my audiologist. At that time, my hearing loss was rather mild.
When I asked for CART for jury duty years ago , they excused me and I never got another jury summons since . I guess they don’t want to deal with the cost of paying for CART plus maybe causes security issues having another CART reporter documenting a case.
I called this morning and CART is not available at my location. The clerk told me that they prefer to make the determination in person as to whether or not my hearing loss will be a problem. Thanks to everyone for all the info. This community is so helpful!
CART is so expensive and difficult to find someone who’s willing to do it in your area , but I honestly believe the state just doesn’t want to spring for the cost of CART reporting for a potential juror , and that is actually discriminatory, but what can you do but complain or find your own CART reporter and see if the court will pick up the cost . That’s if you really want to serve as a juror. Otherwise I wouldn’t bother suffering through lipreading and looking clueless when asked a question.
Ask the court clerk to provide you with CART which is live captioning for your jury service.
Thanks Mary! I’ll check it out.
Bless Christopher by using his intelligence and understanding of hearing loss to enhance the lives of others. As a former audiologist with hearing loss, I’ve witnessed the gradual changes, both technically AND socially by improved public advertisements and awareness showing not only the elderly, but younger men and women with hearing loss as well. Such helps decrease the stigma of hearing loss.
Thank you, Shari
Thank you for the work you do for people with hearing loss.
Amazing idea , however I think I’d prefer just having an amazing app on my phone that translates voice into text , one that really works and isn’t all garbled . From my experiences with wearing caption glasses at the movies , they were heavy , hurt your nose , were gawky looking , and very annoying if you had to fit them over glasses you already wore . On the topic of the movies , I just saw a movie at AMC theater in NYC Saturday night and had the worst experience with a captiview device that just would not fit into the cup holder . Was way too snug , just would not press into that cup holder and I ended up holding the darn thing entire movie. Talk about muscle aches .Plus the stares from people (not that I really care )around me including the person next to me who was not very happy I stole her cup holder , and others wondering what the heck is that black thingy she holding!?!! (That’s NYC for ya!) Werent they supposed be putting built in seat caption devices for Broadway Shows and opera in NYC? Would work wonders in movie theaters as well .
If only these movie chains would spring for the cost.
Thank you for sharing your experiences. A good speech to text app would be a great thing too!
The AMC cup holders are a real problem. I always struggle with them. That said, the captioning is actually very good. The Landmark Theater on W. 57th has much better holders.
I hope people are making the theaters aware of this.
I have told them many times.
Thanks Wanda for mention about that theater . Will check it out .
I plan to contact AMC theaters about their too small cup holders .
Hmmmmm, doubt they will fix the issue but at least they will be aware .
Actually thanks Jon for the theater mention of the better cup holders . Will check it out.
Gina, I feel the same way about the captioning glasses. By the end of the movie my ears are sore.
Also, I’ve seen what you call the Captiview, haven’t tried it, and do not like the idea of it. Why should one have to hold something during the movie? I want captioning glasses that are lighter weight and comfortable.
And totally agree on having an app that that will put it into text options. I’ve been told there are some out there but not Android friendly.
It seems with amazing technology being created on a daily basis that more people would be concentrating on technology for the deaf and HOH; it would be a profitable item for its’ creator! When I think about the language translator, I wonder why this cannoth be done for “us”.
I have been saying for years I wish face to face conversations could be captioned what a fantastic concept. I’ve been impaired my entire life, speech recognition continues to decline, making communication increasingly difficult, a device like this would be life changing.
I agree. Thanks for sharing your thoughts.
This is a dream. Yesterday I attended a Planning Board meeting which I requested and recieved assistance. They provided a court reporter. It was a very helpful as I have been in that meeting room in the past and have never been able to understand the spoken words. The lady worked very hard and made all the difference in my being able to participate. There were some names that didn’t come through as she would have had to enter them in her list before starting. In the past request would be less than adaquate to allow me to fully participate. One incident they would provide a transcript, which would be available after the meeting and not official until it was approved by the board. By then the decision was made so any input I would have could not be considered. Some meetings are more of open format where displays and officials are there to explain various options. Sometimes this works, other times I cannot understand the person speaking. This is where a device which provides instant captions would be wonderful.
I should ask the agency how much they have to pay for court reporter service. The instant caption dream would make government service more available to those who need it.
Thank you for sharing your experiences with captions. If developed fully, these glasses would help so many people.
The practical and affordable realization of these glasses seems to me to be a ways off, but we do live in the age of miracles and wonders. This is an instant electronic message 🙂
Yes, still in early stages of development, but wouldn’t it be great! Thanks for sharing your thoughts.
Reblogged this on Talking Hearing Loss and commented:
This reminds of my first hearing aid which was ‘disguised’ as a pair of glasses!
I must say that I think that glasses able to provide live captions is a brilliant idea. My nerd brain immediately thinks of all the challenges of designing a usable product (latency, accommodating for accents,etc) but I wish the developers all the best because this would be a fantastic product!
There are definitely challenges but I’ve got my fingers crossed they can work it out over time! Thanks for sharing your thoughts.
We need someone to develope Caption (light) glasses. They would show text only back or white from Google’s 2020 Live Captions on Pixel phones where the caption function is in the phone without internet. Ordinary looking glasses, simple text and no camera or other extras. This would be a wonderful advancement for the extreme hard of hearing past hearing aids.
That would be wonderful! I know people are working on this, but I’m not sure how soon it will be viable. Thank you for your comment.
I would use these glasses in a heartbeat, though I do still have the same question I asked when you blogged about this two years ago, which is how will the glasses screen out other speech in the room? When you went to the demo, did they address that? Still, I am thrilled that this project is getting closer to reality. I haven’t met Chris, but viewed a wonderful video about him a couple of years ago. He is an exceptional young man, and I wish him well.
No answers on that issue yet, but keep asking. It is an important consideration. Thanks for your comment.