What If Your Reading Glasses Also Provided Captions?

Advertisements

Sitting in the dimly lit restaurant, I struggled to read the menu. Lucky for me, I keep readers in my purse, which I quickly pulled out and the problem was solved. But when the waiter came to tell us about the specials, I had trouble hearing him, even though I was wearing my hearing aids and lipreading furiously.

Wouldn’t it be wonderful if I could solve this problem just as easily — reaching into my purse for my captioning glasses — ones that would turn speech into text in real-time and with exceptional accuracy? Interestingly, a graduate student team at Cornell Tech is working on something just like this.

The product is in its early stages, but it is exciting to see hearing loss taking center stage in this innovative and important technology project. Part of the reason is due to the project’s leader, Christopher Caulfield, a second-year graduate student at Cornell Tech who is deaf.

Diagnosed with profound hearing loss at thirteen months, Christopher received a cochlear implant at 18 months and has attended mainstream school ever since. His academic and professional accomplishments are numerous and inspiring and he has decided to focus his career on technology, with an emphasis on how technology can be used to enhance accessibility. I imagine this is only the first of many projects he will lead throughout his career.

Captioned Glasses Prototype In Development

The student team describes the product as “an application that could improve deaf and hard of hearing people’s experiences in one-on-one conversations. The app will use augmented reality and automatic speech recognition to display captions of what your conversation partner is saying.” The captions would be seen by wearing special glasses that would impose the captions near the speaker’s face so you could maintain eye contact like in a typical conversation while benefiting from the captions. Wow!

The team recently conducted interviews with potential end users for feedback on their current prototype, specifically the look and feel of the captions themselves. I attended one a few weeks ago where I was asked to evaluate the placement of the captions relative to the person’s face, the color and sizing of the captions and a number of special effects like shading or bolding that were used to demonstrate emphasis or high emotion.

In the exercise, the captions were displayed on a screen while a video played, so it was different from what I imagine the end product could look like and there was no issue with the caption accuracy since they were pre-programmed. I enjoyed having the captions since the video was purposefully set on a very low volume. I don’t think I would have caught much of the content without them.

I use captions frequently when watching TV, attending the movies, and whenever else they are available, but these captions were different. The size of the words varied to indicate the emphasis the speaker placed on each word. Sometimes the words were also shaded red or green to indicate the mood or affect of the speaker. There was a lot more information contained in the captions than is typical. It was an interesting concept, but took some practice to fully appreciate.

The work is ongoing as much still needs to be determined in terms of the end product configuration and the speech to text technology itself. I applaud the team for involving the potential end users in the early stages of the design and throughout the process. This will go a long way towards making the final product a success.

Readers, would you use captioning glasses?

Connect with us on Facebook and Twitter

Never miss a post! Sign up below for email alerts. 

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

42 thoughts on “What If Your Reading Glasses Also Provided Captions?”

  1. Wow, this is a wonderful project! With all the progress of technology and Artificial Intelligence, I think deaf and HoH persons will see great improvement in our quality of life. I also look forward to the time when all movie theatres will offer this technology.

    1. Shari Eberts – NYC – Shari Eberts is a hearing health advocate, writer, speaker and avid Bikram yogi. She blogs at LivingWithHearingLoss.com and serves on the Board of Trustees of Hearing Loss Association of America. She is the former Board Chair of Hearing Health Foundation. Shari has an adult-onset genetic hearing loss and hopes that by sharing her story, it will help others to live more peacefully with their own hearing issues.
      Shari Eberts says:

      The advancements are amazing! Thanks for sharing your thoughts.

  2. Shari,

    Waiters announcing specials rather than providing written copy is one of my pet peeves. It always feels like pretentious performance art, not intended as actual communication. In noisy restaurants, even those without hearing loss are challenged by this practice.

    This is a fabulous idea! I am sure there are many problems to solve in addition to the obvious one of recognizing different voices. In a noisy environment, I wonder how the captioning glasses would screen out speech other than from the speaker I want to hear. Still, I join you in applauding the effort and in admiring Christopher Caulfield. I would love to participate in evaluating products like this. How does someone get on the list?

    Jon

    1. Shari Eberts – NYC – Shari Eberts is a hearing health advocate, writer, speaker and avid Bikram yogi. She blogs at LivingWithHearingLoss.com and serves on the Board of Trustees of Hearing Loss Association of America. She is the former Board Chair of Hearing Health Foundation. Shari has an adult-onset genetic hearing loss and hopes that by sharing her story, it will help others to live more peacefully with their own hearing issues.
      Shari Eberts says:

      There are many details still to be clarified and I’m sure Christopher will be looking for more feedback as the project progresses. The best way to reach him is through his website http://www.caulfield.io. Thanks for sharing your thoughts.

  3. Wow. I most certainly would use captioned glasses. But they would need to be an awfully lot better than the ones provided at my local movie theater…which don’t work half the time. It’s great to see this work being done.

    1. Shari Eberts – NYC – Shari Eberts is a hearing health advocate, writer, speaker and avid Bikram yogi. She blogs at LivingWithHearingLoss.com and serves on the Board of Trustees of Hearing Loss Association of America. She is the former Board Chair of Hearing Health Foundation. Shari has an adult-onset genetic hearing loss and hopes that by sharing her story, it will help others to live more peacefully with their own hearing issues.
      Shari Eberts says:

      These would be for normal conversations, not just at the movies. It is exciting to see these creative ideas in process. Thanks for sharing your thoughts.

  4. This would be awesome to have. I use captioned glasses at Regal Theatres which are awesome and have always felt they would be good to have at Restaurants (to hear conversations in groups or TV’s while watching regular season Bears games (when the place have multiple games going) . I would love to help with this.

    1. Shari Eberts – NYC – Shari Eberts is a hearing health advocate, writer, speaker and avid Bikram yogi. She blogs at LivingWithHearingLoss.com and serves on the Board of Trustees of Hearing Loss Association of America. She is the former Board Chair of Hearing Health Foundation. Shari has an adult-onset genetic hearing loss and hopes that by sharing her story, it will help others to live more peacefully with their own hearing issues.
      Shari Eberts says:

      It is an exciting idea! Thanks for sharing your thoughts.

    2. People think I am joking when I say I would love to have captioned radio, but this might be a way to accomplish it. Even without the glasses, I don’t see any reason radio can’t be captioned and sent to a computer or phone, or even your tv set.

  5. I would wear these around the house,not just to restaurants, or public places.
    I miss out on the key points as my hearing is getting worse. Be less aggravation and humiliation as I talk to family.

    1. Shari Eberts – NYC – Shari Eberts is a hearing health advocate, writer, speaker and avid Bikram yogi. She blogs at LivingWithHearingLoss.com and serves on the Board of Trustees of Hearing Loss Association of America. She is the former Board Chair of Hearing Health Foundation. Shari has an adult-onset genetic hearing loss and hopes that by sharing her story, it will help others to live more peacefully with their own hearing issues.
      Shari Eberts says:

      It would be nice! So glad technology students are taking an interest. Thank you for sharing your thoughts.

    1. Shari Eberts – NYC – Shari Eberts is a hearing health advocate, writer, speaker and avid Bikram yogi. She blogs at LivingWithHearingLoss.com and serves on the Board of Trustees of Hearing Loss Association of America. She is the former Board Chair of Hearing Health Foundation. Shari has an adult-onset genetic hearing loss and hopes that by sharing her story, it will help others to live more peacefully with their own hearing issues.
      Shari Eberts says:

      Thank you for sharing your thoughts.

  6. This would be awesome. I wear glasses, and my hearing aid is the type that goes behind the ear. Sometimes wearing both is a bit much. I’d love to wear just the glasses. This would also be great if it could be used at slightly greater distances. I have jury duty coming up, and I’m not at all comfortable with the fact that I could easily miss a lot of what is said. A note from my audiologist and a copy of my hearing test wasn’t enough for me to be excused. Something like this could make a huge difference in my confidence in these types of situations.

    1. Shari Eberts – NYC – Shari Eberts is a hearing health advocate, writer, speaker and avid Bikram yogi. She blogs at LivingWithHearingLoss.com and serves on the Board of Trustees of Hearing Loss Association of America. She is the former Board Chair of Hearing Health Foundation. Shari has an adult-onset genetic hearing loss and hopes that by sharing her story, it will help others to live more peacefully with their own hearing issues.
      Shari Eberts says:

      It would be great to have such a technology. While it is likely several years away, I am happy the team is working on it. Thanks for sharing your ideas.

    2. wow, I’m amazed Juie! a copy of just my audiogram was enough to excuse me from jury duty. It wouldn’t be fair to the defendent either that you wouldn’t be able to hear everything.
      Since there will be a court reporter you should ASK for CART. This would provide the court reporters work to transcribe onto a tablet or computer for you.
      Make people aware that you cannot do this unless you have CART. This will hopefully allow them to consider a different decision. We need to be advocates for ourselves while also making people aware of accessible needs. Look up the ADA info to help your argument. Good luck!
      Wanda

      1. I don’t think CART is available yet in the courts. A group of us from HLAA went down to evaluate a system in one of the courthouses, and to give advice. The system was actually very good, and they really listened to our comments. However, I don’t think they are actually in use yet.

        I have been excused since 1992 based on a letter from my audiologist. At that time, my hearing loss was rather mild.

      2. When I asked for CART for jury duty years ago , they excused me and I never got another jury summons since . I guess they don’t want to deal with the cost of paying for CART plus maybe causes security issues having another CART reporter documenting a case.

      3. I called this morning and CART is not available at my location. The clerk told me that they prefer to make the determination in person as to whether or not my hearing loss will be a problem. Thanks to everyone for all the info. This community is so helpful!

      4. CART is so expensive and difficult to find someone who’s willing to do it in your area , but I honestly believe the state just doesn’t want to spring for the cost of CART reporting for a potential juror , and that is actually discriminatory, but what can you do but complain or find your own CART reporter and see if the court will pick up the cost . That’s if you really want to serve as a juror. Otherwise I wouldn’t bother suffering through lipreading and looking clueless when asked a question.

  7. Bless Christopher by using his intelligence and understanding of hearing loss to enhance the lives of others. As a former audiologist with hearing loss, I’ve witnessed the gradual changes, both technically AND socially by improved public advertisements and awareness showing not only the elderly, but younger men and women with hearing loss as well. Such helps decrease the stigma of hearing loss.
    Thank you, Shari

    1. Shari Eberts – NYC – Shari Eberts is a hearing health advocate, writer, speaker and avid Bikram yogi. She blogs at LivingWithHearingLoss.com and serves on the Board of Trustees of Hearing Loss Association of America. She is the former Board Chair of Hearing Health Foundation. Shari has an adult-onset genetic hearing loss and hopes that by sharing her story, it will help others to live more peacefully with their own hearing issues.
      Shari Eberts says:

      Thank you for the work you do for people with hearing loss.

  8. Amazing idea , however I think I’d prefer just having an amazing app on my phone that translates voice into text , one that really works and isn’t all garbled . From my experiences with wearing caption glasses at the movies , they were heavy , hurt your nose , were gawky looking , and very annoying if you had to fit them over glasses you already wore . On the topic of the movies , I just saw a movie at AMC theater in NYC Saturday night and had the worst experience with a captiview device that just would not fit into the cup holder . Was way too snug , just would not press into that cup holder and I ended up holding the darn thing entire movie. Talk about muscle aches .Plus the stares from people (not that I really care )around me including the person next to me who was not very happy I stole her cup holder , and others wondering what the heck is that black thingy she holding!?!! (That’s NYC for ya!) Werent they supposed be putting built in seat caption devices for Broadway Shows and opera in NYC? Would work wonders in movie theaters as well .
    If only these movie chains would spring for the cost.

    1. Shari Eberts – NYC – Shari Eberts is a hearing health advocate, writer, speaker and avid Bikram yogi. She blogs at LivingWithHearingLoss.com and serves on the Board of Trustees of Hearing Loss Association of America. She is the former Board Chair of Hearing Health Foundation. Shari has an adult-onset genetic hearing loss and hopes that by sharing her story, it will help others to live more peacefully with their own hearing issues.
      Shari Eberts says:

      Thank you for sharing your experiences. A good speech to text app would be a great thing too!

    2. The AMC cup holders are a real problem. I always struggle with them. That said, the captioning is actually very good. The Landmark Theater on W. 57th has much better holders.

      1. Thanks Wanda for mention about that theater . Will check it out .
        I plan to contact AMC theaters about their too small cup holders .
        Hmmmmm, doubt they will fix the issue but at least they will be aware .

    3. Gina, I feel the same way about the captioning glasses. By the end of the movie my ears are sore.
      Also, I’ve seen what you call the Captiview, haven’t tried it, and do not like the idea of it. Why should one have to hold something during the movie? I want captioning glasses that are lighter weight and comfortable.
      And totally agree on having an app that that will put it into text options. I’ve been told there are some out there but not Android friendly.

      It seems with amazing technology being created on a daily basis that more people would be concentrating on technology for the deaf and HOH; it would be a profitable item for its’ creator! When I think about the language translator, I wonder why this cannoth be done for “us”.

  9. I have been saying for years I wish face to face conversations could be captioned what a fantastic concept. I’ve been impaired my entire life, speech recognition continues to decline, making communication increasingly difficult, a device like this would be life changing.

    1. Shari Eberts – NYC – Shari Eberts is a hearing health advocate, writer, speaker and avid Bikram yogi. She blogs at LivingWithHearingLoss.com and serves on the Board of Trustees of Hearing Loss Association of America. She is the former Board Chair of Hearing Health Foundation. Shari has an adult-onset genetic hearing loss and hopes that by sharing her story, it will help others to live more peacefully with their own hearing issues.
      Shari Eberts says:

      I agree. Thanks for sharing your thoughts.

  10. This is a dream. Yesterday I attended a Planning Board meeting which I requested and recieved assistance. They provided a court reporter. It was a very helpful as I have been in that meeting room in the past and have never been able to understand the spoken words. The lady worked very hard and made all the difference in my being able to participate. There were some names that didn’t come through as she would have had to enter them in her list before starting. In the past request would be less than adaquate to allow me to fully participate. One incident they would provide a transcript, which would be available after the meeting and not official until it was approved by the board. By then the decision was made so any input I would have could not be considered. Some meetings are more of open format where displays and officials are there to explain various options. Sometimes this works, other times I cannot understand the person speaking. This is where a device which provides instant captions would be wonderful.
    I should ask the agency how much they have to pay for court reporter service. The instant caption dream would make government service more available to those who need it.

    1. Shari Eberts – NYC – Shari Eberts is a hearing health advocate, writer, speaker and avid Bikram yogi. She blogs at LivingWithHearingLoss.com and serves on the Board of Trustees of Hearing Loss Association of America. She is the former Board Chair of Hearing Health Foundation. Shari has an adult-onset genetic hearing loss and hopes that by sharing her story, it will help others to live more peacefully with their own hearing issues.
      Shari Eberts says:

      Thank you for sharing your experiences with captions. If developed fully, these glasses would help so many people.

  11. Jerry Henderson – Pownal Maine – Thank you for coming to my space. This is where I post thoughts, opinions and commentary on a variety of subjects at irregular intervals. I try to do something weekly, but have not nailed down a rigid schedule, like every Wednesday, yet. If you would like email notifications of new posts, you can make that happen right on the site. Simply enter your email address to subscribe. Also, if you would like to comment I welcome that. Just do so in the space at the bottom of any selected post. Sharing thoughts, opinion and commentary is a peculiarly human characteristic. It must be exercised to be enjoyed. Jerry Henderson
    Jerry Henderson says:

    The practical and affordable realization of these glasses seems to me to be a ways off, but we do live in the age of miracles and wonders. This is an instant electronic message 🙂

    1. Shari Eberts – NYC – Shari Eberts is a hearing health advocate, writer, speaker and avid Bikram yogi. She blogs at LivingWithHearingLoss.com and serves on the Board of Trustees of Hearing Loss Association of America. She is the former Board Chair of Hearing Health Foundation. Shari has an adult-onset genetic hearing loss and hopes that by sharing her story, it will help others to live more peacefully with their own hearing issues.
      Shari Eberts says:

      Yes, still in early stages of development, but wouldn’t it be great! Thanks for sharing your thoughts.

  12. Daniel Pistritto – Australia – Daniel Pistritto was diagnosed with a progressive hearing impairment in his early teens but he was determined to find a way to help others with hearing loss. Daniel is now an Audiometrist for Blamey Saunders hears. He is also an A.L.D Product Manager which enables him to test and review assistive communications devices. “Knowing that I’m helping to give back to a client one of their most vital means for communication is rewarding beyond measure”. Daniel speaks from experience when he imparts the following: “When you're experiencing the difficulties of hearing loss, it is very common to feel like no one understands what is like, however you can't expect people to understand what its like, unless they understand! The only way they will be able to understand is if you give them what they need to be able to understand! Talk about your hearing loss”
    Daniel Pistritto says:

    Reblogged this on Talking Hearing Loss and commented:
    This reminds of my first hearing aid which was ‘disguised’ as a pair of glasses!

    I must say that I think that glasses able to provide live captions is a brilliant idea. My nerd brain immediately thinks of all the challenges of designing a usable product (latency, accommodating for accents,etc) but I wish the developers all the best because this would be a fantastic product!

    1. Shari Eberts – NYC – Shari Eberts is a hearing health advocate, writer, speaker and avid Bikram yogi. She blogs at LivingWithHearingLoss.com and serves on the Board of Trustees of Hearing Loss Association of America. She is the former Board Chair of Hearing Health Foundation. Shari has an adult-onset genetic hearing loss and hopes that by sharing her story, it will help others to live more peacefully with their own hearing issues.
      Shari Eberts says:

      There are definitely challenges but I’ve got my fingers crossed they can work it out over time! Thanks for sharing your thoughts.

Leave a Reply