For many, grocery shopping is a weekly task that requires minimal effort—create a list, browse the aisles, pay, and return home. But for shoppers with visual impairments, what seems like a simple trip to the store can be tedious, especially when trying to find specific items among the seemingly endless rows of choices.
With insights from the visually impaired community, Penn State professor Vijay Narayanan is leveraging the many resources and bright minds across Penn State—from neuroscientists to undergraduate students—to create a wearable device called “Third Eye” that can “see” and find just what shoppers are looking for.
Vijay Narayanan, Robert Noll Chair and Distinguished Professor of electrical engineering and computer science, College of Engineering
Because everyday tasks shouldn’t be a chore for the visually impaired.
“We’re developing technology that can give the visually impaired community something beyond what I could have ever imagined. Most importantly, it can provide independence.”
A Software that “Sees”
With the support of nine collaborating universities and a five-year, $10 million award from the National Science Foundation, Narayanan’s interdisciplinary Visual Cortex on Silicon research team created a haptic glove and smart camera device that “sees” and scans rows of grocery store shelves.
When wearing the glove (also known as the Third Eye device), users can easily locate exactly what they’re looking for. As the shopper’s hand reaches out, the camera on the palm of the glove sees what the hand is pointing toward, giving the system a continuous view of what’s on the shelves ahead.
Through label and text recognition software, the smart camera is then able to locate specific items among rows of products. After zeroing in on an item, the haptic glove vibrates at different strengths and positions on the hand, guiding users directly to their intended purchases.
“Our whole goal is to augment a person’s natural abilities with this technology,” Narayanan said. “We’re doing this research with the mindset of being as inclusive as possible to the visually impaired community.”
“Society becomes a much better place when you go beyond just helping others and discover what others actually need.”
Understanding the Need
According to the National Federation for the Blind, in 2016, nearly 300,000 Pennsylvanians reported some form of significant vision loss. Globally, the World Health Organization estimates 217 million people were living with moderate to severe visual impairments in 2017.
To maximize the project’s impact right from the start, Narayanan and his team gathered input and suggestions from groups like the Sight Loss Support Group of Central Pennsylvania (the local chapter of the National Federation of the Blind). Sharing their firsthand experiences, group members were instrumental in shaping the project’s goals and outcomes.
“In order to build these systems, we first needed to understand what this community’s specific needs are,” Narayanan said. “Society becomes a much better place when you go beyond just helping others and discover what others actually need.”
“I’m doing this research here at Penn State because of the opportunity to inspire the next generation of computer scientists. They’re the ones who are going to have an even greater impact on the world.”
Looking ahead, Narayanan hopes one day the Third Eye device could be as commonplace in stores as motorized shopping carts and be used to highlight sale prices or cheaper product alternatives. Although these broader applications are still distant opportunities, Narayanan is optimistic of all the future holds.
“The biggest impact from this project is the fact that there have been thirty to forty students who have been involved throughout,” Narayanan said. “I’m doing this research here at Penn State because of the opportunity to inspire the next generation of computer scientists. They’re the ones who are going to have an even greater impact on the world.”
Penn State College of Engineering is one of the leading academies for engineering education, research, and service in the United States. Established in 1896, the College of Engineering is the largest academic unit at University Park, consisting of 12 academic departments and schools supporting 45 undergraduate majors, minors and certificates, 39 master’s programs and 18 Ph.D. programs.