Seeing Without Sight

Penn State researchers are leading the effort to create machines that can interpret a complex visual scene as well as the human brain. 

For many, grocery shopping is a weekly task that requires minimal effort—create a list, browse the aisles, pay, and return home. But for shoppers with visual impairments, what seems like a simple trip to the store can be tedious, especially when trying to find specific items among the seemingly endless rows of choices.

With insights from the visually impaired community, Penn State professor Vijay Narayanan is leveraging the many resources and bright minds across Penn State—from neuroscientists to undergraduate students—to create a wearable device that can “see” and find just what shoppers are looking for.

A member of the Third Eye project tests the newest version of a smart glove that interacts wirelessly with a head-mounted camera.

A member of the Third Eye project tests the newest version of a smart glove that interacts wirelessly with a head-mounted camera. Image: Michelle Bixby

According to the National Federation for the Blind, in 2015, approximately 300,000 Pennsylvanians reported some form of significant vision loss. Globally, the World Health Organization estimates 253 million people were living with visual impairments in 2017.

To maximize the project’s impact right from the start, Narayanan and his team gathered input and suggestions from groups like the Sight Loss Support Group of Central Pennsylvania (the local chapter of the National Federation of the Blind). Sharing their firsthand experiences, group members were instrumental in shaping the project’s goals and outcomes.

“In order to build these systems, we first needed to understand what this community’s specific needs are,” Narayanan said. “Society becomes a much better place when you go beyond just helping others and discover what others actually need.”

With the support of nine collaborating universities and a five-year, $10 million award from the National Science Foundation, Narayanan’s Visual Cortex on Silicon research team created a haptic glove and smart camera device that “sees” and scans rows of grocery store shelves.

“Society becomes a much better place when you go beyond just helping others and discover what others actually need.”

“We’re developing technology that can give the visually impaired community something beyond what I could have ever imagined,” said Narayanan, Distinguished Professor of Computer Science and Engineering. “Most importantly, it can provide independence.”

When wearing the glove (also known as the Third Eye device), users can easily locate exactly what they’re looking for. As the shopper’s hand reaches out, the camera on the palm of the glove sees what the hand is pointing toward, giving the system a continuous view of what's on the shelves ahead.

Through label and text recognition software, the smart camera is then able to locate specific items among rows of products. After zeroing in on an item, the haptic glove vibrates at different strengths and positions on the hand, guiding users directly to their intended purchases.

“Our whole goal is to augment a person’s natural abilities with this technology,” Narayanan said. “We’re doing this research with the mindset of being as inclusive as possible to the visually impaired community.”

Looking ahead, Narayanan hopes one day the Third Eye device could be as commonplace in stores as motorized shopping carts and be used to highlight sale prices or cheaper product alternatives. Although these broader applications are still distant opportunities, Narayanan is optimistic of all the future holds.

“The biggest impact from this project is the fact that there have been thirty to forty students who have been involved throughout,” Narayanan said. “I’m doing this research here at Penn State because of the opportunity to inspire the next generation of computer scientists. They’re the ones who are going to have an even greater impact on the world.”