Real Time Sign Language Conversion to Audio

An innovative sign-language translator developed with a grant from EPICS in IEEE also motivated one of the students who worked on the project to pursue his graduate studies at the University of Illinois at Chicago (UIC).

In 2012, IEEE Student Member Sharath Kalkur was one of four students at RNS Institute of Technology in Bangalore, India, who obtained a grant of US$1,305 to design an inexpensive portable device to translate basic American Sign Language into text and audio in real time. With high school students from Sri Aurobindo Memorial School, Sri Kumaran Public School and the RV Integrated School for the Hearing Impaired, the team devised a ground-breaking system that promises to bridge the communication gap between those unable to speak or hear with those who do.

The system electronically documents signed words and phrases, constructs a 3-D representation, sends motion information to a controller to identify the sign, and then both displays the English word on an LCD screen and plays it audibly.

“When we began work on it, I had already decided that digital image processing (DIP) and embedded systems were what interested me most – but this project really fanned the sparks,” says Kalkur. “Our work was based entirely on those technologies as well as computer vision, and left me wanting to learn much more about them.”

He decided to pursue his Master’s at UCI. At least for now, he prefers studying the practical approaches that he thinks universities in the United States emphasize. “India has many opportunities for learning,” he says, “but they take a more theoretical approach.”

He adds that while the EPICS in IEEE project spurred him to attend graduate school, he was also motivated by his parents, teachers and mentors.

When he completes his graduate studies in Chicago, Kalkur will return home to India where he wants to work in an industry involved with DIP – possibly in development for leap motion controllers.

A new group of RMS Institute of Technology students took over the translator project after Kalkur and his teammates graduated. In 2015, the new team obtained $485 to complete it, and the final prototype will be a handheld device. Eventually, it should be available at a price within almost anyone’s means, providing new dignity and confidence for the speech and hearing-impaired.