Sign Language Recognition for People with Deafness

We are raising funds to develop a computer game that allows players to interact via the British Sign Language (BSL).

Unfortunately, this project was not successful

I am a Senior Lecturer at the University of Westminster and I am crowdfunding to raise awareness of the emerging technologies that can be used in games to help people with severe hearing loss through sign language.

Hearing loss affects millions of families in the UK and we believe that action needs to be taken to better support the communication between parents and children with deafness or severe hearing loss. Based on the statistics from the Deafness support Network in UK, there are more than 10 million people in the UK with some form of hearing loss. From the total 10 million, more than 45,000 are deaf children who use sign language, plus many more who experience temporary hearing loss.

We will develop a vertical slice of the game that will recognise gestures (e.g. finger tracking, large arm-gestures) which will then be used as input in an educational game. The game will track gestures in real time and translates it to spoken English or Images. In addition, the system will allow interaction between multiple players so team work will be encouraged. Incorrect gestures will be shown to the player at a feedback window, reinforcing the signs and introducing an educational element to the game. Currently, the subject we will be using is biology.

Recording Large Gestures

There is already a prototype and algorithms developed by us that can recognise a small subset of the American Sign Language (ASL). Since we are focusing only on UK we will use BSL instead of ASL which is the main language amongst deaf people.

For the prototype we have used the Kinect sensor and the OpenNI open source framework that allowed us to build the point-cloud surface reconstruction algorithm. For the vertical slice the developed algorithms will be protected and will work only with the chosen sensor. The signs were selected based on the fact that animals are a subject area that is enjoyable by the vast majority of children.


Main menu to start the game


Example of how you perform a specific gesture

Currently there are not any commercial BSL products available around the gamification of sign language practice. Evidence suggests that there are not that many people out there actively looking to develop a commercial product apart from the big players in the market such as Intel, Microsoft and Leap Motion. This is not because there’s no money to be made, rather it’s because it’s an emergent market and there’s a lot of value in tapping it first. For example, companies such as Intel are investing in innovative application prototypes that use natural human interfaces such as gestures, voice, and facial tracking (e.g. a Million Minions, Kung Pow Kevin and Magic Doodle Pad).

Most of these games are single player games with hand tracking movement. Our system will be for multiple players and will combine both large arm gestures and finger/hand tracking algorithms. This is the USP.