Implementing Gesture Recognition in Your Project

Gesture recognition is very widely used in consumer devices, and some of its implementations are quite innovative. Recognizing the correct gesture is the key here and this article discusses how your hardware can help you do that
 

The first question that aroused my curiosity was, ‘What exactly is a ‘gesture?’ According to a dictionary, gesture is a sequence of postures that are connected by motions over a short period of time. I did not find that really helpful. I’ll explain it in an easier way—a gesture is a form of non-verbal or non-vocal communication. So your messages are communicated by visible bodily actions like waving, swiping, facial movements or even pointing to the sky.
What is gesture recognition
This technology enables devices to interpret human gestures using sensors, whose data is then processed using mathematical algorithms. The system is built to identify specific human gestures, and then use them either to convey information or to control a device. The gestures used have to be intuitive, simple and universally acceptable to ensure they are easily adopted by users.
Gesture recognition makes computers more accessible for the physically-challenged and also makes interaction more natural in the 3D virtual world or in gaming. In today’s world, gesture recognition is an evolving technology that can be seen all around us. The latest phones, computers, gaming consoles, and even TVs now feature it.
How it works
The human body’s movements are read by an imaging sensor like a camera, which then sends that data to a computer. The computer uses the gestures captured as inputs, processes this input to understand what was gestured, and then sends the answer as a command to the currently running application or control device.
Let us look at an example of a person clapping his hands in front of a camera. When this clapping gesture data is processed by the connected computer, it could produce the sound of cymbals, as programmed. The specialty of this technology is that users do not have to wear any special equipment or attach any device to their bodies, as is the case with some alternative technologies.
Sensing the gestures
When you gesture at your computer or mobile device, it obviously needs a way to ‘see’ or ‘sense’ what you are doing. This is where sensors come in. There are a number of sensing technologies that can be considered, but we’ll stick to the popular ones here.

Accelerometers. Also known as inertial or acceleration sensors, these are used to detect the changes in force resulting from the fall, motion, tilt, shock, positioning or vibration of the body they are connected to, with respect to time. These measure linear acceleration, so if the device is in a free fall, the accelerometer will give a ‘0’ value for acceleration.

Application. In the case of a smartphone, the direction of the screen changes as soon as we tilt or turn the phone. The direction in which we hold the phone (horizontal or vertical) is sensed by an accelerometer and the information in the form of electrical signals is sent to the screen, which automatically adjusts itself either in landscape or in portrait mode. So, here, contact gesture recognition plays an important role.

Accelerometers have paved the way for many games to be developed for smartphones, especially for some of those sports simulation games that let you use the phone as a steering wheel or a fishing pole. To get an idea of the range of accelerometers that you can select from, refer to Table I.


Gyroscopes. Gyroscopes, also popularly known as gyros, have had a major impact in the world of gesture recognition. Their ability to measure the orientation of a device enables them to measure both vertical and horizontal rotation. This has brought smoothness and preciseness to the operation of a mobile device. The next time you hold a phone on its side, and the screen transforms into landscape mode automatically, you know which component to thank.

Application. In the case of a smartphone or a tablet, adding a gyro and an accelerometer allows the device to sense motion on six axes including roll, pitch and yaw—which helps you fly your airplane in a flight simulator game on a phone. So if accelerometers have given more business to game developers, gyros have helped them be more innovative.
Bend sensors. These are also known as flex sensors. As we know, even finger movements are considered as gestures, so bend sensors help us to identify those gestures by recognising finger postures. They work on the principle that resistance increases with an increase in the bend.

Data gloves. Gesture recognition technology has not only been implemented in smartphones and tablets, but also in television sets and computers. In order to implement hand gesture recognition on these devices, data gloves have been developed. These gloves identify the finger and hand gestures in real time scenario. These are wired either to a TV set or to a computer and then worn by the person operating the device. These act as an input device and eliminate the use of a mouse or a keyboard.
Application. The bend sensors are placed on the joints of all the fingers in the data glove, so the more the fingers are bent, the more is the resistance which is introduced in the device for further action.
Infrared proximity sensors. These have the ability to detect the presence of nearby objects without any physical contact. They emit an electromagnetic field and then identify the changes either in the field or in the return signal, which they detect after reflection from an object that has got close to the sensor. There are many types of proximity sensors; however, infrared proximity sensors play a very important role in gesture recognition. As soon as these infrared sensors recognise the movement of a body in front of them, they pass a signal to the device for further action.
Application. Many of us have seen a wash basin with an automated tap. The tap turns on as soon as it senses some hand movement under it. This is because of the presence of infrared proximity sensors that detect your hand movement. Here’s a fun thing to try out—if you keep your hand under the tap without any movement, the tap would not allow the water to fall. This is because if you keep your hand immobile, the sensor does not sense a changing field and thus thinks that your hand is not in front of it any more.
Capacitive proximity sensors. They support gesture recognition enabling robust, easy-to-use and feature-rich user touch interfaces on a device—these are what make up your capacitive touch screen. Such sensors allow users to operate the device just by the touch of their skin or finger rather than by applying pressure. Some of the newer models have superior sensors that do not even require a touch—hovering above the screen is enough to get their attention.
Application. This has helped to bring about a new era of devices with superior touch screens that allow multiple points of input, and better durability. What’s more, capacitive screens allow the device to have a glass or some other rigid protection on top of the touch screen, thus improving the device’s ergonomics.

Touch-sensing controllers, such as the CapSense from Cypress, have specific technical improvements like water tolerant capacitive touch buttons, which eliminates false touches under moist conditions. “They also allow capacitive proximity sensing for up to 30cm, which is instrumental in enabling features such as ‘wake-on approach,’ and can even replace the IR sensors mentioned in the previous section,” explains Anbarasu Samiappan, product apps manager, Cypress Semiconductor.

Microsoft’s Digits prototype is part of an effort to bring gesture-based control in a mobile device. This wrist-worn sensor uses IR laser, LEDs and camera to recognise the fingers movement (Image courtesy: Microsoft)

What’s coming up in the future
I am pretty sure we will see something  like Tom Cruise’s sorcery in ‘Minority Report’ within a couple of decades. But until that happens, the field is taking great strides forward.

Researchers have been working on new kinds of hardware to compute gestures, including FPGAs. “Apart from basic components to capture the images, an FPGA-based gesture recognition system handling large computational complexity of optical flow to perform parallel processing is a low-cost solution. Using an FPGA system, gesture recognition can be implemented at 30 frames per second, and the system software can subsequently schedule all tasks during processing,” explains P. Chow Reddy, manager (R&D), Power Division, ICOMM Tele Pvt Ltd.

Apart from the focus in consumer electronics, gesture recognition seems to have matured enough to be noticed and implemented in industrial settings. The BMW Group has demonstrated a gesture-driven system that allows a quality assurance worker to “…examine and document flaws in a component simply by pointing.” Ford had earlier announced its own gesture-based ambitions in the form of ‘a complete virtual factory.’ Industry aside, even Microsoft Office 2013 has gesture recognition built into it.

At Electronics Rocks‘13 conference, one of our speakers Hemant Surale showcased a cool smart glove product that he had designed. Hemant says that the present scenario in gesture recognition seems that development in this area is at its peak. “Most of the devices comes with their own proprietary SDKs for application developers. In addition there are many emerging startups developing effective way to interact with day to day equipments. But, Internet of things will present new challenges with respect to integration of such systems into day to day life of users.” Hemant is also an active member of the IoTBLR community. IoTBLR is a community based in Bangalore, which is excited about connecting disparate devices to one another via the web to create Internet of Things products. Any engineer can find them on facebook at facebook.com/groups/IoTBLR
The field of gesture recognition seems to be gearing up for quite a future, and design houses working on this technology better get ready for the competition that comes with any high-profile industry.