Hand Tracking Using AI: What’s Under the Hood?

Hand Tracking: Why Do We Need It?

When I look at the potential of modern technologies, I try to visualize what the world of the future will look like. If we have such broad opportunities today, what is waiting for us tomorrow? It seems that our grandparents and parents could not even imagine the devices that today have become an indispensable part of our lives. 

Wearable Technology Definition

The features of present-day technologies never cease to amaze me. If you or people around you have an Apple Watch, you’ve definitely heard about its most interesting and useful functions. Users having monitoring devices get the chance to monitor their heart rhythm, measure blood oxygen level, and even track their breath rate. But what if I tell you that these functions are nothing compared to the overall potential of hand tracking?

First of all, devices like smartwatches (69.3 million sales in 2019 worldwide) function on the basis of wearable technology. What is wearable technology? It is the technology that forms the core of wearable devices. The latter detect, interpret, and gather information about body signals using AI technology. Wearables are known for their wide application: from commercial use to healthcare, from navigation to textile.

Market Analysis

IBISWorld reports on wearable device manufacturing in the US point to the great prospects of this field. It is expected that the wearable technology market will continue to grow even under the conditions of the pandemic. First, wearables are becoming mainstream in fashion. Second, they give users the chance to be aware of their health metrics without leaving their homes, which also allows them to feed two birds with one scone.

Along with wearable devices, technology of hand tracking and gesture recognition is developing. It is more complex and serves as a logical extension of touchscreen technology. By 2025, the gesture recognition market size will likely grow to USD 30.6 billion — Grand View Research states. Moreover, because of its advanced functions and easy adoption by end users, hand tracking is expected to fully transform human-machine interaction. So, we’ve made it clear why AI development companies are so popular now and why hand tracking is called the future that’s available today. And now, let’s discuss how hand tracking actually works.

How It Works

Do you have a wearable device like Apple Watch, Withings, or Galaxy Watch?  Then you know that wearable devices are configured through a smartphone app that uses Bluetooth. The development of wearable technology became possible due to the fourth version of Bluetooth — Bluetooth Low Energy (BLE). BLE significantly reduces power consumption and increases battery longevity. 

Every time you wear a wearable device, it records data from an accelerometer and gyroscope. Data from each axis is recorded to internal memory. The sampling frequency of this data depends on several factors: the configuration of devices of a particular manufacturer, their functionality, and users’ physical activity. 

The most common example of hand tracking in action is counting steps. For this purpose, you can use ready-made methods that are based on mathematical polynomials, neural networks, and machine learning algorithms. However, now this task is solved in a simpler way — using a ready-made motion-detection mechanism of the accelerometer of a device itself. 

Using Digital Motion Processor (DMP), you get initial information about hand tracking with minimal use of wearable device resources and without using your smartphone. For example, there is a hand gesture of putting a smartwatch in a horizontal position. This gesture turns on the screen light of a device so that a user can view information on a screen. Therefore, using a DMP and processor of a wearable device, you can perform basic check-ups on specific movement detection.

Yet, the use of DMP applies only to basic aspects of hand tracking. For more complex functionality, you should use the CPU of a processor or external devices such as a smartphone or cloud computing in the form of lambda functions. That is, to obtain more complex data, a wearable device must record lower-level information: the value of an accelerometer and gyroscope along the x, y, z axes. Below, you can see how the components of Apple Watch look like.

Next, information from internal memory must be transmitted over a certain network interface to a smartphone or remote server (or lambda). Algorithms process this data using neural networks and machine learning. When they have processed data on the database, output metrics are transmitted to the end user. These metrics can include the following: peculiarities of walking or running, swimming style, or patient health indicators.

How to Develop Hand Tracking Technology

The task of developing hand tracking technology can hardly be named easy. One paragraph is not enough to describe all the subtleties that should be considered. To give you the maximum possible, I will subdivide this topic into two discussions: hardware peculiarities and hand tracking configuration. Let’s start with the first.

Hardware for Hand Tracking: What Is Needed?

The end output of any algorithm includes a certain result, metric, or coefficient. In our case, we want to use wearable technology to recognize specific gestures. Consequently, the task is to recognize a particular pattern of a hand movement. To develop hand-tracking, consider the following:

I should note that quite often, one wearable device is not enough to recognize movements. For example, if you need to recognize a combination of movements of two hands when swimming or running, you should use two wearable devices. Meantime, if you need to recognize certain movements of a person working at a telecom tower or oil well, you should have four devices. Only using four devices and more, you can detect specific gestures and get a full picture. Thus, for proper operation of algorithms, you should also determine the required number of wearable devices that will function as sources of data from an accelerometer and gyroscope.

An accelerometer, the source of data, is one more important element of wearable technology. Manufacturers of semiconductor electronics try to produce the most adaptive accelerometers that will fit market needs and be compatible with different sets of devices. A hardware component of accelerometers can differ in energy consumption, input noise of axes, and implemented motion-detection function.

You should know that the energy consumption of an accelerometer affects the durability of a device. The latter depends on how often its function of collecting data is used. I should also highlight that data collection can be performed on each device, including an accelerometer, gyroscope, and magnetometer, and on each of the x, y, z axes.

Data is the most important aspect of machine learning algorithms and neural networks. If data is inaccurate, it leads to poor training and inadequate output results of algorithms. It happens that the noise of certain frequencies is introduced directly by an accelerometer. Such a situation can also lead to the fact that parameters of feature engineering generated on the basis of axes will be invalid. Hence, you should consider the range of input noise when choosing a particular wearable device and its accelerometer.

Combining Hardware Data and Data Science

When a specific wearable device or several devices are selected, you can start configuring them. An important parameter to consider is the frequency of data collection from the accelerometer and gyroscope. In fact, the more data we get, the better for gesture recognition algorithms. As a result, the defined parameters of feature engineering that will be used for training will be more accurate. 

However, apart from the benefits, a large volume of data has its drawbacks. A high sampling frequency of accelerometer and gyroscope leads to higher power consumption and faster memory filling. In addition, the method of data transmission is used. For example, when for one second, a high-frequency recording is performed, and for another second, no data is recorded at all.

Subsequently, you should set the following parameters:

• Sampling frequency;
• Width of data collection window;
• The channel of data transmission.

Data processing can be performed directly on a mobile device. For example, some smartwatches collecting data during your workout may show what exercise you are doing or what muscle group you are training. The gathered data is processed on a mobile device, where correlation parameters of feature engineering of specific axes of accelerometer and gyroscope are actually defined. 

Another way to define certain patterns is to use machine learning algorithms k-means and c-means, which will determine necessary characteristics and sequences of patterns. But what if you want to apply several wearable devices, for example, at a hospital for patients? Then, you should use the so-called edge device. 

Edge devices act as aggregators of device collection chosen from IoT devices — in our case, these are wearable devices. Wearable devices are connected via WiFi or Bluetooth to the edge device, which already sends data, for example, to AWS lambda. Here, the algorithm is directly executed using neural networks for recognizing specific parameters.

Hand Tracking Application

Today, hand tracking software is widely applied both for private purposes to recognize certain simple movements and for medical purposes to predict specific attacks in sick people. Let’s review different types of wearable technology in use to give you a better understanding of the potential of this field.

Use for Private Purposes

We’ll start with a simple example: hand tracking technology is used to detect whether a person wears a device or not. Or whether a wearable device has been taken or not. In fact, it all comes down to checking whether a person is active, whether it’s just a little activity, or whether the device is removed. 

Simple optical sensors can “detect” a person’s heart rate even if the device is not worn on the hand, which leads to erroneous values. At this point, an accelerometer and a gyroscope come in handy. With their help, on the basis of a fully connected neural network, you can define that the device is simply inactive for a long time. This is used to turn off the maximum number of functions when the device is not in use and save battery power in this way.

One more important function of wearable devices like an Apple Watch is defining when a user washes their hands. Apple Watch informs a user that they have not washed their hands yet. This is an instant message displayed directly on the wearable device. In this way, Apple Watch helps fight COVID-19 and other diseases that are transmitted by unclean hands.

Application in Healthcare

Wearable devices can be used for patients who, for instance, are being treated at home. Let’s take a person with Alzheimer’s disease who may suffer from attacks. These attacks can be predicted by wearable devices based on certain micro-oscillations of a hand, which are derived from an accelerometer or gyroscope.

Another common way of hand tracking application is detecting when an elderly using a wearable device has fallen. This function is frequently used by people who leave their older parents at home while being at work. Yet, you should consider that a certain number of falls are not detected due to the large variability of fall events. 

The fall event is determined by two key components — actual-fall event and after-fall event. The main algorithm of falling is executed on a wearable device. Once it is discovered that a person has fallen, a wearable device has to inform end users (for example, relatives or neighbors) about the case via an edge device or instant message.

Application in Sport

The use of wearable devices in sport allows people to define their style of swimming or running. Based on the obtained data, they can predict their physical capabilities, get exercise tips, and detect physical shortcomings that may not be visible at first glance.

Wrap Up

This article explains everything you should know about hand tracking technology if you want to implement it in your product. As you see from the described examples, no matter what area or business you target — AI development will expand your opportunities everywhere. 

Hand tracking and gesture recognition that function on the basis of AI will be useful for those who care about their health and body or about their closest ones. Under the conditions of the pandemic, this description fits almost each of us, which also doubles your chances of developing a great and useful solution, And if you need more details, guidance, or friendly advice, do not hesitate to contact our specialists — we are always happy to help aspiring people reach their goals!