IoT based Gesture Recognition¶
Created by Cainvas Scholars-¶
Photo by Marco Coppeto on Dribbble and Matt Harvey on Vimeo
Gestures are mapped to the corresponding accelerometer and gyroscope values recorded during motion. Here, we have accelerometer and gyroscope values along x, y, z axes recorded 100 times for one gesture, i.e, 600 data points for one gesture.
Dataset: The sensor values were recorded using an app (github link here) and consolidates using csv.
Algorithm: A tensorflow deep learning model with relu and softmax activations was used.
Photo by Adrien King on Unsplash
In [1]:
import pandas as pd
import csv
import os
import tensorflow as tf
import keras
from sklearn.preprocessing import normalize, OneHotEncoder
import numpy as np
import matplotlib.pyplot as plt
import urllib
List of gestures used
In [2]:
gestures = ['down_to_up', 'forward_clockwise', 'left_fall', 'up_clockwise',
'up_anticlockwise', 'left_to_right', 'right_to_left', 'forward_fall']
In [3]:
df = pd.read_csv('https://cainvas-static.s3.amazonaws.com/media/user_data/cainvas-admin/gesture60_8_.csv')
Normalize and shuffle
In [4]:
df[df.columns[:-1]] = normalize(df[df.columns[:-1]])
df = df.sample(frac=1, random_state=13).reset_index(drop = True)
df
Out[4]:
Visualization¶
In [5]:
data = pd.DataFrame(df)
In [6]:
col = ['ax','ay','az','gx','gy','gz']*100
col.append('Gesture')
data.columns = col
In [7]:
def extract(gesture,data_dict):
k = data[data["Gesture"]==gesture].shape[0]
for j in range(0,k):
if j == 0:
data_dict[gesture] = data[data['Gesture']==gesture].iloc[0:1,0:6]
else:
data_dict[gesture] = data_dict[gesture].append(data[data['Gesture']==gesture].iloc[j:j+1,0:6])
for i in range(6,600,6):
data_dict[gesture] = data_dict[gesture].append(data[data['Gesture']==gesture].iloc[j:j+1,i:i+6])
In [8]:
data_dict = {}
for i in gestures:
extract(i,data_dict)
#print(data_dict[i].head())
#data_dict
In [9]:
def visualize(type, gesture):
for i in gestures:
title = type.upper()+' for \"'+i.upper()+'\" gesture'
plt.title(title)
if type == 'acceleration':
index = range(1, len(data_dict[i]['ax']) + 1)
plt.plot(index, data_dict[i]['ax'], 'g.', label='x', linestyle='solid', marker=',')
plt.plot(index, data_dict[i]['ay'], 'b.', label='y', linestyle='solid', marker=',')
plt.plot(index, data_dict[i]['az'], 'r.', label='z', linestyle='solid', marker=',')
if type == 'gyro':
index = range(1, len(data_dict[i]['ax']) + 1)
plt.plot(index, data_dict[i]['gx'], 'g.', label='x', linestyle='solid', marker=',')
plt.plot(index, data_dict[i]['gy'], 'b.', label='y', linestyle='solid', marker=',')
plt.plot(index, data_dict[i]['gz'], 'r.', label='z', linestyle='solid', marker=',')
plt.legend()
plt.show()
In [10]:
visualize('gyro', gestures)
In [11]:
visualize('acceleration', gestures)
Acceleration plot of 'left_to_right' has an outlier.
Identifying and removing it.
In [12]:
outlier = data_dict['left_to_right'][data_dict['left_to_right']['ax'] > 0.06]
print(outlier)
outlier_index = outlier.index.unique().values
df = df.drop(outlier_index).reset_index(drop = True)
In [13]:
df
Out[13]:
In [14]:
df.to_csv('gesture60_8_normalized.csv', index = False)
One hot encoding the labels
In [15]:
y_df = pd.get_dummies(df.Gesture)[gestures]
print(y_df)
print(np.array(y_df)[0].shape)
In [16]:
df = pd.concat([df, y_df], axis=1)
df.head()
Out[16]:
In [17]:
train_count = int(0.8*len(df))
test_count = len(df) - train_count
print(train_count, test_count)
x_train, y_train = np.array(df)[:train_count, :600].astype('float32'), np.array(df)[:train_count, -(len(gestures)):].astype('int64')
x_test, y_test = np.array(df)[train_count:, :600].astype('float32'), np.array(df)[train_count:, -(len(gestures)):].astype('int64')
print(x_train.shape, y_train.shape, x_test.shape, y_test.shape)
In [18]:
x_train[0].shape
Out[18]:
Model¶
In [19]:
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(512, activation="relu", input_shape = (600,)))
model.add(tf.keras.layers.Dense(128, activation="relu"))
model.add(tf.keras.layers.Dense(64, activation="relu"))
model.add(tf.keras.layers.Dense(len(gestures), activation="softmax"))
In [20]:
model.summary()
In [21]:
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])
In [22]:
model.fit(x_train, y_train, epochs = 16, batch_size = 8)
Out[22]:
In [23]:
model.evaluate(x_test, y_test)
Out[23]:
Tiny model¶
In [24]:
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(16, activation="relu", input_shape = (600,)))
model.add(tf.keras.layers.Dense(8, activation="relu"))
model.add(tf.keras.layers.Dense(4, activation="relu"))
model.add(tf.keras.layers.Dense(len(gestures), activation="softmax"))
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])
model.summary()
In [25]:
model.fit(x_train, y_train, epochs = 256, batch_size = 32)
print("\n Test set")
model.evaluate(x_test, y_test)
Out[25]:
In [26]:
y_pred = model.predict(x_test)
for i in range(len(y_pred)):
m = max(y_pred[i])
j = np.where(y_pred[i] == m)[0][0]
k = np.where(y_test[i] == 1)[0][0]
if(j!=k):
print("------- Mismatch -------")
print("Actual: " + gestures[k] + "\nPredicted: " + gestures[j] + "\nConfidence: " + str(m) + "\n")
In [27]:
model.save('gestureTinyModel.h5')
In [28]:
model = tf.keras.models.load_model('gestureTinyModel.h5')
In [29]:
model.evaluate(x_test, y_test)
Out[29]:
In [30]:
!deepCC gestureTinyModel.h5
Gesture based home automation¶
The below snippet is a poc for gesture based home automation - controlling the various appliances at home with gestures.
The aim is to embed the sensors in a wearable device.
We move our hands in so many different ways and there is a high chance that one of these many movements map to the recorded gestures and trigger responses. To counter this, we can include a tiny button/other mechanism to indicate the start of the gesture.
List of gestures and the various categories for reference, mapped to the list of gestures above.
Here, count refers to moving to the next instance of the appliance (eg, countIncrease - moving from fan1 to fan2, countDecrease - moving from fan2 to fan1) and next refers to choosing the next appliance.
In [31]:
gesture_key = {}
gesture_key["appliance"] = ["light", "fan"]
gesture_key["operation"] = ["switch", "increase", "decrease"]
gesture_key["count"] = ["countIncrease", "countDecrease"]
gesture_key["next"] = ["next"]
gestures_home = []
for x in gesture_key:
gestures_home.extend(gesture_key[x])
print(gestures_home)
In [32]:
lines = urllib.request.urlopen('https://cainvas-static.s3.amazonaws.com/media/user_data/cainvas-admin/test7.txt').read()
lines = lines.decode('utf-8').split('\n')[:-1]
appliance_bool = True
operation_bool = False
# mapping each appliance to the operations that can be done on them
appliance_to_operation = {'light' : {'switch'},
'fan' : {'switch', 'increase', 'decrease'}}
# descriptions and cuurent state of the configured (installed) appliances
configured = {}
configured['light'] = { 1 : ["off"],
2 : ["off"]}
configured['fan'] = { 1 : ["off", 0, 5]} # 0 - cuurent speed, 5 - maximum
# list of operations performed (just for reference)
performed = []
# initializations
count = 0
appliance = ""
for line in lines[1:]: # first line of text file contains column headings
line = line.split(' ')[:-1] # last value is gesture name which is blank here
line = np.array(line).astype('float32').reshape(1,600)
result = model.predict(normalize(line))
m = max(result[0])
g = gestures_home[np.where(result[0] == m)[0][0]]
print("Recognized: " + g + " with " + str(m) + " confidence level.")
# if appliance is to be selected
if (appliance_bool):
if g in configured:
print("Selected " + g)
# initializing instance of appliance
count = 0
# Setting appliance selection to false and operation on appliance to true
appliance_bool = False
operation_bool = True
# keeping track of current appliance
appliance = g
performed.append([g, "Selected"])
else:
print("Invalid gesture. Try again!")
# if operation is to be done on appliance
elif (operation_bool):
# gesture to move to next instance of appliance
if g == 'countIncrease':
if count < len(configured[appliance]):
count = count + 1
print("Instance " + str(count) + " of " + appliance + " selected.")
performed.append([appliance, g, count])
else:
print("Last instance reached.")
# gesture to move to previous instance of appliance
elif g == 'countDecrease':
if count > 1:
count = count - 1
print("Instance " + str(count) + " of " + appliance + " selected.")
performed.append([appliance, g, count])
else:
print("First instance reached.")
# gesture to change current selelcted appliance
elif g == 'next':
print("Change appliance")
application = ""
# Setting appliance selection to true and operation on appliance to false
appliance_bool = True
operation_bool = False
performed.append(['Changed'])
# gesture to perform an operation on the selelcted appliance (checking if the operation is valid for the selected appliance)
elif g in appliance_to_operation[appliance]:
# checking if instance of appliance is selected
if count !=0:
# toggle switch on/off
if g == 'switch':
configured[appliance][count][0] = 'on' if configured[appliance][count][0]=='off' else 'off'
performed.append([appliance, g, configured[appliance][count][0]])
print("Instance " + str(count) + " of " + appliance + " switched " + configured[appliance][count][0])
# increase fan speed
elif g == 'increase':
if configured[appliance][count][1]<configured[appliance][count][2]:
configured[appliance][count][1] = configured[appliance][count][1] + 1
performed.append([appliance, g, configured[appliance][count][1]])
print("Instance " + str(count) + " of " + appliance + " - speed increased to " + str(configured[appliance][count][1]))
else:
print("Max speed reached!")
#decrease fan speed
elif g == 'decrease':
if configured[appliance][count][1] > 0:
configured[appliance][count][1] = configured[appliance][count][1] - 1
performed.append([appliance, g, configured[appliance][count][1]])
print("Instance " + str(count) + " of " + appliance + " - speed decreased to " + str(configured[appliance][count][1]))
else:
print("Min speed reached!")
else:
print('Select instance of appliance to work with!')
else:
print("Invalid operation for appliance!")
print()
print('\nOperations performed')
for x in performed:
print(x)
print('\nFinal state of appliances')
print(configured)
Other applications¶
Sign language convertor
Embedded in a wearable, this allows the gestures performed to be mapped to voice output.
Gaming
Move your hands around for a more real life experience.
Music generation