Cainvas
Model Files
muscle_gesture.h5
keras
Model
deepSea Compiled Models
muscle_gesture.exe
deepSea
Ubuntu

Gesture recognition using muscle activity

Credit: AITS Cainvas Community

Photo by Yohanen on Dribbble

Electromyography is a technique for recording and evaluating electrical acitvity produced in the skeletal muscles.

The readings from the muscle activity sensor is then fed to a model that can be trained on gestures, separately for each user (customization for a user) or to obtain a pre trained model ready for direct implementation in various applications like prosthetic arms, game control etc.

In [1]:
import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.model_selection import train_test_split
from keras import layers, models, losses, callbacks, optimizers
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
import random

The datatset

On Kaggle by Kirill Yashuk

Four different motion gestures were recorded using the MYO armband.

Each reading has 8 consecutive readings of all 8 sensors resulting in 64 columns of EMG data. The last column is the gesture corresponding to the readings.

In [2]:
!wget -N "https://cainvas-static.s3.amazonaws.com/media/user_data/cainvas-admin/muscle_gesture.zip"
!unzip -o "muscle_gesture.zip"
!rm "muscle_gesture.zip"
--2021-04-17 14:10:04--  https://cainvas-static.s3.amazonaws.com/media/user_data/cainvas-admin/muscle_gesture.zip
Resolving cainvas-static.s3.amazonaws.com (cainvas-static.s3.amazonaws.com)... 52.219.62.36
Connecting to cainvas-static.s3.amazonaws.com (cainvas-static.s3.amazonaws.com)|52.219.62.36|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 763626 (746K) [application/zip]
Saving to: ‘muscle_gesture.zip’

muscle_gesture.zip  100%[===================>] 745.73K  --.-KB/s    in 0.004s  

2021-04-17 14:10:04 (183 MB/s) - ‘muscle_gesture.zip’ saved [763626/763626]

Archive:  muscle_gesture.zip
  inflating: muscle_gesture/2.csv    
  inflating: muscle_gesture/3.csv    
  inflating: muscle_gesture/1.csv    
  inflating: muscle_gesture/0.csv    
In [3]:
df0 = pd.read_csv('muscle_gesture/0.csv', header = None)
df1 = pd.read_csv('muscle_gesture/1.csv', header = None)
df2 = pd.read_csv('muscle_gesture/2.csv', header = None)
df3 = pd.read_csv('muscle_gesture/3.csv', header = None)

df = pd.concat([df0, df1, df2, df3])
df
Out[3]:
0 1 2 3 4 5 6 7 8 9 ... 55 56 57 58 59 60 61 62 63 64
0 26.0 4.0 5.0 8.0 -1.0 -13.0 -109.0 -66.0 -9.0 2.0 ... -28.0 61.0 4.0 8.0 5.0 4.0 -7.0 -59.0 16.0 0
1 -47.0 -6.0 -5.0 -7.0 13.0 -1.0 35.0 -10.0 10.0 -4.0 ... -25.0 47.0 6.0 6.0 5.0 13.0 21.0 111.0 15.0 0
2 -19.0 -8.0 -8.0 -8.0 -21.0 -6.0 -79.0 12.0 0.0 5.0 ... -83.0 7.0 7.0 1.0 -8.0 7.0 21.0 114.0 48.0 0
3 2.0 3.0 0.0 2.0 0.0 22.0 106.0 -14.0 -16.0 -2.0 ... -38.0 -11.0 4.0 7.0 11.0 33.0 39.0 119.0 43.0 0
4 6.0 0.0 0.0 -2.0 -14.0 10.0 -51.0 5.0 7.0 0.0 ... 38.0 -35.0 -8.0 2.0 6.0 -13.0 -24.0 -112.0 -69.0 0
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2917 -3.0 -1.0 -1.0 -1.0 -28.0 20.0 5.0 0.0 -5.0 0.0 ... -3.0 1.0 4.0 3.0 4.0 -51.0 -49.0 5.0 -9.0 3
2918 -13.0 -5.0 -4.0 -3.0 -4.0 -24.0 -10.0 -8.0 20.0 9.0 ... 6.0 -3.0 -3.0 -3.0 -5.0 -4.0 -45.0 -12.0 -15.0 3
2919 -1.0 -3.0 -1.0 1.0 30.0 38.0 -1.0 36.0 -10.0 1.0 ... 14.0 -8.0 -4.0 -4.0 -4.0 -21.0 -29.0 -5.0 0.0 3
2920 1.0 4.0 4.0 5.0 9.0 -10.0 4.0 1.0 -2.0 -1.0 ... -16.0 -3.0 0.0 -3.0 -5.0 -36.0 -90.0 3.0 5.0 3
2921 -2.0 4.0 2.0 -4.0 12.0 3.0 -2.0 9.0 -8.0 -2.0 ... 2.0 1.0 0.0 -1.0 -2.0 -30.0 64.0 11.0 5.0 3

11678 rows × 65 columns

In [4]:
df[64].value_counts()
Out[4]:
2    2943
3    2922
0    2910
1    2903
Name: 64, dtype: int64

Its a fairly balanced dataset.

In [5]:
df.describe()
Out[5]:
0 1 2 3 4 5 6 7 8 9 ... 55 56 57 58 59 60 61 62 63 64
count 11678.000000 11678.000000 11678.000000 11678.000000 11678.000000 11678.000000 11678.000000 11678.000000 11678.000000 11678.000000 ... 11678.000000 11678.000000 11678.000000 11678.000000 11678.000000 11678.000000 11678.000000 11678.000000 11678.000000 11678.000000
mean -0.520380 -0.726837 -0.739082 -0.729748 -0.159103 -0.554890 -1.272649 -0.661843 -0.665953 -0.654222 ... -0.932694 -0.836958 -0.740623 -0.768710 -0.705343 -0.146686 -0.374807 -1.449306 -0.609094 1.503254
std 18.566709 11.766878 4.989944 7.441675 17.850402 25.809528 25.089972 15.408896 18.123854 11.841260 ... 15.158993 18.204465 12.005206 4.969758 7.384410 17.841479 25.551082 25.259736 15.530091 1.117541
min -116.000000 -104.000000 -33.000000 -75.000000 -121.000000 -122.000000 -128.000000 -128.000000 -110.000000 -128.000000 ... -128.000000 -116.000000 -128.000000 -46.000000 -74.000000 -103.000000 -128.000000 -128.000000 -124.000000 0.000000
25% -9.000000 -4.000000 -3.000000 -4.000000 -10.000000 -15.000000 -6.000000 -8.000000 -9.000000 -4.000000 ... -8.000000 -9.000000 -4.000000 -3.000000 -4.000000 -10.000000 -14.000000 -6.000000 -8.000000 1.000000
50% -1.000000 -1.000000 -1.000000 -1.000000 0.000000 -1.000000 -1.000000 -1.000000 -1.000000 -1.000000 ... -1.000000 -1.000000 -1.000000 -1.000000 -1.000000 0.000000 -1.000000 -1.000000 -1.000000 2.000000
75% 7.000000 3.000000 2.000000 3.000000 10.000000 13.000000 4.000000 6.000000 6.000000 3.000000 ... 6.000000 6.000000 3.000000 2.000000 3.000000 10.000000 13.000000 3.000000 6.000000 3.000000
max 111.000000 90.000000 34.000000 55.000000 92.000000 127.000000 127.000000 126.000000 127.000000 106.000000 ... 114.000000 127.000000 105.000000 29.000000 51.000000 110.000000 127.000000 127.000000 127.000000 3.000000

8 rows × 65 columns

The column values are have different range (min, max) and varying standard deviations.

These can be standardized to have mean=0 and sd=1 using the StandardScaler later in the notebook.

Defining the input and output

In [6]:
X = df[df.columns.tolist()[:-1]]
X
Out[6]:
0 1 2 3 4 5 6 7 8 9 ... 54 55 56 57 58 59 60 61 62 63
0 26.0 4.0 5.0 8.0 -1.0 -13.0 -109.0 -66.0 -9.0 2.0 ... 21.0 -28.0 61.0 4.0 8.0 5.0 4.0 -7.0 -59.0 16.0
1 -47.0 -6.0 -5.0 -7.0 13.0 -1.0 35.0 -10.0 10.0 -4.0 ... -105.0 -25.0 47.0 6.0 6.0 5.0 13.0 21.0 111.0 15.0
2 -19.0 -8.0 -8.0 -8.0 -21.0 -6.0 -79.0 12.0 0.0 5.0 ... -128.0 -83.0 7.0 7.0 1.0 -8.0 7.0 21.0 114.0 48.0
3 2.0 3.0 0.0 2.0 0.0 22.0 106.0 -14.0 -16.0 -2.0 ... -54.0 -38.0 -11.0 4.0 7.0 11.0 33.0 39.0 119.0 43.0
4 6.0 0.0 0.0 -2.0 -14.0 10.0 -51.0 5.0 7.0 0.0 ... 60.0 38.0 -35.0 -8.0 2.0 6.0 -13.0 -24.0 -112.0 -69.0
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2917 -3.0 -1.0 -1.0 -1.0 -28.0 20.0 5.0 0.0 -5.0 0.0 ... -3.0 -3.0 1.0 4.0 3.0 4.0 -51.0 -49.0 5.0 -9.0
2918 -13.0 -5.0 -4.0 -3.0 -4.0 -24.0 -10.0 -8.0 20.0 9.0 ... 5.0 6.0 -3.0 -3.0 -3.0 -5.0 -4.0 -45.0 -12.0 -15.0
2919 -1.0 -3.0 -1.0 1.0 30.0 38.0 -1.0 36.0 -10.0 1.0 ... 12.0 14.0 -8.0 -4.0 -4.0 -4.0 -21.0 -29.0 -5.0 0.0
2920 1.0 4.0 4.0 5.0 9.0 -10.0 4.0 1.0 -2.0 -1.0 ... -2.0 -16.0 -3.0 0.0 -3.0 -5.0 -36.0 -90.0 3.0 5.0
2921 -2.0 4.0 2.0 -4.0 12.0 3.0 -2.0 9.0 -8.0 -2.0 ... -10.0 2.0 1.0 0.0 -1.0 -2.0 -30.0 64.0 11.0 5.0

11678 rows × 64 columns

In [7]:
y = pd.get_dummies(df[64])
y
Out[7]:
0 1 2 3
0 1 0 0 0
1 1 0 0 0
2 1 0 0 0
3 1 0 0 0
4 1 0 0 0
... ... ... ... ...
2917 0 0 0 1
2918 0 0 0 1
2919 0 0 0 1
2920 0 0 0 1
2921 0 0 0 1

11678 rows × 4 columns

In [8]:
# Splitting into train, val and test set -- 80-10-10 split

# First, an 80-20 split
Xtrain, Xvaltest, ytrain, yvaltest = train_test_split(X, y, test_size = 0.2)

# Then split the 20% into half
Xval, Xtest, yval, ytest = train_test_split(Xvaltest, yvaltest, test_size = 0.5)

print("Number of samples in...")
print("Training set: ", Xtrain.shape, ytrain.shape)
print("Validation set: ", Xval.shape, yval.shape)
print("Testing set: ", Xtest.shape, ytest.shape)
Number of samples in...
Training set:  (9342, 64) (9342, 4)
Validation set:  (1168, 64) (1168, 4)
Testing set:  (1168, 64) (1168, 4)

Normalization

To obtain mean = 0 and standard deviation = 1

In [9]:
ss_scaler = StandardScaler()

# Fit on training set alone
Xtrain = ss_scaler.fit_transform(Xtrain)

# Use it to transform val and test input
Xval = ss_scaler.transform(Xval)
Xtest = ss_scaler.transform(Xtest)

The model

In [10]:
model = models.Sequential([
    layers.Dense(256, activation = 'relu', input_shape = Xtrain[0].shape),
    layers.Dense(64, activation = 'relu'),
    layers.Dense(16, activation = 'relu'),
    layers.Dense(4, activation = 'softmax')
])

cb = [callbacks.EarlyStopping(patience = 5, restore_best_weights = True)]
In [11]:
model.compile(optimizer=optimizers.Adam(0.0001), loss=losses.CategoricalCrossentropy(), metrics=['accuracy'])

history = model.fit(Xtrain, ytrain, epochs = 256, validation_data = (Xval, yval), callbacks = cb)
Epoch 1/256
292/292 [==============================] - 1s 2ms/step - loss: 1.3245 - accuracy: 0.3382 - val_loss: 1.2286 - val_accuracy: 0.4623
Epoch 2/256
292/292 [==============================] - 1s 2ms/step - loss: 1.1197 - accuracy: 0.5847 - val_loss: 1.0130 - val_accuracy: 0.6644
Epoch 3/256
292/292 [==============================] - 0s 2ms/step - loss: 0.8923 - accuracy: 0.7395 - val_loss: 0.8013 - val_accuracy: 0.7628
Epoch 4/256
292/292 [==============================] - 0s 2ms/step - loss: 0.6977 - accuracy: 0.8102 - val_loss: 0.6389 - val_accuracy: 0.8271
Epoch 5/256
292/292 [==============================] - 1s 2ms/step - loss: 0.5546 - accuracy: 0.8516 - val_loss: 0.5271 - val_accuracy: 0.8579
Epoch 6/256
292/292 [==============================] - 0s 2ms/step - loss: 0.4505 - accuracy: 0.8806 - val_loss: 0.4473 - val_accuracy: 0.8733
Epoch 7/256
292/292 [==============================] - 0s 2ms/step - loss: 0.3746 - accuracy: 0.9032 - val_loss: 0.3874 - val_accuracy: 0.8947
Epoch 8/256
292/292 [==============================] - 0s 2ms/step - loss: 0.3178 - accuracy: 0.9151 - val_loss: 0.3512 - val_accuracy: 0.9041
Epoch 9/256
292/292 [==============================] - 0s 2ms/step - loss: 0.2735 - accuracy: 0.9268 - val_loss: 0.3204 - val_accuracy: 0.9101
Epoch 10/256
292/292 [==============================] - 0s 2ms/step - loss: 0.2382 - accuracy: 0.9377 - val_loss: 0.3024 - val_accuracy: 0.9127
Epoch 11/256
292/292 [==============================] - 1s 2ms/step - loss: 0.2093 - accuracy: 0.9437 - val_loss: 0.2845 - val_accuracy: 0.9178
Epoch 12/256
292/292 [==============================] - 0s 2ms/step - loss: 0.1858 - accuracy: 0.9492 - val_loss: 0.2727 - val_accuracy: 0.9212
Epoch 13/256
292/292 [==============================] - 0s 2ms/step - loss: 0.1654 - accuracy: 0.9555 - val_loss: 0.2636 - val_accuracy: 0.9187
Epoch 14/256
292/292 [==============================] - 0s 2ms/step - loss: 0.1482 - accuracy: 0.9602 - val_loss: 0.2572 - val_accuracy: 0.9212
Epoch 15/256
292/292 [==============================] - 0s 2ms/step - loss: 0.1331 - accuracy: 0.9653 - val_loss: 0.2537 - val_accuracy: 0.9272
Epoch 16/256
292/292 [==============================] - 0s 2ms/step - loss: 0.1202 - accuracy: 0.9695 - val_loss: 0.2485 - val_accuracy: 0.9247
Epoch 17/256
292/292 [==============================] - 0s 2ms/step - loss: 0.1088 - accuracy: 0.9732 - val_loss: 0.2461 - val_accuracy: 0.9272
Epoch 18/256
292/292 [==============================] - 0s 2ms/step - loss: 0.0982 - accuracy: 0.9754 - val_loss: 0.2494 - val_accuracy: 0.9272
Epoch 19/256
292/292 [==============================] - 0s 2ms/step - loss: 0.0884 - accuracy: 0.9794 - val_loss: 0.2439 - val_accuracy: 0.9272
Epoch 20/256
292/292 [==============================] - 1s 2ms/step - loss: 0.0799 - accuracy: 0.9815 - val_loss: 0.2431 - val_accuracy: 0.9324
Epoch 21/256
292/292 [==============================] - 0s 2ms/step - loss: 0.0716 - accuracy: 0.9842 - val_loss: 0.2404 - val_accuracy: 0.9307
Epoch 22/256
292/292 [==============================] - 0s 2ms/step - loss: 0.0648 - accuracy: 0.9863 - val_loss: 0.2459 - val_accuracy: 0.9315
Epoch 23/256
292/292 [==============================] - 0s 2ms/step - loss: 0.0580 - accuracy: 0.9883 - val_loss: 0.2472 - val_accuracy: 0.9272
Epoch 24/256
292/292 [==============================] - 0s 2ms/step - loss: 0.0520 - accuracy: 0.9902 - val_loss: 0.2497 - val_accuracy: 0.9349
Epoch 25/256
292/292 [==============================] - 0s 2ms/step - loss: 0.0464 - accuracy: 0.9924 - val_loss: 0.2535 - val_accuracy: 0.9298
Epoch 26/256
292/292 [==============================] - 0s 2ms/step - loss: 0.0416 - accuracy: 0.9934 - val_loss: 0.2562 - val_accuracy: 0.9358
In [12]:
model.evaluate(Xtest, ytest)
37/37 [==============================] - 0s 1ms/step - loss: 0.2327 - accuracy: 0.9204
Out[12]:
[0.2327262908220291, 0.920376718044281]
In [13]:
cm = confusion_matrix(np.argmax(np.array(ytest), axis = 1), np.argmax(model.predict(Xtest), axis = 1))
cm = cm.astype('int') / cm.sum(axis=1)[:, np.newaxis]

fig = plt.figure(figsize = (5, 5))
ax = fig.add_subplot(111)

for i in range(cm.shape[1]):
    for j in range(cm.shape[0]):
        if cm[i,j] > 0.8:
            clr = "white"
        else:
            clr = "black"
        ax.text(j, i, format(cm[i, j], '.2f'), horizontalalignment="center", color=clr)

_ = ax.imshow(cm, cmap=plt.cm.Blues)
ax.set_xticks(range(4))
ax.set_yticks(range(4))
ax.set_xticklabels(range(4), rotation = 90)
ax.set_yticklabels(range(4))
plt.xlabel('Predicted')
plt.ylabel('True')
plt.show()

Plotting the metrics

In [14]:
def plot(history, variable, variable2):
    plt.plot(range(len(history[variable])), history[variable])
    plt.plot(range(len(history[variable2])), history[variable2])
    plt.legend([variable, variable2])
    plt.title(variable)
In [15]:
plot(history.history, "accuracy", 'val_accuracy')
In [16]:
plot(history.history, "loss", "val_loss")

Predictions

In [17]:
# pick random test data sample from one batch
x = random.randint(0, len(Xtest) - 1)

output = model.predict(Xtest[x].reshape(1, -1))[0]
pred = np.argmax(output)
print("Predicted: ", pred, "(", output[pred], ")")    

print("True: ", np.argmax(np.array(ytest)[x]))
Predicted:  1 ( 0.99952507 )
True:  1

deepC

In [18]:
model.save('muscle_gesture.h5')

!deepCC muscle_gesture.h5
[INFO]
Reading [keras model] 'muscle_gesture.h5'
[SUCCESS]
Saved 'muscle_gesture_deepC/muscle_gesture.onnx'
[INFO]
Reading [onnx model] 'muscle_gesture_deepC/muscle_gesture.onnx'
[INFO]
Model info:
  ir_vesion : 4
  doc       : 
[WARNING]
[ONNX]: terminal (input/output) dense_input's shape is less than 1. Changing it to 1.
[WARNING]
[ONNX]: terminal (input/output) dense_3's shape is less than 1. Changing it to 1.
WARN (GRAPH): found operator node with the same name (dense_3) as io node.
[INFO]
Running DNNC graph sanity check ...
[SUCCESS]
Passed sanity check.
[INFO]
Writing C++ file 'muscle_gesture_deepC/muscle_gesture.cpp'
[INFO]
deepSea model files are ready in 'muscle_gesture_deepC/' 
[RUNNING COMMAND]
g++ -std=c++11 -O3 -fno-rtti -fno-exceptions -I. -I/opt/tljh/user/lib/python3.7/site-packages/deepC-0.13-py3.7-linux-x86_64.egg/deepC/include -isystem /opt/tljh/user/lib/python3.7/site-packages/deepC-0.13-py3.7-linux-x86_64.egg/deepC/packages/eigen-eigen-323c052e1731 "muscle_gesture_deepC/muscle_gesture.cpp" -D_AITS_MAIN -o "muscle_gesture_deepC/muscle_gesture.exe"
[RUNNING COMMAND]
size "muscle_gesture_deepC/muscle_gesture.exe"
   text	   data	    bss	    dec	    hex	filename
 258809	   2584	    760	 262153	  40009	muscle_gesture_deepC/muscle_gesture.exe
[SUCCESS]
Saved model as executable "muscle_gesture_deepC/muscle_gesture.exe"
In [19]:
x = random.randint(0, len(Xtest) - 1)

np.savetxt('sample.data', Xtest[x])    # xth sample into text file

# run exe with input
!muscle_gesture_deepC/muscle_gesture.exe sample.data

# show predicted output
nn_out = np.loadtxt('deepSea_result_1.out')

pred = np.argmax(nn_out)
print("Predicted: ", pred, "(", nn_out[pred], ")")    

print("True: ", np.argmax(np.array(ytest)[x]))
writing file deepSea_result_1.out.
Predicted:  0 ( 1.0 )
True:  2
In [ ]: