Cainvas
Model Files
confused.h5
keras
Model
deepSea Compiled Models
confused.exe
deepSea
Ubuntu

Confusion detection with EEG signals

Credit: AITS Cainvas Community

Photo by George Vald on Dribbble

Detecting whether a person is confused or not based on the EEG recordings.

In [1]:
import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler, StandardScaler
from sklearn.model_selection import train_test_split
from keras import layers, models, optimizers, losses, callbacks
from sklearn.metrics import accuracy_score, confusion_matrix, f1_score
import matplotlib.pyplot as plt
import random

The dataset

On Kaggle by Haohan Wang

Wang, H., Li, Y., Hu, X., Yang, Y., Meng, Z., & Chang, K. M. (2013, June). Using EEG to Improve Massive Open Online Courses Feedback Interaction. In AIED Workshops. PDF

EEG signal data was collected from 10 college students while watching MOOC video clips of subjects ranging from simple ones like basic algebra or geometry to Stem Cell research and Quantum Mechanics that can be confusing if we are not familiar with the topic. There were 20 videos, 10 simple ones and 10 complex, each 2 minute long. The clips were copped in the middle of a topic to make it more confusing.

The students wore a single-channel wireless MindSet that measured activity over the frontal lobe. The MindSet measures the voltage between an electrode resting on the forehead and two electrodes (one ground and one reference) each in contact with an ear.

There are two label columns - user-defined label (self labelled by the students based on their experience) and predefined label (where they are expected to be confused).

In [2]:
df = pd.read_csv('https://cainvas-static.s3.amazonaws.com/media/user_data/cainvas-admin/EEG_data.csv')
df
Out[2]:
SubjectID VideoID Attention Mediation Raw Delta Theta Alpha1 Alpha2 Beta1 Beta2 Gamma1 Gamma2 predefinedlabel user-definedlabeln
0 0.0 0.0 56.0 43.0 278.0 301963.0 90612.0 33735.0 23991.0 27946.0 45097.0 33228.0 8293.0 0.0 0.0
1 0.0 0.0 40.0 35.0 -50.0 73787.0 28083.0 1439.0 2240.0 2746.0 3687.0 5293.0 2740.0 0.0 0.0
2 0.0 0.0 47.0 48.0 101.0 758353.0 383745.0 201999.0 62107.0 36293.0 130536.0 57243.0 25354.0 0.0 0.0
3 0.0 0.0 47.0 57.0 -5.0 2012240.0 129350.0 61236.0 17084.0 11488.0 62462.0 49960.0 33932.0 0.0 0.0
4 0.0 0.0 44.0 53.0 -8.0 1005145.0 354328.0 37102.0 88881.0 45307.0 99603.0 44790.0 29749.0 0.0 0.0
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
12806 9.0 9.0 64.0 38.0 -39.0 127574.0 9951.0 709.0 21732.0 3872.0 39728.0 2598.0 960.0 1.0 0.0
12807 9.0 9.0 61.0 35.0 -275.0 323061.0 797464.0 153171.0 145805.0 39829.0 571280.0 36574.0 10010.0 1.0 0.0
12808 9.0 9.0 60.0 29.0 -426.0 680989.0 154296.0 40068.0 39122.0 10966.0 26975.0 20427.0 2024.0 1.0 0.0
12809 9.0 9.0 60.0 29.0 -84.0 366269.0 27346.0 11444.0 9932.0 1939.0 3283.0 12323.0 1764.0 1.0 0.0
12810 9.0 9.0 64.0 29.0 -49.0 1164555.0 1184366.0 50014.0 124208.0 10634.0 445383.0 22133.0 4482.0 1.0 0.0

12811 rows × 15 columns

Preprocessing

Time based dataframe

Since this is a time based dataset, the features are appended to include values for previous timesteps of the same subject watching the same video.

In [3]:
# Defining the time window, that is, how many timesteps to include
time_window = 5

# Dataframes that hold rows grouped by subject
df_subject_grouped = df.groupby('SubjectID')

# Column values affected by time
time_affected_columns = list(df.columns)
time_affected_columns.remove('SubjectID')
time_affected_columns.remove('VideoID')
time_affected_columns.remove('predefinedlabel')
time_affected_columns.remove('user-definedlabeln')


# Final dataframe
df_final = pd.DataFrame()

# For each subject
for subject in df_subject_grouped:
    # For each video:
    for video in subject[1].groupby('VideoID'):
        # If the df has timesteps greater than or equal to the time window, else discard
        if time_window <= len(video[1]):
            # Skipping time_window-1 rows from the beginning, and looping to till the end 
            for row_num in range(time_window, len(video[1])+1):
                # picking the time_window th row
                df_temp = video[1].iloc[row_num-1, :]
                # Appending values from time_window-1 rows before that
                for i in range(time_window-1):                
                    df_temp_i = video[1].iloc[row_num-1-i][time_affected_columns]    # Pick necessary columns                
                    df_temp = pd.concat([df_temp, df_temp_i], axis = 0)    # Append values

                df_temp = df_temp.to_frame().transpose()    # Series to DataFrame

                df_final = pd.concat([df_final, df_temp])    # Add as row to final dataframe

# Reset index            
df_final = df_final.reset_index(drop = True)

Dropping unwanted columns

SubjectID and VideoID should not influence the final results and hence are removed. User defined labels are more reliable in assessing the level of confusion rather than the predefined labels.

In [4]:
df = df_final.drop(columns = ['SubjectID', 'VideoID', 'predefinedlabel'])
In [5]:
df['user-definedlabeln'].value_counts()
Out[5]:
1.0    6363
0.0    6048
Name: user-definedlabeln, dtype: int64

This is an almost balanced dataset.

Train-val-test split

In [6]:
# Splitting into train, val and test set -- 80-10-10 split

# First, an 80-20 split
train_df, val_test_df = train_test_split(df, test_size = 0.2, random_state = 113)

# Then split the 20% into half
val_df, test_df = train_test_split(val_test_df, test_size = 0.5, random_state = 113)

len(train_df), len(val_df), len(test_df)
Out[6]:
(9928, 1241, 1242)
In [7]:
ic = df.columns.tolist()
ic.remove('user-definedlabeln')

oc = ['user-definedlabeln']

ytrain = train_df[oc]
Xtrain = train_df.drop(columns = oc)

yval = val_df[oc]
Xval = val_df.drop(columns = oc)

ytest = test_df[oc]
Xtest = test_df.drop(columns = oc)

Standardization

Scaling the values to have mean = 0 and standard deviation = 1.

In [8]:
ss = StandardScaler()

Xtrain = ss.fit_transform(Xtrain)
Xval = ss.transform(Xval)
Xtest = ss.transform(Xtest)

The model

In [9]:
model = models.Sequential([
    layers.Dense(32, activation = 'relu', input_shape = Xtrain[0].shape),
    layers.Dropout(0.2),
    layers.Dense(16, activation = 'relu'),
    layers.Dense(1, activation = 'sigmoid')
])

cb = callbacks.EarlyStopping(patience = 5, restore_best_weights = True)
In [10]:
model.compile(optimizer = optimizers.Adam(0.01), loss = losses.BinaryCrossentropy(), metrics = ['accuracy'])

history = model.fit(Xtrain, ytrain, validation_data = (Xval, yval), epochs = 256, callbacks = cb)
Epoch 1/256
311/311 [==============================] - 1s 2ms/step - loss: 0.6603 - accuracy: 0.6264 - val_loss: 0.6514 - val_accuracy: 0.6326
Epoch 2/256
311/311 [==============================] - 1s 2ms/step - loss: 0.6401 - accuracy: 0.6477 - val_loss: 0.6421 - val_accuracy: 0.6398
Epoch 3/256
311/311 [==============================] - 1s 2ms/step - loss: 0.6349 - accuracy: 0.6561 - val_loss: 0.6444 - val_accuracy: 0.6358
Epoch 4/256
311/311 [==============================] - 0s 2ms/step - loss: 0.6272 - accuracy: 0.6657 - val_loss: 0.6327 - val_accuracy: 0.6382
Epoch 5/256
311/311 [==============================] - 1s 2ms/step - loss: 0.6253 - accuracy: 0.6615 - val_loss: 0.6410 - val_accuracy: 0.6495
Epoch 6/256
311/311 [==============================] - 1s 2ms/step - loss: 0.6210 - accuracy: 0.6642 - val_loss: 0.6472 - val_accuracy: 0.6511
Epoch 7/256
311/311 [==============================] - 1s 2ms/step - loss: 0.6188 - accuracy: 0.6661 - val_loss: 0.6392 - val_accuracy: 0.6422
Epoch 8/256
311/311 [==============================] - 1s 2ms/step - loss: 0.6105 - accuracy: 0.6687 - val_loss: 0.6324 - val_accuracy: 0.6551
Epoch 9/256
311/311 [==============================] - 0s 2ms/step - loss: 0.6090 - accuracy: 0.6703 - val_loss: 0.6338 - val_accuracy: 0.6640
Epoch 10/256
311/311 [==============================] - 1s 2ms/step - loss: 0.6089 - accuracy: 0.6713 - val_loss: 0.6363 - val_accuracy: 0.6600
Epoch 11/256
311/311 [==============================] - 1s 2ms/step - loss: 0.6049 - accuracy: 0.6772 - val_loss: 0.6313 - val_accuracy: 0.6608
Epoch 12/256
311/311 [==============================] - 0s 2ms/step - loss: 0.6042 - accuracy: 0.6717 - val_loss: 0.6292 - val_accuracy: 0.6680
Epoch 13/256
311/311 [==============================] - 1s 2ms/step - loss: 0.5985 - accuracy: 0.6796 - val_loss: 0.6296 - val_accuracy: 0.6616
Epoch 14/256
311/311 [==============================] - 1s 2ms/step - loss: 0.5977 - accuracy: 0.6790 - val_loss: 0.6213 - val_accuracy: 0.6600
Epoch 15/256
311/311 [==============================] - 0s 2ms/step - loss: 0.5968 - accuracy: 0.6824 - val_loss: 0.6308 - val_accuracy: 0.6575
Epoch 16/256
311/311 [==============================] - 0s 2ms/step - loss: 0.5932 - accuracy: 0.6802 - val_loss: 0.6221 - val_accuracy: 0.6591
Epoch 17/256
311/311 [==============================] - 1s 2ms/step - loss: 0.5915 - accuracy: 0.6884 - val_loss: 0.6215 - val_accuracy: 0.6656
Epoch 18/256
311/311 [==============================] - 1s 2ms/step - loss: 0.5943 - accuracy: 0.6824 - val_loss: 0.6342 - val_accuracy: 0.6616
Epoch 19/256
311/311 [==============================] - 1s 2ms/step - loss: 0.5941 - accuracy: 0.6814 - val_loss: 0.6280 - val_accuracy: 0.6672
In [11]:
model.evaluate(Xtest, ytest)
39/39 [==============================] - 0s 1ms/step - loss: 0.6242 - accuracy: 0.6498
Out[11]:
[0.6241828203201294, 0.6497584581375122]
In [12]:
cm = confusion_matrix(ytest, (model.predict(Xtest)>0.5).astype('int'))
cm = cm.astype('int') / cm.sum(axis=1)[:, np.newaxis]

fig = plt.figure(figsize = (10, 10))
ax = fig.add_subplot(111)

for i in range(cm.shape[1]):
    for j in range(cm.shape[0]):
        if cm[i,j] > 0.8:
            clr = "white"
        else:
            clr = "black"
        ax.text(j, i, format(cm[i, j], '.2f'), horizontalalignment="center", color=clr)

_ = ax.imshow(cm, cmap=plt.cm.Blues)
ax.set_xticks(range(2))
ax.set_yticks(range(2))
ax.set_xticklabels(range(2), rotation = 90)
ax.set_yticklabels(range(2))
plt.xlabel('Predicted')
plt.ylabel('True')
plt.show()

The low accuracy rate may be increased with better labelled data. Self labelled data indicting mental state is easy to be mislabelled.

Plotting the metrics

In [13]:
def plot(history, variable, variable2):
    plt.plot(range(len(history[variable])), history[variable])
    plt.plot(range(len(history[variable2])), history[variable2])
    plt.title(variable)
In [14]:
plot(history.history, "accuracy", 'val_accuracy')
In [15]:
plot(history.history, "loss", "val_loss")

Prediction

In [16]:
# pick random test data sample from one batch
x = random.randint(0, len(Xtest) - 1)

output = model.predict(Xtest[x].reshape(1, -1))[0][0] 
pred = (output>0.5).astype('int')
print("Predicted: ", pred, "(", output, "-->", pred, ")")    

print("True: ", np.array(ytest)[x][0])
Predicted:  1 ( 0.69753677 --> 1 )
True:  0.0

deepC

In [17]:
model.save('confused.h5')

!deepCC confused.h5
[INFO]
Reading [keras model] 'confused.h5'
[SUCCESS]
Saved 'confused.onnx'
[INFO]
Reading [onnx model] 'confused.onnx'
[INFO]
Model info:
  ir_vesion : 4
  doc       : 
[WARNING]
[ONNX]: terminal (input/output) dense_input's shape is less than 1. Changing it to 1.
[WARNING]
[ONNX]: terminal (input/output) dense_2's shape is less than 1. Changing it to 1.
WARN (GRAPH): found operator node with the same name (dense_2) as io node.
[INFO]
Running DNNC graph sanity check ...
[SUCCESS]
Passed sanity check.
[INFO]
Writing C++ file 'confused_deepC/confused.cpp'
[INFO]
deepSea model files are ready in 'confused_deepC/' 
[RUNNING COMMAND]
g++ -std=c++11 -O3 -fno-rtti -fno-exceptions -I. -I/opt/tljh/user/lib/python3.7/site-packages/deepC-0.13-py3.7-linux-x86_64.egg/deepC/include -isystem /opt/tljh/user/lib/python3.7/site-packages/deepC-0.13-py3.7-linux-x86_64.egg/deepC/packages/eigen-eigen-323c052e1731 confused_deepC/confused.cpp -o confused_deepC/confused.exe
[RUNNING COMMAND]
size "confused_deepC/confused.exe"
   text	   data	    bss	    dec	    hex	filename
 117618	  12384	    760	 130762	  1feca	confused_deepC/confused.exe
[SUCCESS]
Saved model as executable "confused_deepC/confused.exe"
In [18]:
# pick random test data sample from one batch
x = random.randint(0, len(Xtest) - 1)

np.savetxt('sample.data', Xtest[x])    # xth sample into text file

# run exe with input
!confused_deepC/confused.exe sample.data

# show predicted output
nn_out = np.loadtxt('deepSea_result_1.out')

pred = (nn_out>0.5).astype('int')
print("Predicted: ", pred, "(", nn_out, "-->", pred, ")")    

print("True: ", np.array(ytest)[x][0])
reading file sample.data.
writing file deepSea_result_1.out.
Predicted:  1 ( 0.548143 --> 1 )
True:  1.0