Cainvas
Model Files
vww_custom_45.h5
keras
Model
deepSea Compiled Models
vww_custom_45.exe
deepSea
Ubuntu

Visual Wake word detection

Credit: AITS Cainvas Community

Photo by LCR on Dribbble

Visual wake word detection is the classification of images into 2 classes - with person(s) or without. Just as audio wake word systems respond to a specific phrase, visual wake word systems respond to the presence of humans in the frame.

In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import tensorflow as tf
from keras import layers, callbacks, optimizers
import keras 
import os
import random
from PIL import Image

Dataset

The dataset is derived from COCO 2017 and reduced to 100mb using the script here.

In [2]:
!wget -N https://cainvas-static.s3.amazonaws.com/media/user_data/cainvas-admin/vww.zip

!unzip -qo vww.zip  

!rm vww.zip
--2021-06-02 05:26:02--  https://cainvas-static.s3.amazonaws.com/media/user_data/cainvas-admin/vww.zip
Resolving cainvas-static.s3.amazonaws.com (cainvas-static.s3.amazonaws.com)... 52.219.156.19
Connecting to cainvas-static.s3.amazonaws.com (cainvas-static.s3.amazonaws.com)|52.219.156.19|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 95042991 (91M) [application/zip]
Saving to: ‘vww.zip’

vww.zip             100%[===================>]  90.64M  76.4MB/s    in 1.2s    

2021-06-02 05:26:04 (76.4 MB/s) - ‘vww.zip’ saved [95042991/95042991]

The dataset folder has two sub-folders - person and notperson containing images of respective types.

In [3]:
data_dir = 'vww/'

print("Number of samples")
for f in os.listdir(data_dir + '/'):
    if os.path.isdir(data_dir + '/' + f):
        print(f, " : ", len(os.listdir(data_dir + '/' + f +'/')))
Number of samples
notperson  :  300
person  :  300

It is a balanced dataset.

In [4]:
batch_size = 64

print("Training set")
train_ds = tf.keras.preprocessing.image_dataset_from_directory(
  data_dir,
  image_size=(96, 96),
  validation_split=0.2,
  subset="training",
  seed=113, 
  batch_size=batch_size)

print("Validation set")
val_ds = tf.keras.preprocessing.image_dataset_from_directory(
  data_dir,
  image_size=(96, 96),
  validation_split=0.2,
  subset="validation",
  seed=113, 
  batch_size=batch_size)
Training set
Found 600 files belonging to 2 classes.
Using 480 files for training.
Validation set
Found 600 files belonging to 2 classes.
Using 120 files for validation.

Define the class_names for use later.

In [5]:
class_names = train_ds.class_names
print(class_names)
['notperson', 'person']

Visualization

In [6]:
num_samples = 4    # the number of samples to be displayed in each class

for x in class_names:
    plt.figure(figsize=(10, 10))

    filenames = os.listdir(data_dir + x)

    for i in range(num_samples):
        ax = plt.subplot(1, num_samples, i + 1)
        img = Image.open(data_dir + x + '/' + filenames[i])
        plt.imshow(img)
        plt.title(x)
        plt.axis("off")

Preprocessing

Defining the input shape

In [7]:
print("Shape of one training batch")

for image_batch, labels_batch in train_ds:
    input_shape = image_batch[0].shape
    print("Input: ", image_batch.shape)
    print("Labels: ", labels_batch.shape)
    break
Shape of one training batch
Input:  (64, 96, 96, 3)
Labels:  (64,)

Normalizing the pixel values

Pixel values are now integers between 0 and 255. Changing them to the range [0, 1] for faster convergence.

In [8]:
# Normalizing the pixel values

normalization_layer = layers.experimental.preprocessing.Rescaling(1./255)

train_ds = train_ds.map(lambda x, y: (normalization_layer(x), y))
val_ds = val_ds.map(lambda x, y: (normalization_layer(x), y))

The model

Transfer learning

In [9]:
base_model = keras.applications.MobileNetV2(weights= 'imagenet', input_shape=input_shape, include_top=False)    # False, do not include the classification layer of the model

base_model.trainable = False

inputs = tf.keras.Input(shape=input_shape)

x = base_model(inputs, training=False)
x = keras.layers.GlobalAveragePooling2D()(x)
outputs = keras.layers.Dense(len(class_names), activation = 'softmax')(x)    # Add own classififcation layer

model = keras.Model(inputs, outputs)

cb = [callbacks.EarlyStopping(monitor = 'val_loss', patience = 5, restore_best_weights = True)]
In [10]:
model.compile(loss='sparse_categorical_crossentropy', optimizer=optimizers.Adam(0.01), metrics=['accuracy'])

history = model.fit(train_ds, validation_data =  val_ds, epochs=32, callbacks = cb)
Epoch 1/32
8/8 [==============================] - 2s 196ms/step - loss: 1.6694 - accuracy: 0.5917 - val_loss: 0.9845 - val_accuracy: 0.7167
Epoch 2/32
8/8 [==============================] - 1s 71ms/step - loss: 0.5857 - accuracy: 0.8229 - val_loss: 0.9113 - val_accuracy: 0.7833
Epoch 3/32
8/8 [==============================] - 1s 78ms/step - loss: 0.3307 - accuracy: 0.8833 - val_loss: 0.6281 - val_accuracy: 0.8750
Epoch 4/32
8/8 [==============================] - 1s 80ms/step - loss: 0.2450 - accuracy: 0.9229 - val_loss: 0.5793 - val_accuracy: 0.8167
Epoch 5/32
8/8 [==============================] - 1s 81ms/step - loss: 0.1383 - accuracy: 0.9458 - val_loss: 0.4683 - val_accuracy: 0.9000
Epoch 6/32
8/8 [==============================] - 1s 78ms/step - loss: 0.0449 - accuracy: 0.9854 - val_loss: 0.4410 - val_accuracy: 0.8500
Epoch 7/32
8/8 [==============================] - 1s 81ms/step - loss: 0.0272 - accuracy: 0.9958 - val_loss: 0.4304 - val_accuracy: 0.8750
Epoch 8/32
8/8 [==============================] - 1s 80ms/step - loss: 0.0190 - accuracy: 0.9979 - val_loss: 0.4271 - val_accuracy: 0.8583
Epoch 9/32
8/8 [==============================] - 1s 72ms/step - loss: 0.0143 - accuracy: 1.0000 - val_loss: 0.4325 - val_accuracy: 0.8667
Epoch 10/32
8/8 [==============================] - 1s 71ms/step - loss: 0.0100 - accuracy: 1.0000 - val_loss: 0.4402 - val_accuracy: 0.8750
Epoch 11/32
8/8 [==============================] - 1s 69ms/step - loss: 0.0095 - accuracy: 1.0000 - val_loss: 0.4399 - val_accuracy: 0.8833
Epoch 12/32
8/8 [==============================] - 1s 75ms/step - loss: 0.0080 - accuracy: 1.0000 - val_loss: 0.4417 - val_accuracy: 0.8667
Epoch 13/32
8/8 [==============================] - 1s 78ms/step - loss: 0.0073 - accuracy: 1.0000 - val_loss: 0.4442 - val_accuracy: 0.8917
In [11]:
model.evaluate(val_ds)
2/2 [==============================] - 0s 8ms/step - loss: 0.4271 - accuracy: 0.8583
Out[11]:
[0.4271244406700134, 0.8583333492279053]

Plotting the metrics

In [12]:
def plot(history, variable, variable2):
    plt.plot(range(len(history[variable])), history[variable])
    plt.plot(range(len(history[variable2])), history[variable2])
    plt.title(variable)
In [13]:
plot(history.history, "accuracy", 'val_accuracy')
In [14]:
plot(history.history, "loss", "val_loss")

Prediction

In [15]:
# pick random test data sample from one batch
x = random.randint(0, batch_size - 1)

for i in val_ds.as_numpy_iterator():
    img, label = i    
    plt.axis('off')   # remove axes
    plt.imshow(img[x])    # shape from (64, 256, 256, 3) --> (256, 256, 3)
    output = model.predict(np.expand_dims(img[x],0))    # getting output; input shape (256, 256, 3) --> (1, 256, 256, 3)
    pred = np.argmax(output[0])    # finding max
    print("Predicted: ", class_names[pred])    # Picking the label from class_names base don the model output
    print("True: ", class_names[label[x]])
    print("Probability: ", output[0][pred])
    break
Predicted:  notperson
True:  notperson
Probability:  0.6090462

deepC

In [16]:
model.save('visual_wake_word.h5')
In [17]:
!deepCC visual_wake_word.h5
[INFO]
Reading [keras model] 'visual_wake_word.h5'
[SUCCESS]
Saved 'visual_wake_word_deepC/visual_wake_word.onnx'
[INFO]
Reading [onnx model] 'visual_wake_word_deepC/visual_wake_word.onnx'
[INFO]
Model info:
  ir_vesion : 5
  doc       : 
[WARNING]
[ONNX]: graph-node Conv1's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node expanded_conv_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node expanded_conv_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_1_expand's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_1_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_1_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_2_expand's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_2_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_2_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_3_expand's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_3_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_3_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_4_expand's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_4_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_4_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_5_expand's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_5_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_5_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_6_expand's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_6_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_6_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_7_expand's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_7_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_7_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_8_expand's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_8_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_8_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_9_expand's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_9_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_9_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_10_expand's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_10_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_10_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_11_expand's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_11_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_11_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_12_expand's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_12_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_12_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_13_expand's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_13_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_13_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_14_expand's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_14_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_14_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_15_expand's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_15_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_15_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_16_expand's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_16_depthwise's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: graph-node block_16_project's attribute auto_pad has no meaningful data.
[WARNING]
[ONNX]: terminal (input/output) input_2's shape is less than 1. Changing it to 1.
[WARNING]
[ONNX]: terminal (input/output) dense's shape is less than 1. Changing it to 1.
WARN (GRAPH): found operator node with the same name (dense) as io node.
[INFO]
Running DNNC graph sanity check ...
[SUCCESS]
Passed sanity check.
[INFO]
Writing C++ file 'visual_wake_word_deepC/visual_wake_word.cpp'
[INFO]
deepSea model files are ready in 'visual_wake_word_deepC/' 
[RUNNING COMMAND]
g++ -std=c++11 -O3 -fno-rtti -fno-exceptions -I. -I/opt/tljh/user/lib/python3.7/site-packages/deepC-0.13-py3.7-linux-x86_64.egg/deepC/include -isystem /opt/tljh/user/lib/python3.7/site-packages/deepC-0.13-py3.7-linux-x86_64.egg/deepC/packages/eigen-eigen-323c052e1731 "visual_wake_word_deepC/visual_wake_word.cpp" -D_AITS_MAIN -o "visual_wake_word_deepC/visual_wake_word.exe"
[RUNNING COMMAND]
size "visual_wake_word_deepC/visual_wake_word.exe"
   text	   data	    bss	    dec	    hex	filename
9166569	   3768	    760	9171097	 8bf099	visual_wake_word_deepC/visual_wake_word.exe
[SUCCESS]
Saved model as executable "visual_wake_word_deepC/visual_wake_word.exe"
In [ ]: