Gesture-Detecting-Macro-Keyboard - Glorified Bluetooth macro keyboard with machine learning (TensorFlow Lite for Microcontrollers) running on an ESP32.

Gesture detection

tldr; Glorified Bluetooth macro keyboard with machine learning (TensorFlow Lite for Microcontrollers) running on an ESP32.

Main features

  • Capacitive touch pad for drawing gestures on that maps to actions on the computer.
  • Capacitive touch bar for volume control.
  • 6 Buttons for generic control.
  • Optional AutoHotkey for programatical control exactly what shall happen when keys are pressed or a gesture is drawn.
  • Small OLED display for various info such as crypto prices, remote Bluetooth status, local IP and gesture prediction debug info.
  • BLE HID to connect to a computer and act as a keyboard.
  • Such very RGB.

training training training

How it works

The switch presses and the touchpad gestures will end up being sent as a keyboard shortcut. This CAN be enough for some use cases, but it gets really powerful when coupled together with something AutoHotkey scripts running on the computer.

Current AutoHotkey script works as following

On the computer focus one window and then hold down one of SWITCH_1-5 for 2s. This will connect this key to that specific window. When clicking the key it will focus window if not in focus, and if in focus minimize it.

Gestures are mapped as following right now

Gesture Action How to remember
ARROW_RIGHT Media key next track > as in next
ARROW_UP git push current branch ĘŚ as in away/push
C_GESTURE Display all the COM ports on the computer as a notification 'C' as in COM Port
LINE_DOWN_GESTURE git log 'l' as in log
LINE_HORIZONTAL_GESTURE git diff
S_GESTURE git status 'S' as in status
ARROW_DOWN git fetch and git pull current branch 'V' as in pull
... whatever ...

Note the commands involving git above will also focus the git terminal window before executing the action. For this to work SWITCH_2 is expected to be mapped with a git terminal window. For more detailed info about the actions look in hotkeys/hotkey_handler.ahk

SWITCH_6

Is special key and used for enable Bluetooth pairing (hold down for 5s) and moving through the different "menus" on the OLED screen.

Compiling ESP32 code

Follow instruction on https://github.com/espressif/esp-idf to set up the esp-idf, then just run idf.py build or use the VSCode extension.

Tensorflow gesture/shape detection

Tensorflow Lite is used to recognize what is being drawn on the touchpad. There are already some shapes/gestures training data collected by me and is found in the /training/train_data folder and a pre trained model using those is what is being used by default. The trackpad outputs x, y coordinates in the range [0,1792], coordinates collected between touch start and touch release are then converted into a 28x28 2D matrix which is used as input to TensorFlow.

Example how simple it is to add a new gesture training Higher res video here

Gather train data

First you need to enter training mode on the macro keybord by holding down SWITCH_6 while plugging in power. In training mode the device pretty much just pipes the raw data from the trackpad over UART as a string with this format: [x1, y1, x2, y2,...]\n. Then go to the /training folder. For each new gesture collect_train_data.py needs to run twice, first to collect train data and then to collect test data. For current number of gestures (7) about 150 train samples and about 45 test samples seems to be enough to get very good predictions.

python collect_train_data.py --port COMX --gesture_name v_shape
python collect_train_data.py --port COMX --gesture_name v_shape --test_data

After each drawn gesture a plot of the input data will be shown, if it looks bad because you made a misstake, then just press ENTER in the console and the last sample will be removed from the data. Press ctrl+c when done.

Training the model

When collection of data is finished run python train_model.py --center_gesture it will take about 30s depending on computer. It will pick up all train data in the train_data folder, so remove any data for unwanted gestures. This script will generate a gesture_model_tflite.c and gesture_model_tflite.h files containing the TF Lite model that can be imported by embedded TF Lite. The gesture_model_tflite.h file is also generated from the Python code and contains various information such as the map between predictions (int) and their meaning, if --center_gesture was used when training, etc. All output from the train_model.py will be put in a folder training/output. When compiling the embedded code the generated .c and .h files are automatically copied into the C code directory so latest model will always be compiled.

--center_gesture

Option to center the gesture in the input matrix, this will help when the training data is drawn at a specific location on the trackpad, but you want it to be recognized anywhere. This option will remove functionality to have for example two line down gestures, one at the right side and one at the left side of the trackpad as this option will move both gestures to the center. For this to work the same change is needed on target before inputing the drawn gesture to the embedded model. This is handled automatically as this is added into the gesture_model_tflite.h file and handled accordingly in embedded code.

Study the graphs to see how well the model performed on test data. training

Testing/verifying the model

Before moving the model into the microcontroller it's a good idea to test it first. First make sure the device is in training mode as described before, then test the model by running python test_model.py --port COMX --center_gesture and watch the predictions show as you draw on the trackpad.

AutoHotkey setup

  • Install Auto Hotkey from https://www.autohotkey.com/
  • Right click on the hotkeys/hotkey_handler.ahk -> compile
  • Now you will get an .exe file, right click on this and create a shortcut
  • To auto start with Windows type Windows key + r
  • Run shell:startup and paste the hotkey_handler.exe - Shortcut into the opened folder.
  • Done

CAD model

Full Fusion 360 project is found in CAD folder.

Trill sensors

Bought from their Kickstarter in 2019, but can be bought today from here now

Similar Resources

PoC capable of detecting manual syscalls from usermode.

syscall-detect PoC capable of detecting manual syscalls from usermode. More information available at: https://winternl.com/detecting-manual-syscalls-f

Sep 23, 2022

flashrom is a utility for detecting, reading, writing, verifying and erasing flash chips

flashrom is a utility for detecting, reading, writing, verifying and erasing flash chips

Oct 2, 2022

A tool for detecting manual/direct syscalls in x86 and x64 processes using Nirvana Hooks.

A tool for detecting manual/direct syscalls in x86 and x64 processes using Nirvana Hooks.

manual-syscall-detect A tool for detecting manual/direct syscalls in x86 and x64 processes using Nirvana Hooks. Description A full write-up of this to

Sep 9, 2022

Tool for detecting violations of ordering axioms in STL comparators

SortChecker++ is an extension of SortChecker tool to C++ sorting APIs like std::sort or std::binary_search. It verifies that comparators satisfy the S

Aug 13, 2022

Arduino-controlled bed that helps in reducing rate of disease infection by detecting whether a person accessed the safe space of a subject who is infected

Arduino-controlled bed that helps in reducing rate of disease infection by detecting whether a person accessed the safe space of a subject who is infected

Infection Control Bed BACKGROUND Spread of COVID-19 occurs via airborne parricels and droplets. People who are infected with COVID an release particle

Mar 17, 2022

An experiment of running a lot of wasm apps on esp32 M5 atom matrix board

Wasms This is a POC project to run many wasm apps on a single esp32 board (in my case, M5 Atom Matrix, with a nice neopixel 5x5 display matrix). All I

Sep 21, 2022

split89 keyboard - a 3d printed 89 key split TKL keyboard base powered by ATmega32U4 Pro Micro controllers with QMK Configurator support.

 split89 keyboard - a 3d printed 89 key split TKL keyboard base powered by ATmega32U4 Pro Micro controllers with QMK Configurator support.

split89 keyboard - a 3d printed 89 key split TKL keyboard base powered by ATmega32U4 Pro Micro controllers with QMK Configurator support. This keyboar

Aug 7, 2022

Raw HID keyboard forwarder to turn the Pi 400 into a USB keyboard

Raspberry Pi 400 as a USB HID Keyboard Hook your Pi 400 up to your PC somehow, using a USB Type-C cable into the power port. Anker make good ones- I u

Sep 28, 2022

Use Atari keyboard as USB keyboard with a Teensy 3.6

Use Atari keyboard as USB keyboard with a Teensy 3.6

Atari Keyboard Convert an Atari 600/800/1200 XL into a USB keyboard. I bricked my Atari mainboard. My goal is to use the keyboard of the Atari on a Ra

Dec 3, 2021
Exploits the Wii U's bluetooth stack to gain IOSU kernel access via bluetooth.

BluuBomb Exploits the Wii U's bluetooth stack to gain IOSU kernel access via bluetooth. For a more detailed write-up see WRITEUP.md. Not to be confuse

Sep 10, 2022
Bluetooth Joystick : A wireless joystick with ESP-32 microcontroller and Dual Axis Joystick Module using the Bluetooth connectivity.
Bluetooth Joystick : A wireless joystick with ESP-32 microcontroller and Dual Axis Joystick Module using the Bluetooth connectivity.

BluetoothJoystick Bluetooth Joystick : A wireless joystick with ESP-32 microcontroller and Dual Axis Joystick Module using the Bluetooth connectivity.

Feb 24, 2022
The littlest Bluetooth macro pad

The littlest wireless macro pad! AWWWWW!!! (c) 2021 Zack Freedman and Voidstar Lab. Licensed CC-BY-NC (credit me and don't sell it) Built partially on

Feb 26, 2022
Arduino Sketch and a Web Bluetooth API for loading models and running inference on the Nano Sense 33 BLE device.
Arduino Sketch and a Web Bluetooth API for loading models and running inference on the Nano Sense 33 BLE device.

TF4Micro Motion Kit This repo contains the Arduino Sketch and a Web Bluetooth API for loading models and running inference on the device. Install and

Sep 1, 2022
My humble attempt at getting tensorflow and the ESP32-CAM to cooperate

ESP32-Object-Recognition My humble attempt at getting tensorflow and the ESP32-CAM to cooperate (among other things) Notes: "Webcam_detection.py" isn'

Jun 21, 2022
ESP32-Skid-Steer - Bruder Catepillar Skid Steer model converted to RC, controlled by an ESP32 with 2 analog joysticks and a receiver that is an ESP32 on the model.
ESP32-Skid-Steer - Bruder Catepillar Skid Steer model converted to RC, controlled by an ESP32 with 2 analog joysticks and a receiver that is an ESP32 on the model.

ESP32-Skid-Steer Bruder Catepillar Skid Steer model converted to RC, controlled by an ESP32 with 2 analog joysticks and a receiver that is an ESP32 on

Feb 4, 2022
Bluetooth low energy (BLE) tracker for ESP32

BLEcker Bluetooth low energy (BLE) tracker for ESP32 This software is written for ESP32 boards to track BLE devices. It can be used for your smart hom

Aug 17, 2022
A handy little system information monitor using and ESP32 + ILI9488 TFT. Receives data over Serial Bluetooth.
A handy little system information monitor using and ESP32 + ILI9488 TFT. Receives data over Serial Bluetooth.

Bluetooth-System-Monitor A handy little system information monitor using and ESP32 + ILI9488 TFT. Receives data over Serial Bluetooth and thus giving

Sep 19, 2022
Bluetooth Monitor port for the ESP32
Bluetooth Monitor port for the ESP32

ESP32 BT Monitor What is it? This is a (at the moment partial) port of andrewjfreyer/monitor for the popular and super cheap ESP32 boards. For a more

Jul 20, 2022
Bluetooth Gateway for Phantom Remote Control based on ESP32
Bluetooth Gateway for Phantom Remote Control based on ESP32

Phantom remote control Bluetooth gateway An ESP3232 firmware for the gateway of Phantom remote control, which can push the temperature and humidity data of Phantom remote control through LAN, and also support to control Phantom remote control to send and receive IR data through LAN.

Jul 9, 2022