Fast AI models on Espruino?

[gfwilliams]

Someone just showed me a really interesting project called ML4F - this basically allows you to compile a Tensorflow AI model down to native ARM Thumb assembly code.

So in theory you could:

• Use Teachable Machine to train an AI model to detect something (let's say words from audio)
• Download the model in 'Tensorflow.js' format
• Use https://github.com/microsoft/ml4f to convert it to ARM assembly
• Convert that to base64, and upload it wrapped with a E.nativeCall function
• Feed the waveform data recorded on an Espruino device into it

... and that's something that could pretty much be built into a website. BBC Micro:bit appears to have already done it for accelerometer data: https://createai.microbit.org/

On Bangle.js we do have Tensorflow built in, but I don't think anyone is actually using that at the moment (and we don't have the space to build it into other boards by default). It's not that fast either.

Is there any interest in this? I can't imagine that many use-cases at the moment, but being able to speak commands to something like a Jolt.js and have it turn stuff on or off would be pretty cool.

[baljo]

Well I've created a couple of tutorials where I'm using Edge Impulse to create ML-models that are then deployed as Tensorflow Lite to Bangle. These tutorials are for classifying gestures and fitness activities.
If this ML4F is supported in Espruino, I'd at least test it out. I only have Bangles (11 pcs!) and 3 Pucks, so might be limited to accelerometer related projecs.

[baldursigurds]

My work on getting running speed from a Puck.js accelerometer/gyro/magneto attached to a shoe, connected via bluetooth to a Bangle.js 2 is going very slowly, but an idea that I had, which I havn't implemented, would be to take in accel/gyro/magneto data, split it into intervals corresponding to steps, and then using an algorithm to measure the length of the step. The simplest would be to integrate the data twice, the most complicated would be to use some fancy ML-models to get a number from the data. Could a model do something like this? In order to train the model, I should be able to generate data along with known speed, either using a reliable gps device in very clear conditions, or on a running track.

[gfwilliams]

I wouldn't advise using an ML model for it - it wouldn't be very efficient, and also to make it work properly you're going to need a whole bunch of labelled data from lots of different people with different walking/running styles - otherwise you have no guarantee that just because it works for you, it'll provide accurate outputs for different people with different stride lengths or gaits.