-
Notifications
You must be signed in to change notification settings - Fork 51
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #806 from WildernessLabs/develop
v1.15.0.0 release notes + TensorFlow docs
- Loading branch information
Showing
7 changed files
with
154 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
91 changes: 91 additions & 0 deletions
91
docs/Meadow/Meadow.Foundation/Libraries_and_Frameworks/TensorFlowLite/index.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| --- | ||
| layout: Meadow | ||
| title: TensorFlowLite for Microcontrollers Library | ||
| subtitle: Using the Meadow.Foundation TensorFlow integration for running machine learning models on Meadow devices. | ||
| --- | ||
|
|
||
|
|
||
| The Meadow.TensorFlow library provides Meadow users with the ability to run machine learning models directly on microcontrollers using TensorFlow Lite. It is designed to enable inference from pre-trained models efficiently, using minimal resources, while being optimized for IoT environments. | ||
|
|
||
| With this library, you can load and run models for tasks like image recognition, gesture detection, and other forms of inference. This documentation will walk you through how to integrate TensorFlow into your Meadow application, load models, and run inferences. | ||
|
|
||
| # Using TensorFlow on Meadow | ||
|
|
||
| TensorFlow Lite for Microcontrollers on Meadow provides a modern .NET API, allowing you to run machine learning inference directly on your Meadow devices. This enables you to build intelligent IoT applications that can process data locally without relying on cloud services. | ||
|
|
||
| ## Integrating TensorFlow into Your Meadow Application | ||
|
|
||
| To get started, you'll need to add the `Meadow.TensorFlowLiteForMicrocontrollers` library to your Meadow application. You can do this by adding the NuGet package to your project: | ||
|
|
||
| ```csharp | ||
| dotnet add package Meadow.TensorFlowLiteForMicrocontrollers | ||
| ``` | ||
|
|
||
| ## Loading a TensorFlow Lite Model | ||
|
|
||
| First, include your TensorFlow Lite model (.tflite file) in your Meadow project. Set its Build Action to EmbeddedResource so it gets embedded into your application assembly. | ||
|
|
||
| To load the model, read it as a byte array from the embedded resources: | ||
|
|
||
| ```csharp | ||
| // Load the model from embedded resources | ||
| var assembly = Assembly.GetExecutingAssembly(); | ||
|
|
||
| using (var stream = assembly.GetManifestResourceStream("YourNamespace.Models.gesture_model.tflite")) | ||
| { | ||
| byte[] modelData = new byte[stream.Length]; | ||
| stream.Read(modelData, 0, modelData.Length); | ||
|
|
||
| // Create the GestureModel object with the model data | ||
| gestureModel = new GestureModel(modelData); | ||
| } | ||
| ``` | ||
|
|
||
| Replace "YourNamespace.Models.gesture_model.tflite" with the actual namespace and resource path of your model file. | ||
|
|
||
| ## Preparing Input Data | ||
|
|
||
| The `GestureModel` class provides a `ModelInput` property to manage inputs. You need to prepare your input data according to the model's expected input format. | ||
|
|
||
| For example, if your model expects a float array of a specific size: | ||
|
|
||
| ```csharp | ||
| // Prepare your input data (e.g., sensor readings) | ||
| float[] inputData = new float[] { /* your input data */ }; | ||
|
|
||
| // Set the input data | ||
| gestureModel.ModelInput.SetData(inputData); | ||
| ``` | ||
|
|
||
| ## Running Inference | ||
|
|
||
| With the model loaded and input data prepared, you can run inference by calling the `Predict` method: | ||
|
|
||
| ```csharp | ||
| // Run the prediction | ||
| var output = gestureModel.Predict(); | ||
| ``` | ||
|
|
||
| ## Interpreting the Results | ||
|
|
||
| The `ModelOutput` object contains the output data from the model, which you can interpret according to your application's logic. | ||
|
|
||
| For example, if the model outputs probabilities for different classes: | ||
|
|
||
| ```csharp | ||
| // Assuming output contains a float array of probabilities | ||
| float[] probabilities = output.GetOutputData(); | ||
|
|
||
| // Find the class with the highest probability | ||
| int predictedClass = Array.IndexOf(probabilities, probabilities.Max()); | ||
|
|
||
| // Use the predicted class in your application | ||
| Console.WriteLine($"Predicted Class: {predictedClass}"); | ||
| ``` | ||
|
|
||
| # Best Practices | ||
|
|
||
| * **Model Optimization** - Optimize your model for size and performance using quantization and pruning techniques. | ||
| * **Resource Management** - Meadow devices have limited resources. Keep your models small and efficient. | ||
| * **Input Validation** - Ensure that the input data matches the model's expected format and data types. | ||
| * **Error Handling** - Implement error handling to manage issues with model loading and inference execution. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| --- | ||
| layout: Meadow | ||
| title: Artificial Intellegence (AI) | ||
| subtitle: Artifical Intelligence (AI) at the edge with Meadow. | ||
| --- | ||
|
|
||
| AI @ the edge is a critically important part of many IoT workflows and growing exponentially. IoT devices routinely not only provide sensory input to model training in the cloud but in turn, execute those models locally for efficient, low-latency outcomes: | ||
|
|
||
| * **Sensory Input** - ML training pipelines and modern LLMs are gobbling up data at an unprecedented pace. The edge provides a wealth of data from the real world that can only be gathered from field-deployed devices. Digital twins would be impossible without the edge sensory input that feeds them, and the insights from not only individual devices, but in aggregate, they can unlock insights that can massively increase overall efficiency and productivity. | ||
| * **Local Model Execution** - While it’s generally not practical to train models on edge, these devices have outsized capabilities to execute models locally. For instance, realtime defect detection via an attached camera can run easily on a microcontroller (MCU), and object detection in multi-channel video can be done on a multi-GPU Jetson NX rather efficiently. This seminal post by Pete Warden (Useful Sensors) explains why non-LLM AI/ML models run so efficiently on low-powered devices. | ||
|
|
||
| # Benefits of Running @ the Edge | ||
|
|
||
| Additionally, running AI at the edge rather than in the cloud has a number of important efficiencies and advantages: | ||
| * **Lower Latency** - By executing locally, model execution outcomes are available immediately, not requiring a round-trip to the server. | ||
| * **Reduced Bandwidth Cost** - It’s magnitudes more efficient to send the output of a model than the raw data that fed it (once training has occurred). For instance, if you’re creating a smart outlet that tracks appliance usage, all that the cloud cares about is what appliances are running and if there are anomalies. It doesn’t need the raw waveform data of actual electrical usage which may not even be possible to feed over a low-bandwidth IoT connection. | ||
| * **Reduced TCO** - By outsourcing ML processing to the edge and running in situ, server loads (and costs) can be reduced and devices can use available processing cycles as part of their normal operating lifecycle. | ||
|
|
||
| # AI on Meadow Devices | ||
|
|
||
| Meadow provides a productive and powerful, unified, multi-platform AI experience at the edge for .NET developers, whether it be an MCU-based device like a Meadow F7, Single-Board-Computer (SBC) such as a Raspberry Pi or Multi-GPU Jetson Nano, or desktop machine. | ||
|
|
||
| Today Meadow enables AI across the following vectors: | ||
|
|
||
| * **[TensorFlowLite for Microcontrollers](/Meadow/Meadow.Foundation/Libraries_and_Frameworks/TensorFlowLite/)** - TensorFlowLite is an model execution runtime that is baked into Meadow.OS as a first-class feature. Simply add the nuget package, load your model, feed it data, and get the results. For more information see the [TensorFlowLite for Microcontrollers guide](/Meadow/Meadow.Foundation/Libraries_and_Frameworks/TensorFlowLite/). | ||
|
|
||
| * **Useful Sensors Drivers** - We have Meadow.Foundation drivers for Useful Sensors, which are purpose built, commodity AI sensors for doing person detection and QR code reading. See the [Meadow.Foundation Peripheral driver list page](/Meadow/Meadow.Foundation/Peripherals/) for the latest Useful Sensors drivers. | ||
|
|
||
| * **Single-Board-Computer (SBC) Meadow Support** - Meadow.Linux has full support for a number of the most popular SBCs, including Raspberry Pi boards, Beagle Bone, Nvidia Jetson, and more. With Meadow's first class hardware and industrial protocol support, it's super easy to get sensor data into the SBC AI workflows. For more information, see the [Getting Started guides](/Getting_Started/) on how to get up and running with Meadow on SBCs. | ||
|
|
||
| # Future Integrations | ||
|
|
||
| * **Edge Impulse** - Edge Impulse provides not only pre-trained, optimized models for embedded scenarios, but they also provide a powerful workflow for importing training data, labeling it, and then training, tuning, and outputting a useful model. We're currently in the process of integrating Edge Impulse into Meadow.OS as a first class feature. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters