9.9.3 Machine Learning

Machine Learning is a cornerstone of Artificial Intelligence. It is attempting to provide the machine with the ability to learn and anticipate outcomes based on previous successes and failures.

Machine Learning remains a relatively complex field that could feel intimidating to many of us. Luckily, in the last couple of years, several organizations and open source communities have been developing tools and libraries that help abstract the complex mathematical algorithms in order to encourage developers to easily create learning models and train them using any programming languages.

Caligrafy created a built-in framework that is implemented on top of several open source libraries. It uses the ML5.js library built on top of Google's Tensor Flow to give developers the means to implement detection and recognition based on pre-trained models or to implement simple neural networks to train models. More over, in order to create web user interfaces more easily to interact with the AI models, Caligrafy uses the P5 library.

In this section, we will explain and demonstrate how Caligrafy can be used to build compelling detection and recognition applications.

Creating and training an AI with Caligrafy

In this video, we use Caligrafy to build neural networks and computer vision algorithms to create an AI and train it to detect and recognize humans, objects and to predict outcomes.

Initializing a `Machine Learning` application

Step 1: Scaffold an `ml` application using Caligrafer

The Caligrafy Machine Learning framework is a javascript library that is agnostic to the framework. It can be used with bare-bone Javascript implementations or with any other frameworks.

The Caligrafy-Quill distribution uses a bare bone Javascript implementation that is built on top of the ML5.js and P5.js. You need to use Caligrafer to create a boiler-plate code template to get your started.

In order to do so, run the following command from the command line: .bin/caligrafer ml <app_name> or php caligrafer.php ml <app_name>

Step 2: Verify the application

Once the scaffolding is completed, verify that a new folder <app_name> got created in the public folder.

You can even immediately try some of the features that are provided out-of-the-box by running it in the browser:

General Framework: http://localhost/<caligrafy_project_root>/ml/<app_name>
Feature Extractor: http://localhost/<caligrafy_project_root>/ml/<app_name>?app=featureextractor
Pose Detector: http://localhost/<caligrafy_project_root>/ml/<app_name>?app=posedetector
Neural Network: http://localhost/<caligrafy_project_root>/ml/<app_name>?app=neuralnetwork

Step 3 (optional): Customize the Page Route

The Page Route to the application is already created for all client-side apps in the web.php file.

    /* ML Routes
     *
     * If you don't specify an app, it will go to the face, body detector.
     * Specify an app as a URI Parameter ?app=<name of app>
     * 
     * There are 3 possible apps: featureextractor, neuralnetwork and posedetector
    */
    Route::get('/ml/{appName}', 'ClientController');

Learn more about how to create your own Page Routes here

Understanding the application structure

Caligrafy Machine Learning comes with a structure of its own. Understanding that structure is key to understanding the different components of the framework and how they interact with each other. All the client applications that you create through Caligrafer or manually reside in the public folder. The <app_name> contains the following parts:

scripts/main.js: The javascript files that use the Caligrafy ML Core library to show the 4 different implementations (General, Feature Extractor, Pose Detector and Neural Network)
.php files: The markup HTML files that will reference each of the scripts. You need to make sure to load the right js file from the markup.
resources folder: This folder is empty out-of-the-box. It will be the folder that will contain any models or data needed to load onto the application
css folder: The css folder all the css files needed for your application.

Caligrafy ML Core

Caligrafy ML Core is a client-side framework. All the logic takes place on the browser side using Javascript. The ML Core framework relies on 3 main components:

MlCore Class: This is the core class for using the machine learning in your application. The core code is in mlcore.js that can be found in the public/js/services folder
ML5.js: ML5 is a Google-backed open source project that aims to make machine learning approachable for a broad audience of artists, creative coders and students. It provides access to several machine learning algorithms and models directly in the browser building on top of TensorFlow.js without any additional dependencies. Learn more about ML5

The Ml Core Class builds on top of the ML5 library. The Ml5 library therefore needs to be imported into the markup before loading ML Core as shown in the example files index.php and featureextrator.php that are in your <app_name> folder.
```
   

   
   <script src="https://unpkg.com/ml5@0.12.2/dist/ml5.min.js"></script>
```
P5.js: P5 is an Javascript open-source library for creative coding that focuses on making coding accessible and approachable to a larger audience of artists, designers, educators, beginners and anyone else. P5 provides a full set of drawing capabilities to be used in HTML. Learn more about P5

The Ml Core Class builds on top of the P5 library. The P5 library therefore needs to be imported into the markup before loading ML Core as shown in the example files index.php and featureextrator.php that are in your <app_name> folder.
```
   

   
   <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/p5.min.js"></script>
   <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/addons/p5.dom.min.js"></script>
	
   
   <script src="https://unpkg.com/ml5@0.12.2/dist/ml5.min.js"></script>
```

Understanding ML Core

ML Core relies fundamentally on the following concept: "There is a Brain that detects and learns and there is a UI that needs to be drawn to interface with it". There are therefore 2 main paradigms that play together to create that intelligence: A Brain and an Interface

A Brain that detects and learns

The ML Core Brain relies on neural networks to achieve both detection and learning. Caligrafy come pre-packaged with trained networks that can be refined and improved. Or, it provides a framework for you to build and train your own neural network.

Detecting Human Faces and Postures

Initialization

Out of the box the brain has been pre-trained to detect human faces and body postures in videos, photos or webcams. ML Core uses Face Detection (using UNet model) and Posture Detection (using the PoseNet model) to achieve those detections.
```
   
    // Initializing ML Core for detections

    var myMl = new MlCore();

    myMl.detect((myMl) => {
     // do something upon detections

    });
```

Methods

    
    /* 
     * Training Methods 
     */
    
    myMl.addImage(myMl, labelInputs) // adds sample image data for training the brain

    myMl.train(myMl, options = {}) // trains the model and upon results automatically starts classifying
    // options: { batchSize: 24, epochs: 32 }

    myMl.classify(myMl) // classifies the detections

    myMl.save(outputName, callback, myMl) // saves model

    myMl.load(fileOrpath, callback, myMl) // loads modal


    /*
     * Visualization/Drawing methods
     */

    myMl.drawFeature(myMl, Array features to draw) // draws particular feature of the face (nose, leftEye, rightEye, leftEar, rightEar, mouth, topLip, bottomLip etc.

    myMl.drawKeypoints(myMl) // draws the keypoints detected on the entire body

    myMl.drawSkeleton(myMl) // draws the lines for arms, shoulders, legs etc...

Learning with images

Initialization

Caligrafy relies on well established computer vision models to train the machine to recognize data and image patterns. In order to learn from images, MLCore uses Feature Extractor that relies on a highly pre-trained computer vision algorithm MobileNet.
```
   // Initializing a feature classifier
   var myMl = new MlCore({ brain: {type: 'featureextractor', options: { numLabels: 3, epochs: 20, batchSize: 0.4 }}}); 
```

Methods

     
   myMl.addImage(myMl, labelInputs) // adds sample image data for training the brain

   myMl.train(myMl, options = {}) // trains the model and upon results automatically starts classifying
   // options: { batchSize: 24, epochs: 32 }

   myMl.classify(myMl) // classifies the detections

   myMl.save(outputName, callback, myMl) // saves model

   myMl.load(fileOrpath, callback, myMl) // loads modeal

Custom Neural Networks

Initialization

When we talk about learning, under the hood it means improving the ability of the machine to predict an outcome (output) in a particular context (input) by training it (manually or programmatically) to recognize patterns. The predictions of outcomes could be either qualitative or quantitative.

A qualitative prediction for example could be the recognition of a human pose (crouching, standing, etc.). This is about labeling a combination of posture coordinates. ML Core achieves this type of learning through what is called Classification in the AI field.
```
   // Initializing a neural network for classification
   var myMl = new MlCore({ brain: { type: 'neuralnetwork', options: { inputs: 3, outputs: 1, debug:true, task: 'classification'}}}); 
```
A quantitative prediction for example could be the recognition of a price based on the recognition of a historical trend. ML Core achieves this type of learning through what is called Regression in the AI field.
```
   // Initializing a neural network for regression
   var myMl = new MlCore({ brain: { type: 'neuralnetwork', options: { inputs: 3, outputs: 1, debug:true, task: 'regression'}}}); 
```

Methods

   myMl.brain.addData(labelInputs) //adds sample data for training the brain

   myMl.brain.addImage(labelInputs) // adds sample image data for training the brain

   myMl.brain.train(options = {}) // trains the model and upon results automatically starts classifying
   // options: { batchSize: 24, epochs: 32 }

   myMl.brain.classify() // classifies the detections
    
   myMl.brain.normalizeData() // normalizes the data (to be used when continuous data and not discrete)
    
   myMl.brain.save(outputName, callback) // saves the model to specified file name

   myMl.brain.load(fileOrpath, callback) // loads the saved model

An Interface to interact with the Brain

When the Brain makes sense of the information, it needs a way to communicate its findings and reciprocally, when it needs intervention from the human operator, it needs to be able to provide them with graphical input elements.

ML Core uses the P5 library to allow the creation of more elaborate visualizations and interfaces using simple language. You can learn more about the capabilities of the P5 library here

In order to use the P5 library, MLCore needs to hand the canvas over to P5

   function setup() {
          
       myMl.toP5(myMl); // hand off to P5

       /* P5 UI components for interfacing with the brain go here */

   }
   
   function draw() {
        
     /* P5 Canvas drawing for visualizing detections goes here. 
This function is executed only if the hand off to P5 is executed in setup */

   }

Code Patterns

ML Core has 3 distinct code patterns that are optimized for different purposes.

Detect and Visualize

In this mode, the ML Core code structure flows as such: initialize the brain, set it up, visualize the detections and optionally do custom drawings.

A typical example of this flow is the General Framework example http://localhost/<caligrafy_project_root>/ml/<app_name>.

View Code

Live Example
Unattended Learning

In this mode, the ML Core flow is designed to programmatically train the machine to predict an outcome. The ML Core structure flows as such: initialize the neural network, load or create the samples, add the samples to the network, optionally normalize the data, train the network, classify/predict.

A typical example of this flow is the Neural Network example http://localhost/<caligrafy_project_root>/ml/<app_name>?app=neuralnetwork.

View Code

Live Example
Attended Learning

In this mode, the ML Core flow is designed for the intervention of a human operator in the training of the machine. It therefore requires a user interface. The Ml Core structure flows as such: initialize the brain, visualize the detections, set up a P5 user interface that adds samples, trains and classifies/predicts, visualize the predictions on the user interface.

Typical examples of this flow are the Feature Extractor and Pose Detector examples:
- Feature Extractor: http://localhost/<caligrafy_project_root>/ml/<app_name>?app=featureextractor.
  
  View Code
  
  Live Example
- Pose Detector: http://localhost/<caligrafy_project_root>/ml/<app_name>?app=posedetector.
  
  View Code
  
  Live Example

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

9.9.3 Machine Learning

Creating and training an AI with Caligrafy

Initializing a `Machine Learning` application

Step 1: Scaffold an `ml` application using Caligrafer

Step 2: Verify the application

Step 3 (optional): Customize the Page Route

Understanding the application structure

Caligrafy ML Core

Understanding ML Core

A Brain that detects and learns

Detecting Human Faces and Postures

Learning with images

Custom Neural Networks

An Interface to interact with the Brain

Code Patterns

Learning Caligrafy

Caligrafy and VueJS

Basics - Vue.js as a library

For large scale application purposes

AI in Caligrafy

Clone this wiki locally

9.9.3 Machine Learning

Creating and training an AI with Caligrafy

Initializing a Machine Learning application

Step 1: Scaffold an ml application using Caligrafer

Step 2: Verify the application

Step 3 (optional): Customize the Page Route

Understanding the application structure

Caligrafy ML Core

Understanding ML Core

A Brain that detects and learns

Detecting Human Faces and Postures

Learning with images

Custom Neural Networks

An Interface to interact with the Brain

Code Patterns

Learning Caligrafy

Caligrafy and VueJS

Basics - Vue.js as a library

For large scale application purposes

AI in Caligrafy

Clone this wiki locally

Initializing a `Machine Learning` application

Step 1: Scaffold an `ml` application using Caligrafer