Home
Tutorial videos can be referred here. (Dated May 2022)
- CPU/GPU Backend Switching
- Dataset Generation Through Webcam
- Construct Neural Network & Segmentation & ReTrain Pretrain model for object detection
- Object Detection using Pretrained Model (Tiny YOLO)
- Object Classification Deployment
- MQTT Signaling and Output
For enabling different backends at runtime, users can set the priority with the environment via environment variable
> SETX BACKEND_PRIORITY_CPU [NUMBER]
> SETX BACKEND_PRIORITY_GPU [NUMBER]
To check which backend is used, the simplest approach is to look the at the "Backend section" located in "NN & Segmentation & Retrain Pretrained Model", "Object Detection" and "Object Classification" page.
GPU setup (Optional)
Follow the instructions below if you plan to use GPU setup.
Install CUDA and cuDNN Requirements:
- CUDA 10.2
- cuDNN 7.6.5
CUDA and cuDNN can be downloaded from here and here. Step by step installation guides can be found here.
Dependencies are needed to be included into Maven project if we wish to use GPU for training. Follow the links below for instructions in details.
Watch a DEMO about dataset generation using DL4JRA.
- Navigate to "GENERATE DATASET" section from main page.
- Configure the dimension (width & height) of images, number of image to take and the location (directory) to save.
- Select a camera/webcam/video source.
- Click on Screenshot button.
Three components:
Controls
- drag and drop nodes from tool panel to canvas.
- double click on dropped nodes to configure details.
- click on any dropped nodes, and backspace to delete any drop nodes.
- click on CONSTRUCT button to run the network built.
- Export node can be added for Neural Network and ReTrain pretrained model for Object Detection components.
- below image shows where is CONSTRUCT button and Export node are located:
Nodes for this component:
-
StartNode Auto-Split, Training StartNode Image, Validating Startnode Image:
This is to start the process to iterate your image dataset. If you have Training and Validating dataset, use Training Startnode for Training dataset and Validating startnode for Validating Dataset. If you have one whole folder of your dataset, you can use the Auto-Split startnode to make DL4JRA to auto-split your model to 80:20; 80% for Training and 20% for Validating. -
Load DS: Stands for Load Dataset, you can specify the path of your dataset with the node, set the dimension, number of labels in your dataset and batchsize. *If you encounter a memory related error, you can lower down the number of batchsize.
-
Image preprocess nodes (Flip, Rotate, Resize):
The purpose of Image preprocess nodes is to give variation of dataset for your image classification model.
* 'Flip' node will flip your images, according to the axis you set. There's flipping through both X & Y axis or X or Y axis.
* 'Rotate' node will rotate your images, according to the angle you have set.
* 'Resize' node will resize your images, according to the dimension you set in the configuration.
-
Training Startnode CSV and Validation Startnode CSV:
Similar to image inputs training and validating startnode, use this nodes to command DL4JRA to start the data iterator for CSV inputs. -
Load TimeSeries CSV:
This node is to specify the path of your datasets for your time-series CSV files. Specify the number of labels you have in the CSV and numskiplines is to specify the row number of where you want the neural network to skip. This can be used in situations where you have a header because the header is not related to the data for the machine learning. *Ensure that your data in CSV are all in numerical values.
-
Load Numerical CSV: This node is to specify the path of your dataset for your general CSV. Specify the number of labels you have in the CSV and numskiplines is to specify the row number of where you want the neural network to skip. This can be used in situations where you have a header because the header is not related to the data for the machine learning. In fractiontrain, you can specify the ratio for training and the rest will be used for validating. Example: 0.8 means 80% used for training and the rest 20% is used for validating. *Ensure that your data in CSV are all in numerical values.
-
Multilayer (S): This is the start node for Multilayer network. Use this to specify that this process is a multilayer process.
-
Multilayer Config: To configure the multilayer network.
Seed: Seed in machine learning means the initialization state of a pseudo-random number generator. If you use the same seed in another machine/computer, you will get exactly the same pattern of numbers.
Learning rate: To specify the learning rate for the model. 0.001 is default.
Optimization Algorithm: An optimization algorithm is a procedure which is executed iteratively by comparing various solutions till an optimum or a satisfactory solution is found. Stochastic Gradient Descent(SGD) is default.
Convolution Mode: Sets the convolution mode for convolutional layers, which impacts padding and output sizes. Truncate is default.
Activation Function: Activation function in a neural network defines how the weighted sum of the input is transformed into an output from a node or nodes in a layer of the network. RELU (Rectified Linear Unit) is default.
Weight Init(Initialization): Weight initialization is a procedure to set the weights of a neural network to small random values that define the starting point for the optimization (learning or training) of the neural network model. Xavier is default.
Gradient Normalization: Normalization is a data pre-processing tool used to bring the numerical data to a common scale without distorting its shape. -
Convolutional Layer :
KernelX and KernelY: Kernel is a filter that is used to extract the features from the images. You can read this article to understand further about Kernels. 2 is default.
StrideX and StrideY : Stride is a parameter of the neural network's filter that modifies the amount of movement over the image or video. 1 is default.
PaddingX and PaddingY : Padding refers to the amount of pixels added to an image when it is being processed by the kernel of a CNN. Padding is added to the outer frame of image in order to allow more space for the filter to cover. Adding padding to an image allows CNN to give a more accurate analysis of the image. 0 is default.
Nin and Nout : NIn|(Node In) and NOut|(Node out). Nodes are like neurons in our brains. Node is a computational unit that has one or more weighted input connections, a transfer function that combines the inputs in some way, and an output connection. Read this article to understand further about NOut. 10 is default.
Dropout : Dropout are the regularization technique that is used to prevent overfitting in the model. Dropouts usually not advised to use after the convolution layers and usually used after dense layers of network. 0 is default.
BiasInit : For ReLU non-linearities, some people like to use small constant value such as 0.01 for all biases because this ensures that all ReLU units fire in the beginning and therefore obtain and propagate some gradient. However, it is not clear if this provides a consistent improvement (in fact some results seem to indicate that this performs worse) and it is more common to simply use 0 bias initialization. Quoted from CS231n -
Subsampling Layer: Sub-sampling is a method to downsample feature maps as we move along the network.
-
Dense: Dense layer is a layer that fully connected with its preceding layer which means the nodes of the layer are connected to every node of its preceding layer. This is the most commonly used layer.
-
Output: Output layer, the final layer for the convolutional neural network mode.
Activation Function: Activation for this layer should be 'Softmax'.
NOut: NOut for this layer should be the same number as number of labels of your dataset. -
Local Response Normalization Layer: Local Response Normalization Layer implements the lateral inhibition which is to subdue its neighbouring nodes to normalized the excited nodes. Read this article to learn more.
-
Image Input Configuration:
Channels: A channel in this context is the grayscale image of the same size as a color image, made of just one of these primary colors. For instance, an image from a standard digital camera will have a red, green and blue channel. A grayscale image has just one channel. Default is 3, for 3 colour channels which is RGB. -
MultiLayer Construct: The final node to command DL4JRA to start constructing the network as the nodes arranged.
-
ComputationGraph Startnode: This is the start node for Computation Graph network. Use this to specify that this process is ComputationGraph network process.
-
ComputationGraph Config: To configure the computationgraph network.
-
Add Input: A list of strings telling the network what layers to use as input layers. trainFeatures is the default. Advisable to leave it as default.
-
Set Output: A list of strings telling the network what layers to use as output layers. predictActivity is the default. Advisable to leave it as default.
-
Convolutional: Convolutional layer configuration, read above to understand what does the configuration meaning and what it does. Advisable to leave it as default.
-
LSTM: LSTM (Long-Short Term Memory) layer learns long-term dependencies between time steps in time series and sequence data. Advisable to leave it as default.
-
Rnn Output Layer: The final/output layer's configuration for ComputationGraph network.
RNN format: For array manipulations. Options available are NCW and NWC, which stands for: (N)umber of samples, (W)idth (or sequence length) and (C)hannels.
NIn: Place nIn the same value as previous nOut to ensure all nodes are connected.
NOut: Number of out nodes for output layer equals to the number of labels we have in the dataset.
-
ComputationGraph Construct: The final node to command DL4JRA to start constructing the network as the nodes arranged.
How it works? There are two parts:
Work with data [load and preprocess(optional)] -> Contrust the Neural network for evaluation (train and validate)
1) Work with data [load and preprocess(optional)].
DL4JRA supports two types of data, either image data or CSV data.
If the train and validation folders are separate then construct sequence as below:
train start node -> image pre process nodes (if needed) -> load DS noded -> data iterator node
test start node -> image pre process nodes (if needed) -> load DS noded -> data iterator node
If data comes in all in one folder without having being split into train and validation, construct sequence as below.
Do ensure that:
- the number of images in each folder are roughly similar. Having some folders with very few images will cause images in other folders to be deleted, and may lead to a mismatch of the number of labels between the test and train dataset if there are insufficient images to evenly split between them
- that there sufficient images so that the images can be split into test and train, yet also be of sufficient numbers so that meaningful characteristics can be extracted from them
start node auto split-> image pre process nodes (if needed) -> load DS noded -> data iterator node
Load Dataset node (for DATASETS for NN)
- Image inputs
In PATH, enter full file address of the parent file that contains files of images, with the label as the name of folder.
Example:
- CSV inputs
In PATH, enter full file address of the parent file that contains features CSV file and labels CSV file.
NUMCLASSLABELS refer to the number of labels for the dataset. Value of labels must be in numeric.
NUMSKIPLINES refer to the row in the CSV that you wish to skip, for example data header cannot be process during training. You can leave it 0 if there is no row of CSV that wish to skip.
2) Construct the Neural network for evaluation (train and validate).
DL4JRA also supports two types of network configuration, Multilayer Configuration or ComputationGraph Configuration. Both are used for construction and does similar works just required input parameters are a little different.
a). Multilayer Configuration Network
MultiLayer(S) -> MultiLayer Config -> ... layer nodes ... -> OutputLayer node -> I.Type node -> Multilayer Construct node -> Train -> Validate -> Export node
NOTE: Nodes name after being dropped to the canvas shows different name as before when it was on tool bar.
b). ComputationGraph Configuration Network
ComputationGraph (S) -> ComputationGraph Config -> Add Input node -> Set Output node -> ... layer nodes ... -> Rnn OutputLayer node -> ComputationGraph Construct node -> Train -> Validate
NOTE: Nodes name after being placed on the canvas show different names compared to when they are on the toolbar.
Demo Neural network classifier construction for object detection with videos
How to construct? link to DEMO.
How to use? link to DEMO.
Nodes for this component:
-
Segmentation (S): This is the start node for Segmentation network. Use this to specify that this process is Segmentation network process.
-
Import Unet: U-Net is a semantic segmentation technique, one of the earlier deep learning segmentation models. Read here to understand further about U-Net model. Use this node to import the U-Net model into the network flow.
-
Configure FineTune: This node will finetune the model, and you can set the random number generator seed in this configuration node.
-
Configure Transfer Learning: To configure the U-Net for the dataset we have set.
nInName: The name of the layer that responsible for the node input. Default is conv2d_1. As there is only one nIn name input, nIn should be 1(default) too.
-
Set Output: This node is to set the output layer for the segmentation model.
OutputName: is the name of the output layer. Default is predictActivity. If you faced an error in regards of 'predictActivity', please change it to 'output' (small letters)
-
Build Transfer Learning: The final node to command DL4JRA to start building the transfer learning process following how the nodes are arranged.
- Data Start Node: This node is to start the process of loading and iterating your image dataset for your segmentation training process.
-
Setup Iterator: To configure the data iterator.
Path: This is where you can put the location address of your dataset folder.
BatchSize: To specify how many number of samples that will be passed through to the network at one time. Default is 2.
TrainPerc: The percentage ratio for dividing the dataset for training and validating purposes. Default is 0.8.
Channels: Default is 1, for grayscale. Segmentation only works with grayscale image, so it is advisable to let it be 1.
-
Generate Iterator: The node to tell DL4JRA to start the iterator process.
Note for Path:
- If you have a sample data from .deeplearning4j, please change the path to the parent file of your dataset.
For example:
[your home directory].deeplearning4j\data\data-science-bowl-2018\data-science-bowl-2018\data-science-bowl-2018-2\train\inputs
Where [your home directory] is the path to your home directory.
- if you would like to use your own images, please set up your dataset in folders as follows:
└── [parent-folder]
├── inputs
└── [mask-folder-name]Where [parent-folder] and [mask-folder-name] are user determined folder names. Place the masks in [mask-folder-name] and input the folder name in the node configurations "maskFolderName" like below.
- Mask name should follow the following naming convention [name-of-original-image]_mask.png .
How it works? It consist of two parts also:
Building Transfer Learning -> Load data and evaluation (train and validate)
1) Building Transfer Learning.
Segmentation(S) -> Pretrained Model node -> Configure FineTune node -> Configure Transfer Learning node -> Add CnnLossLayer node -> Set Output node -> Build Transfer Learning node
2) Load data and evaluation (train and validate).
Data Start node -> Setup Iterator node -> Generate Iterator node -> Train (TrainNN or Train No Ui) -> Validate
Demo Segmentation construction with screen shots
Using all nodes in Segmentation to duplicate the same network from Segmentation sample:
Networks duplicated using DL4JRA:
Configuration on each node (double click to configure nodes on canvas) that requires user's input:
Configure FineTune node -> 
Configure Transfer Learning node ->
Add CnnLossLayer node ->
SetOutput node ->
Setup Iterator node ->
Other than that please set the rest of the settings as indicated in the following images.
Train node ->
NOTE: nodes are has not been explained with screenshots but were in the duplicated network screenshot does not requiring user's input
Nodes for this component:
-
Train & Test: To start the train and test process of the transfer learning. You can specify the number of epochs of the training process.
-
Dataset ODetection (S): To start the data iterator for Object Detection dataset.
-
Load Dataset ODetection: To load and configure the dataset for iteration.
trainPath & testPath: The address of your folder location for train folder and test folder. Object Detection dataset is a bit different to CNN dataset.
The format of files arrangement as follows:
└── [parent-folder]
├── images files
└── [annotations folder]Inside the annotations folder are PascalVOC annotations folder of the images file. The annotations files' name is the same as the name of the images files.
-
Dataset Iterator ODetection: The node to tell DL4JRA to start the iterator process.
-
StartNode(s) : This is the start node for Object Detection Pretrained network. Use this to specify that this process is Object Detection Pretrained model process.
-
Import New TinyYolo : This node is to import TinyYolo model and structure to start the transfer learning process by using TinyYOLO as the base model.
-
Import Yolo2: This node is to import YOLOv2 model and structure to start the transfer learning process by using YOLOv2 as the base model.
-
Load Existing Model: This node is to import the existing custom TinyYOLO model that we have trained earlier.
-
Configure Transfer Learning: To configure the learning process, you can specify the learning rate for your model to train with.
How it works? It consist of two parts also:
Object Detection Dataset -> Train Pretrained model
1) Object Detection Dataset.
Dataset ODetection(S) -> Load Dataset ODetection node -> Dataset Iterator Odetection node <br />
2) Train Pretrained model.
StartNode(S) -> Import New TinyYOLO node __OR__ Load Existing TinyYOLO node -> Configure Transfer Learning node -> Train & Test node-> Export NN node
Watch the first DEMO about Object Detection with original TinyYOLO.
The second DEMO about Object Detection with Retrained TinyYOLO in DL4JRA.
Here, an earlier DEMO about Object Detection with original TinyYOLO, this video contained how LOGGING button works.
This component only support Multilayer networks. Watch a DEMO about image classification deployment in DL4JRA.
- Change the "PATH TO CLASSIFIER" to the file (classifier) location. Users can train and export model through the CNN section.
- Click the UPDATE button and wait the server to load the target model.
- Configure settings
- Change the input width and height to the dimension of training dataset of the classifier
- Set the name of labels (in alphabetical order)
- Select a camera/webcam/video source.
- Hit on START streaming button to start classification.
What is MQTT?
MQTT is a simple messaging protocol, designed for constrained devices with low bandwidth. So, it's the perfect solution to exchange data between multiple IoT devices.
Current protocol used: EMQX
For files generation (Dataset Generated Screenshots and Logging file):
- Make a 2nd tab, one with Generate Dataset menu, one with MQTT Signalling.
- Click on "Add Configuration" button, a yellow menu box will come out.
- Click connect. *Disclaimer: EMQX protocol is an open-source/public protocol. Be wary with what you are sharing if you are opening the webcam.
- Copy the Signal path and navigate to the MQTT tab. Paste the path to the destination box.
- Click signal and you will see that the Dataset Generation have screenshot the image and saved in the set address for the generated dataset.
- Same concept applies to generating logging file in object detection and object classification.
Using DL4JRA to show output in another IoT device:
The broker address can be placed into EMQX with the destination name.
- Navigate to Object Detection menu.
- Configure everything that are needed first, (model, train/test dataset path, etc.)
- Scroll down until you see Output. Click "Add Configuration".
- Click "Start" for streaming and click "connect" on Output Configuration.
- An output stream will show up at the IoT device that have been set up with the protocol broker path.
Watch a DEMO about MQTT Signaling in DL4JRA.