awesome-deep-learning-techniques

A curated list of awesome deep learning techniques for deep neural networks training, testing, optimization, regularization etc.

• Weight Initialization
• Data/Input Processing
• Data Augmentation
• Decreasing/Changing Learning Rate
• Regularization
• Optimization/Gradient Descent
• Normalization
• Activation Function
• Segmentation

Weight Initialization

• Xavier Initialization
• He Initialization

Data/Input Processing

• Input pipelining
• Queues

Data Augmentation

• Random cropping
• Random padding
• Random horizontal flipping
• Random RGB color shifting

Decreasing/Changing Learning Rate

• Learning rate decay
• Cyclic learning rate

Regularization

• Weight decay a. L2 loss b. L1 loss
• Dropout

Optimization/Gradient Descent

• Adam Optimizer
• SGD with momentum
• Nesterov Accelerated Gradient (NAG)
• Stochastic Gradient Descent (SGD)

Normalization

• Batch Normalization
• Local Response Normalization (LRN)

Activation Function

• ReLU
• Sigmoid

Segmentation

• Fully Convolutional models
• Conditional Random Fields (CRF)
• Skip Connections/ Fusions
• Upsampling/ Transpose Convolutions
• Atrous/Dilated Convolutions
• Multi-scale inputs/nets with weights
• Attention to scale
• Pixel wise cross entropy loss
• Dataset and annotations