Design, implementation and testing of deep-learning-based driving scenario classifiers for automated vehicles

The aim of this project is to design, implement and test the performance of different neural networks for driving scenarios classification.

The first implemented networks are two versions of a residual 3D convolutional network:

• the first one (R3D) uses fully 3D kernels
• the second one (R(2+1)D) separates, in each constituting block, the 2D spatial convolution from the temporal convolution.

Finally, a (YOLOP + CNN) + LSTM stack was realized, which spatially processes each frame only once (instead than every time it appears in a sliding window of scenario analysis), in order to try and obtain latency times compatible with online analysis of driving scenarios. The networks performance were tests using a synthetic dataset composed by four simple class (approaching static leading vehicle, change lane, following leading vehicle and free ride) in three different conditions (day, night and rain with fog).

Networks’ structure

The three networks are implemented using pytorch and tensorflow:

Dataset:

Syntetic dataset obtained using Carla simulator:

Example of previous dataset's sample processed with YOLOP

Results

The experimental results obtained seem to suggest two different use cases for the analyzed networks:

• Offline:
  • R(2+1)D that is more suited when the scenario changes quickly or a low latency is required
  • R3D that is better for slow-changing scenarios or with low FPS rates.
• Online:
  • (YOLOP + CNN) + LSTM

Static analysis: Confusion matrix (full video)

Dinamic analysis (Different length video-clip and conditions)

Accuracy and precision performances with different number of testing and training frames

Residual convolutional neural networks tested with 2 seconds video-clip length:

(YOLOP + CNN) + LSTM tested with 4 seconds video-clip length:

Time inference performances:

Requirements

see requirements.txt

Acknowledge

YOLOP

R(2+1)D