Starting 24.08.2020 we are hosting a Kaggle competition about predicting future movements of other traffic participants. This page serves as introduction point for it and gives additional information.
When taking part in the competition, you will be asked to submit predictions for a private test set (no ground truth is available), and your solutions will be scored by Kaggle. Overall 30.000 USD as prizes are available! As traffic scenes can contain a large amount of ambiguity and uncertainty, we encourage the submission of multi-modal predictions. For scoring, we calculate the negative log-likelihood of the ground truth data given these multi-modal predictions. Let us take a closer look at this. Assume, ground truth positions of a sample trajectory are
,
and we predict K hypotheses, represented by means
.
In addition, we predict confidences c of these K hypotheses. We assume the ground truth positions to be modelled by a mixture of multi-dimensional independent Normal distributions over time, yielding the likelihood
yielding the loss
You can find our implementation here, which uses error as placeholder for the exponent
and for numeral stability further applies the log-sum-exp trick: Assume, we need to calculate the logarithm of a sum of exponentials:
Then, we rewrite this by substracting the maximum value x* from each exponent, resulting in much increased numerical stability:
Scoring multi-modal prediction models is a highly complex task, and while we chose the metric described above due to its elegance and support for multi-modality, we encourage participants to also employ other metrics for assessing their models. Examples of such other metrics, commonly used in literature, are Average Displacement Error (ADE) and Final Displacement Error (FDE) (see our dataset paper or SophieGAN): ADE is the average displacement error (L2 distance between prediction and ground truth averaged over all timesteps), while FDE reports the final displacement error (L2 distance between prediction and ground truth, evaluated only at the last timestep). As we consider multiple predictions, we offer implementations for both these metrics either averaging over all hypotheses or using the best hypothesis (oracle variant) - ignoring generated confidence scores in both cases.