Trajectron++ evaluation metrics

[mzahran001]

Hi @BorisIvanovic

Quick question, I am wondering why Trajectron++ did not report minADE as the case in MultiPath paper and in CoverNet? 🤔

I found that the only difference between the current implementation for the FDE function and the implementation used by these two papers is the ranking part.

In Argoverse and NuScence's challenges, they will use both minFDE and minADE
What do you think? Am I missing something here? 😅

[BorisIvanovic]

Hi @mzahran001 ,

We do! Although the "minADE/minFDE" we report is in line with prior stochastic methods, namely the methods referenced in Table 1 (b) (where we sample N=20 times from the model and report the best-performing trajectory, i.e., the minimum error trajectory).

As for why we didn't specifically report the "minADE/minFDE" metrics as defined in the nuScenes challenge for our nuScenes results, well, the nuScenes prediction challenge didn't exist when we were writing the paper! :P

[mzahran001]

Thank you so much for your answer! I missed this part 😅

So if I want to calculate the "minADE/minFDE" for the current version of the paper taking in the consideration the nuScenes metrics, I will only use the same code used on the pedestrian datasets

[BorisIvanovic]

No worries!

I think the more correct way to evaluate the method for the nuScenes prediction metrics would be to actually get the z values from the model, rank them according to their probability (from the model's CVAE likelihood p(z|x)), get the corresponding mean predictions from each z (it's a Gaussian at the output), and then take the min of those for the metrics. I'll post some code below that might help.

[BorisIvanovic]

@mzahran001 You'd want to call the model's prediction function like so:

probs, predictions = trajectron_model.predict(test_scene,
                                                  np.array([timestep]),
                                                  ph,
                                                  num_samples=1,
                                                  all_z_sep=True,
                                                  gmm_mode=True,
                                                  full_dist=False)

where I've modified the predict function to also return the mode likelihoods (right now it doesn't do that, I'll explain below how you can do this, it's an easy change). Please note that none of this requires retraining, it's just changing the end of the forward inference logic.

You're going to want to change this line (https://github.com/StanfordASL/Trajectron-plus-plus/blob/master/trajectron/model/mgcvae.py#L1142) from

return our_sampled_future

to

return self.latent.p_dist.probs, our_sampled_future

Then, you'll want to change https://github.com/StanfordASL/Trajectron-plus-plus/blob/master/trajectron/model/trajectron.py#L173 to handle the new outputs from the mgcvae model.

You'll want to make it look like

probs, predictions = model.predict(...)

predictions_np = predictions.cpu().detach().numpy()
future_dist_np = future_dist.cpu().detach().numpy()

# Assign predictions to node
for i, ts in enumerate(timesteps_o):
    if ts not in predictions_dict.keys():
        predictions_dict[ts] = dict()
        future_dist_dict[ts] = dict()

        predictions_dict[ts][nodes[i]] = np.transpose(predictions_np[:, [i]], (1, 0, 2, 3))
        future_dist_dict[ts][nodes[i]] = future_dist_np[i]


return future_dist_dict, predictions_dict

As for the evaluation metrics themselves, it might look like this (assuming preds is a Prediction object from the nuScenes challenge), feel free to change this code so that it works for your use case.

def eval_metrics(preds, gt_traj, k):
    available_k = min(preds.prediction.shape[0], k)
    rank_order = np.argsort(preds.probabilities)[::-1]
    ranked_predictions = preds.prediction[rank_order]
    
    topk_predictions = ranked_predictions[:available_k]
    
    de_k = np.linalg.norm(topk_predictions - gt_traj, axis=-1)
    ade_k = np.mean(de_k, axis=-1)
    argmin_ade_k = np.argmin(ade_k, axis=0)
    min_ade_k = ade_k[argmin_ade_k]
    
    fde_k = de_k[:, -1]
    argmin_fde_k = np.argmin(fde_k, axis=0)
    min_fde_k = fde_k[argmin_fde_k]
    
    return min_ade_k, min_fde_k