Some problems with the evaluation on leaderboard

[Sou0602]

I was trying to submit the transfuser model to the CARLA Leaderboard, and the submission says it was pending as it uses computes and never progresses. I tested the image on a local docker container and found that run_evaluation.sh times out as the client cannot connect to the server. Normally, I run the server on a different terminal. When I try to do this on the container, make_docker.sh does not copy the ./CarlaUE4.sh to the image, so we don't have a usual way to run the CARLA server on the container. Can someone lend more details on how to connect/run the server on the docker container, or is if someone faced a similar issue?

[Kait0]

You simply run CARLA outside of the docker container.
The server can communicate with the client running inside the docker container.
You can see how to configure the docker start properly here.

[Sou0602]

I tried running the run_docker.sh in the tools folder of the transfuser directory. I get an error saying:
docker: Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /home/transfuser/results/. See 'docker run --help'.
There exists a folder already but the error says a source path does not exist.
I changed the --mount to -v and this is the error:
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

I am not sure how to get around this.

Another question I had is that, run_docker,sh just ensures the local version of the docker can be tested. But my submission to the leaderboard server ran for ~150 hours with a status of finished. However, I cannot see any values on the log and this is the data shown on leaderboard:
{ "values":[] "labels":[ 0: "Driving score" 1: "Route completion" 2: "Infraction penalty" 3: "Collisions pedestrians" 4: "Collisions vehicles" 5: "Collisions layout" 6: "Red light infractions" 7: "Stop sign infractions" 8: "Off-road infractions" 9: "Route deviations" 10: "Route timeouts" 11: "Agent blocked" ] "progress":[ 0: 4 1: 100 ] "icons":[ 0: "carla_camera" 1: "carla_camera" 2: "carla_camera" 3: "carla_imu" 4: "carla_gnss" 5: "carla_speedometer" 6: "carla_lidar" ] "entry_status":"Started" }

Are there are any suggestions/ similar problems you faced regarding this ?

[Kait0]

You might need to install nvidia support for docker, see here.

For problems with the online leaderboard you should talk to the organizers they will usually help you.
You can talk with Guillermo López, Joel Moriana or German Ros.
You can do that via their e-mail or the CARLA discord.

[Sou0602]

@Kait0 , thanks for clarifying that. I am now able to run the evaluation locally but keep running into other problems with the submission to leaderboard.