Skip to content

Implementation and Knowledge Graphs of the ICCV 2023 workshop paper "nuScenes Knowledge Graph - A comprehensive semantic representation of traffic scenes for trajectory prediction"

License

Notifications You must be signed in to change notification settings

boschresearch/nuScenes_Knowledge_Graph

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
 
 
 
 
 
 

Repository files navigation

nuScenes_Knowledge_Graph

Ontologies and Knowledge Graphs of the ICCV 2023 workshop paper:

  • nSKG (nuScenes Knowledge Graph): knowledge graph for the nuScenes dataset, that models all scene participants and road elements, as well as their semantic and spatial relationships
  • nSTP (nuScenes Trajectory Prediction Graph): heterogeneous graph of the nuScenes dataset for trajectory prediction in PyTorch Geometric (PyG) format. It extends nSKG for example by transformation into agents' local coordinate systems, relevant agent extraction, semantic relationships between agents.
  • nuScenes_agent_onto: ontology for the traffic participants (agents)
  • nuScenes_map_onto: ontology for the extended map
  • stardog_rules: SPARQL rules for mapping nuScenes concepts to the agent ontology

The content is available at Zenodo

nuScenes ontology excerpt

How to use

nSKG represents the KG created on the basis of the nuScenes ontologies nuScenes_agent_onto.ttl and nuScenes_map_onto.ttl and materializing the nuScenes annotation dataset. It can be used for applications where relational information between entities are important. The ontologies are in Turtle format and can be viewed by ontology editors such as Protege.

nSTP represents the extended version of nSKG, where agents are represented in local coordinate systems to enforce shift- and rotation-invariance. It also includes semantic relationships between agents, e.g. whether agents are on neighboring lanes, the same lane or might intersect. This is done based on the semantic scene graph describe in (Towards Traffic Scene Description: The Semantic Scene Graph). The data is provided in PyTorch Geometric format and directly be used to train graph neural network for trajectory prediction.

Publications using the nuScenes Knowledge Graph

  • Sun, Z., Wang, Z., Halilaj, L., & Luettin, J. (2024). SemanticFormer: Holistic and Semantic Traffic Scene Representation for Trajectory Prediction using Knowledge Graphs. To appear in IEEE Robotics and Automation Letters. Preprint: ArXiv, abs/2404.19379.
  • Wang, Z., Sun, Z., Luettin, J., & Halilaj, L. (2024). SocialFormer: Social Interaction Modeling with Edge-enhanced Heterogeneous Graph Transformers for Trajectory Prediction. ArXiv, abs/2405.03809.
  • (please let us know if you like us to add your article here)

Related work

Acknowledgements

Special thanks to Motional for the permission to distribute this modified version of the nuScenes dataset.

Citation

If you use this work please cite

@inproceedings{
title={nuScenes Knowledge Graph - A comprehensive semantic representation of traffic scenes for trajectory prediction},
author={Leon Mlodzian and Zhigang Sun and Hendrik Berkemeyer and Sebastian Monka and Zixu Wang and Stefan Dietze and Lavdim Halilaj and Juergen Luettin},
booktitle={International Conference on Computer Vision (ICCV), Workhsop on Scene Graphs and Graph Representation Learning (SG2RL)},
year={2023}
}

License

Shield: CC BY-NC-SA 4.0

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, with additional term described by the nuScenes Terms of use, in particular the "Licenses" section.

CC BY-NC-SA 4.0

About

Implementation and Knowledge Graphs of the ICCV 2023 workshop paper "nuScenes Knowledge Graph - A comprehensive semantic representation of traffic scenes for trajectory prediction"

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published