Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new "Air Traffic Data" example #5449

Merged
merged 21 commits into from
Nov 27, 2024
Merged

Conversation

abey79
Copy link
Member

@abey79 abey79 commented Mar 10, 2024

What

This example demonstrates multiple aspects of the Rerun viewer:

  • Use of the map view.
  • Use of pyproj to transform geospatial data from one coordinate system to another.
  • Use GeoPandas to load geospatial data into a 3D view.
  • Use Polars to batch data to be sent via rr.send_columns() (use --batch).
pixi run py-build-examples && pixi run -e examples air_traffic_data

Checklist

  • I have read and agree to Contributor Guide and the Code of Conduct
  • I've included a screenshot or gif (if applicable)
  • I have tested the web demo (if applicable):
  • The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG
  • If applicable, add a new check to the release checklist!

Copy link

github-actions bot commented Apr 10, 2024

Deployed docs

Commit Link
02b2ab1 https://landing-1pfgfnhvx-rerun.vercel.app/docs

@Wumpf
Copy link
Member

Wumpf commented Nov 15, 2024

@abey79 the reason we don't have this in yet is that it simply is too bad of a user experience, right?

@abey79
Copy link
Member Author

abey79 commented Nov 15, 2024

@abey79 the reason we don't have this in yet is that it simply is too bad of a user experience, right?

Yes. It's here to remind us that we need to improve our "lots of entities" story :)

@teh-cmc teh-cmc force-pushed the antoine/air-traffic-data-example branch from 02b2ab1 to 6dc2c7c Compare November 21, 2024 16:23
Copy link

github-actions bot commented Nov 21, 2024

Latest documentation preview deployed successfully.

Result Commit Link
296054b https://landing-1x8st44fh-rerun.vercel.app/docs

Note: This comment is updated whenever you push a commit.

Copy link

github-actions bot commented Nov 21, 2024

Web viewer built successfully. If applicable, you should also test it:

  • I have tested the web viewer
Result Commit Link
296054b https://rerun.io/viewer/pr/5449

Note: This comment is updated whenever you push a commit.

@emilk
Copy link
Member

emilk commented Nov 21, 2024

I would love to get this merged as a benchmark, even though we're not hitting the benchmark yet.

@abey79
Copy link
Member Author

abey79 commented Nov 21, 2024

I would love to get this merged as a benchmark, even though we're not hitting the benchmark yet.

May I interest you in #8163? This is a blocker for the --batch logging strategy of this example. Which is interesting to have along side the "regular" (aka rr.log()) logging pathway for ingestion benchmark (aka why is shuffling data in python faster than our compactor).

Then I can further dust it off and try to push it to mergeable state.

@abey79 abey79 marked this pull request as ready for review November 26, 2024 15:29
@abey79
Copy link
Member Author

abey79 commented Nov 26, 2024

As of today on my machine, using the 2h dataset, I get the following ingestion times:

  • with --batch, I get 45s of ingestion time, plus 3-4s of additional latency in the viewer
  • without --batch, I get 1min20s of ingestion tine, plus ~35s of additional latency in the viewer

I'm probably missing something, but our batches (SDK-side and viewer-side) should probably do a better job at beating my home-grown, Polars-based batching code.

@teh-cmc teh-cmc self-requested a review November 27, 2024 13:47
examples/manifest.toml Outdated Show resolved Hide resolved
examples/python/air_traffic_data/air_traffic_data.py Outdated Show resolved Hide resolved
examples/python/air_traffic_data/air_traffic_data.py Outdated Show resolved Hide resolved
@teh-cmc teh-cmc force-pushed the antoine/air-traffic-data-example branch from 2327627 to 9fa098e Compare November 27, 2024 14:50
@teh-cmc teh-cmc merged commit 0493fc8 into main Nov 27, 2024
36 of 37 checks passed
@teh-cmc teh-cmc deleted the antoine/air-traffic-data-example branch November 27, 2024 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
examples Issues relating to the Rerun examples include in changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants