Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Workaround for mmap crash under Emscripten #20418

Merged
merged 1 commit into from
Dec 23, 2024

Conversation

georgestagg
Copy link
Contributor

When running Polars in Pyodide, using read_csv() (and possibly other functions reading from file) causes a crash during mmap -- a huge chunk of memory is allocated as the length of the file is incorrectly provided to mmap by the file's metadata.

This PR implements a workaround that applies only when running under Emscripten so that the file length is derived by seeking to the end of the file, and setting the mmap size explicitly in the options before mapping.

With this, things work under Pyodide again:

Screenshot 2024-12-23 at 13 01 09

Emscripten is currently working on reimplementing the virtual filesystem from scratch (WasmFS). Once that work has been merged and hits Pyodide, it's likely that we can remove this workaround.

cc @ritchie46

@github-actions github-actions bot added fix Bug fix python Related to Python Polars rust Related to Rust Polars labels Dec 23, 2024
Copy link

codecov bot commented Dec 23, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.99%. Comparing base (2ae01fb) to head (199b748).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #20418   +/-   ##
=======================================
  Coverage   78.98%   78.99%           
=======================================
  Files        1562     1562           
  Lines      220103   220103           
  Branches     2486     2486           
=======================================
+ Hits       173855   173863    +8     
+ Misses      45675    45667    -8     
  Partials      573      573           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ritchie46
Copy link
Member

Nice. I think I will follow up ensuring that all our mmap's go through a single utility function.

Note; the flaky test is unrelated.

@ritchie46 ritchie46 merged commit f8671e8 into pola-rs:main Dec 23, 2024
22 checks passed
@georgestagg georgestagg deleted the pyodide-read-csv branch December 23, 2024 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix Bug fix python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants