-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(python)!: Implement binary serialization of LazyFrame/DataFrame/Expr and set it as the default format #17223
Conversation
py-polars/src/lazyframe/mod.rs
Outdated
fn deserialize_binary(py_f: PyObject) -> PyResult<Self> { | ||
let file = get_file_like(py_f, false)?; | ||
let reader = BufReader::new(file); | ||
let lp = ciborium::from_reader::<DslPlan, _>(reader) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ciborium also offers a from_reader_with_buffer
which allows finetuning the size of the buffer (default is 4K). I don't really know what size data we're usually looking at, so leaving this at default for now.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #17223 +/- ##
==========================================
+ Coverage 80.82% 80.84% +0.01%
==========================================
Files 1466 1470 +4
Lines 192313 192381 +68
Branches 2745 2750 +5
==========================================
+ Hits 155446 155538 +92
+ Misses 36364 36335 -29
- Partials 503 508 +5 ☔ View full report in Codecov by Sentry. |
Closes #17108
Changes
format
parameter to theserialize/deserialize
methods onDataFrame/LazyFrame/Expr
with the options"json"
(existing) and"binary"
(new)."binary"
as the default format. This is a breaking change. You can retain the previous behavior by specifyingformat="json"
Example
Before:
After: