Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixed compilation errors #1

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+10 −9
Open

Fixed compilation errors #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a83f6d8
Try fixing errors due to new polars/arrows version (?!)
Jun 4, 2024
c6ff4e0
Removed arrow as dependency
Jun 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions extendr-polars/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ description = "extendr bindings to polars"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
polars = { version = "*", default_features = false, features = ["dtype-struct"]}
polars-core = { version = "*", default_features = false }
Comment on lines -13 to -14
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes are needed?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cargo warned me that this parameter is not available for polars & polars_core. To be honest: I am unsure about the implications of these changes and just wanted to remove compiler warnings along the way.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this change is ideal here because the default functionality includes extra stuff.

polars = { version = "*", features = ["dtype-struct"]}
polars-core = { version = "*" }
polars-plan = { version = "*", default_features = false, optional = true }
polars-lazy = { version = "*", default_features = false, optional = true }
thiserror = "1"
Expand Down
2 changes: 1 addition & 1 deletion extendr-polars/src/export_dataframe.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ pub fn to_rpolars_dataframe(df: pl::DataFrame) -> EResult<Robj> {
// safety: requires a valid array stream pointer, robj_str_ref
unsafe fn export_df_as_stream(df: pl::DataFrame, robj_str_ref: &Robj) -> EResult<()> {
let stream_ptr = rptr::robj_str_ptr_to_usize(robj_str_ref)? as *mut ffi::ArrowArrayStream;
let schema = df.schema().to_arrow();
let schema = df.schema().to_arrow(true);
let data_type = pl::ArrowDataType::Struct(schema.fields);
let field = pl::ArrowField::new("", data_type, false);
let iter_boxed = Box::new(odi::OwnedDataFrameIterator::new(df));
Expand Down
13 changes: 7 additions & 6 deletions extendr-polars/src/odi.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
// OwnedDataFrameIterator written by @paleolimbot / nanoarrow in PR for pola-rs/r-polars

use polars::error::PolarsError;
use polars_core::utils::arrow;
use polars_core::utils::arrow::datatypes::DataType as ADataType;
use polars_core::utils::arrow::datatypes::ArrowDataType as ADataType;
use polars_core::utils::arrow::array::Array;

pub struct OwnedDataFrameIterator {
columns: Vec<polars::series::Series>,
Expand All @@ -12,7 +14,7 @@ pub struct OwnedDataFrameIterator {

impl OwnedDataFrameIterator {
pub fn new(df: polars::frame::DataFrame) -> Self {
let schema = df.schema().to_arrow();
let schema = df.schema().to_arrow(true);
let data_type = ADataType::Struct(schema.fields);
let vs = df.get_columns().to_vec();
Self {
Expand All @@ -25,20 +27,19 @@ impl OwnedDataFrameIterator {
}

impl Iterator for OwnedDataFrameIterator {
type Item = std::result::Result<Box<dyn arrow::array::Array>, arrow::error::Error>;
type Item = std::result::Result<Box<dyn arrow::array::Array>, PolarsError>;

fn next(&mut self) -> Option<Self::Item> {
if self.idx >= self.n_chunks {
None
} else {
// create a batch of the columns with the same chunk no.
let batch_cols = self.columns.iter().map(|s| s.to_arrow(self.idx)).collect();
let batch_cols: Vec<Box<dyn Array>> = self.columns.iter().map(|s| s.to_arrow(self.idx, true)).collect();
self.idx += 1;

let chunk = polars::frame::ArrowChunk::new(batch_cols);
let array = arrow::array::StructArray::new(
self.data_type.clone(),
chunk.into_arrays(),
batch_cols,
std::option::Option::None,
);
Some(std::result::Result::Ok(Box::new(array)))
Expand Down