You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Motivation: Parquet and Arrow are chunked formats. Therefore we shouldn't need to wait for the entire dataset to load/parse before getting some data back.
However I'm still not aware of a way to return an iterable or an async iterable from rust to js. To get around this, I think we can "drive" the iteration from JS. Essentially this:
import*aswasmfrom'parquet-wasm';constarr=newUint8Array();// Parquet bytes// name readSchema to align with pyarrow api?constparquetFile=newwasm.ParquetFile(arr);constschemaIPC=parquetFile.schema();for(leti=0;i<parquetFile.numRowGroups;i++){constrecordBatchIPC=parquetFile.readRowGroup(i);}
And ideally we'll have an async version of this too
The text was updated successfully, but these errors were encountered:
Motivation: Parquet and Arrow are chunked formats. Therefore we shouldn't need to wait for the entire dataset to load/parse before getting some data back.
However I'm still not aware of a way to return an iterable or an async iterable from rust to js. To get around this, I think we can "drive" the iteration from JS. Essentially this:
And ideally we'll have an async version of this too
The text was updated successfully, but these errors were encountered: