Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Streams implementation #296

Merged
merged 13 commits into from
Aug 24, 2023
Merged

Streams implementation #296

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
2da4734
wip ffi for arrow1
kylebarron Aug 7, 2023
ab7c571
add make_array
kylebarron Aug 7, 2023
7ad666b
Merge branch 'kyle/wip-arrow1-ffi'
H-Plus-Time Aug 22, 2023
359922e
Fix up From<FFIArrowRecordBatch>, shift create_reader into common/fetch
H-Plus-Time Aug 22, 2023
18f9cbb
Internal-only arrow1 equivalents of read_metadata_async, read_row_gro…
H-Plus-Time Aug 22, 2023
4690ad9
arrow1 ffi stream functional api + cargo fmt
H-Plus-Time Aug 22, 2023
7c598f3
Match ffi api of arrow2, get parseRecordBatch tests to pass for synch…
H-Plus-Time Aug 22, 2023
058f606
Satisfy ffi stream tests for arrow1
H-Plus-Time Aug 22, 2023
0dcc0b2
functional api for arrow2 ffi stream reads + tests
H-Plus-Time Aug 22, 2023
8051f22
Formatting!
H-Plus-Time Aug 22, 2023
35fac83
Fix feature flag combinations
H-Plus-Time Aug 23, 2023
2a2ab3d
Avoid build parallelism in CI (still highly useful for arrow1 wasm-op…
H-Plus-Time Aug 23, 2023
bdf582d
Bump node to v18 (ReadableStream global, purely for testing)
H-Plus-Time Aug 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ jobs:

- uses: actions/setup-node@v2
with:
node-version: "16"
node-version: "18"
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this require a Node 18+ for users as well? Or is this just for our tests? Is there a shim/polyfill that users can use for pre-18 versions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, turns out the ReadableStream constructor only showed up in Node 18. There is indeed a poly/ponyfill, written by a familar face: MattiasBuelens/web-streams-polyfill.

That does bring up a good point - overall README level documentation, and doc-strings for the new exports. On that note, the codeblocks embedded in rust doc comments, do those show up with syntax highlighting for embedded code fragments (a bit like jsdocs) in editors for you?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah they show up in editors and on the docs site https://kylebarron.dev/parquet-wasm/modules/bundler_arrow1.html#readParquet


- name: Build bundle
run: yarn build:test
Expand Down
56 changes: 54 additions & 2 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ async = [
"dep:futures",
"dep:range-reader",
"dep:reqwest",
"dep:wasm-streams",
"dep:async-compat",
"dep:async-stream",
"parquet?/async",
]
debug = ["console_error_panic_hook", "clap"]

Expand Down Expand Up @@ -75,6 +79,7 @@ parquet2 = { version = "0.17", default_features = false, optional = true }

arrow = { version = "45.0", default-features = false, optional = true, features = [
"ipc",
"ffi",
] }
parquet = { version = "45.0", default-features = false, optional = true, features = [
"arrow",
Expand All @@ -94,6 +99,9 @@ reqwest = { version = "0.11.19", optional = true, default-features = false }

# Pass "wasm" and "thin" down to the transitive zstd dependency
zstd = { version = "*", features = ["wasm", "thin"], default-features = false, optional = true }
async-compat = { version = "0.2.1", optional = true }
async-stream = { version = "0.3.5", optional = true }
wasm-streams = { version = "0.3.0", optional = true }

[dependencies.web-sys]
version = "0.3.4"
Expand Down
5 changes: 4 additions & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,21 @@
"test": "ts-node node_modules/tape/bin/tape ./tests/js/index.ts"
},
"devDependencies": {
"@fastify/static": "^6.10.2",
"@types/node": "^17.0.21",
"@types/tape": "^4.13.2",
"apache-arrow": "^12.0.0",
"arrow-js-ffi": "0.3.0",
"benny": "^3.7.1",
"fastify": "^4.21.0",
"gh-pages": "^3.2.3",
"tape": "^5.5.2",
"ts-node": "^10.7.0",
"typedoc": "^0.22.13",
"typescript": "^4.6.2"
},
"volta": {
"node": "16.17.0"
"node": "18.17.0",
"yarn": "1.22.19"
}
}
12 changes: 9 additions & 3 deletions scripts/build.sh
100644 → 100755
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,9 @@ wasm-pack build \
--out-dir tmp_build/node \
--out-name arrow1 \
--target nodejs \
$FLAGS
--features async \
$FLAGS &
[ -n "$CI" ] && wait;
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome; it parallelizes each build? Should we do this for arrow2 as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, turning the parallel behaviour off in CI took a bit of bash-fu (conditionally backgrounding a command isn't possible, but nobody said anything about conditionally running wait 😉 ).

Not much point for arrow2 (the wasm-opt step for arrow2 builds is quite short, while the compile step is highly parallelized, so trying to run that in parallel ended up slowing things down).

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder why wasm-opt would be so much slower for arrow1. I don't think this used to be the case; I only recently noticed it taking longer


# Build web version into tmp_build/esm
echo "Building arrow-rs esm"
Expand All @@ -28,7 +30,9 @@ wasm-pack build \
--out-dir tmp_build/esm \
--out-name arrow1 \
--target web \
$FLAGS
--features async \
$FLAGS &
[ -n "$CI" ] && wait;

# Build bundler version into tmp_build/bundler
echo "Building arrow-rs bundler"
Expand All @@ -37,7 +41,9 @@ wasm-pack build \
--out-dir tmp_build/bundler \
--out-name arrow1 \
--target bundler \
$FLAGS
--features async \
$FLAGS &
wait

######################################
# ARROW 2 turn on the feature manually
Expand Down
11 changes: 11 additions & 0 deletions src/arrow1/error.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ pub enum ParquetWasmError {

#[error(transparent)]
ParquetError(Box<ParquetError>),

#[cfg(feature = "async")]
#[error("HTTP error: `{0}`")]
HTTPError(Box<reqwest::Error>),
}

pub type Result<T> = std::result::Result<T, ParquetWasmError>;
Expand All @@ -26,3 +30,10 @@ impl From<ParquetError> for ParquetWasmError {
Self::ParquetError(Box::new(err))
}
}

#[cfg(feature = "async")]
impl From<reqwest::Error> for ParquetWasmError {
fn from(err: reqwest::Error) -> Self {
Self::HTTPError(Box::new(err))
}
}
Loading
Loading