Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Attempt to support arrow serialization #119

Closed
wants to merge 6 commits into from

Conversation

jonmmease
Copy link
Collaborator

This PR is an experiment towards supporting base64-encoded arrow IPC data. The idea is to support data URLs of the form:

data:application/vnd.apache.arrow.file;base64,//////gAA...

Then convert the base64 data to binary and parse using the arrow JavaScript library. Then pass the arrow Table as the values property of the dataset. Arrow Tables support the array-of-objects format that Vega uses, so this mostly just works. But I've run into a few issues:

  • Arrow Int columns are converted to JavaScript BigInt values, which don't play well with Vega
  • Arrow Tables don't support mutation, and in one example I tried there is a mutation error raised (the example is at vl-convert-rs/tests/vl-specs/circle_binned_b64.vl.json).
test test_png_no_theme::test::case_02 ... ERROR TypeError: 'set' on proxy: trap returned falsish for property 'bin_maxbins_10_IMDB_Rating'
    at https://cdn.skypack.dev/-/vega-transforms@v4.10.2-ZwuxbjLubhdUwIKHKtlL/dist=es2020,mode=imports,min/optimized/vega-transforms.js:1:11242
    at Array.forEach (<anonymous>)

Given these issues, I'm not sure it makes sense to move forward with this right now. But wanted to document my findings.

Update vendoring to traverse vendor directory so that we get all files, not just those in import_map.json.
test test_png_no_theme::test::case_1 ... ERROR TypeError: 'set' on proxy: trap returned falsish for property 'bin_maxbins_10_Rotten_Tomatoes_Rating'
    at https://cdn.skypack.dev/-/vega-transforms@v4.10.2-ZwuxbjLubhdUwIKHKtlL/dist=es2020,mode=imports,min/optimized/vega-transforms.js:1:11242
    at Array.forEach (<anonymous>)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant