-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update index.ts to support RLE_DICTIONARY #112
Conversation
@@ -1,5 +1,5 @@ | |||
export * as PLAIN from './plain' | |||
export * as RLE from './rle' | |||
export * as PLAIN_DICTIONARY from './plain_dictionary' | |||
|
|||
export * as RLE_DICTIONARY from './plain_dictionary' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Turns out the plain_dictionary has support for both!
// Tracked in https://github.com/LibertyDSNP/parquetjs/issues/113 | ||
it.skip('rle_boolean_encoding.parquet loads', async function() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added this in a separate issue as it still fails.
test/test-files.js
Outdated
it('rle-dict-snappy-checksum.parquet loads', async function() { | ||
const data = await readData('rle/rle-dict-snappy-checksum.parquet'); | ||
assert.deepEqual(data[0],{ binary_field: "c95e263a-f5d4-401f-8107-5ca7146a1f98", long_field: 0 }); | ||
}); | ||
|
||
it('rle-dict-uncompressed-corrupt-checksum.parquet loads', async function() { | ||
const data = await readData('rle/rle-dict-uncompressed-corrupt-checksum.parquet'); | ||
assert.deepEqual(data[0],{ binary_field: "6325c32b-f417-41aa-9e02-9b8601542aff", long_field: 0 }); | ||
}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reference files from https://github.com/apache/parquet-testing work!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you!
Problem
problem statement - when trying to read a parquet file that was generated using V2 parquet and had RLE_DICTIONARY, got an error: invalid encoding: RLE_DICTIONARY #96
Reported issue: #96
Solution
What I/we did to solve this problem
added: export * as RLE_DICTIONARY from './plain_dictionary';
with @pairperson1
Change summary:
Steps to Verify:
I added this line to an existing project in the node modules and it works. without this line I get an an error with this line added - it passed