Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Parquet compression libraries on M1 macs #1589

Open
rcaudy opened this issue Nov 18, 2021 · 1 comment
Open

Support Parquet compression libraries on M1 macs #1589

rcaudy opened this issue Nov 18, 2021 · 1 comment
Assignees
Labels
feature request New feature or request triage
Milestone

Comments

@rcaudy
Copy link
Member

rcaudy commented Nov 18, 2021

Trying to write a compressed Parquet file elicits exceptions like:

[FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Mac and os.arch=aarch64
org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Mac and os.arch=aarch64
	at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:361)
	at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:195)
	at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:167)
	at org.xerial.snappy.Snappy.init(Snappy.java:69)
	at org.xerial.snappy.Snappy.<clinit>(Snappy.java:46)
	at org.apache.parquet.hadoop.codec.SnappyCompressor.compress(SnappyCompressor.java:67)
	at org.apache.hadoop.io.compress.CompressorStream.compress(CompressorStream.java:81)
	at org.apache.hadoop.io.compress.CompressorStream.finish(CompressorStream.java:92)
	at org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.compress(CodecFactory.java:167)
	at io.deephaven.parquet.ColumnWriterImpl.writeDictionaryPage(ColumnWriterImpl.java:125)
	at io.deephaven.parquet.ColumnWriterImpl.addDictionaryPage(ColumnWriterImpl.java:115)
	at io.deephaven.db.v2.parquet.ParquetTableWriter.writeColumnSource(ParquetTableWriter.java:499)
	at io.deephaven.db.v2.parquet.ParquetTableWriter.write(ParquetTableWriter.java:258)
	at io.deephaven.db.v2.parquet.ParquetTableWriter.write(ParquetTableWriter.java:197)
	at io.deephaven.db.v2.parquet.ParquetTableWriter.write(ParquetTableWriter.java:218)
	at io.deephaven.db.tables.utils.ParquetTools.writeParquetTableImpl(ParquetTools.java:672)
	at io.deephaven.db.tables.utils.ParquetTools.writeTable(ParquetTools.java:225)
	at io.deephaven.db.tables.utils.ParquetTools.writeTable(ParquetTools.java:141)
	at io.deephaven.db.tables.utils.TestParquetTools.testWriteTable(TestParquetTools.java:134)
[...]

We can workaround by changing the default codec to UNCOMPRESSED, by making this change in ParquetInstructions:

    private static volatile String defaultCompressionCodecName = CompressionCodecName.UNCOMPRESSED.toString();

We should find or build native libraries for this architecture, since it's relevant to a few of our devs as well as part of our target audience.

@rcaudy rcaudy added feature request New feature or request triage labels Nov 18, 2021
@rcaudy rcaudy added this to the Backlog milestone Jan 7, 2022
@rcaudy
Copy link
Member Author

rcaudy commented Jan 7, 2022

This is pending release of a newer parquet-mr library (1.13.0).

@rcaudy rcaudy self-assigned this Jan 7, 2022
mofojed pushed a commit that referenced this issue Oct 25, 2023
Release notes https://github.com/deephaven/web-client-ui/releases/tag/v0.51.0

# [0.51.0](deephaven/web-client-ui@v0.50.0...v0.51.0) (2023-10-24)


### Bug Fixes

* Adjusted Monaco "white" colors ([#1594](deephaven/web-client-ui#1594)) ([c736708](deephaven/web-client-ui@c736708)), closes [#1592](deephaven/web-client-ui#1592)
* cap width of columns with long names ([#1574](deephaven/web-client-ui#1574)) ([876a6ac](deephaven/web-client-ui@876a6ac)), closes [#1276](deephaven/web-client-ui#1276)
* Enabled pointer capabilities for Firefox in Playwright ([#1589](deephaven/web-client-ui#1589)) ([f440a38](deephaven/web-client-ui@f440a38)), closes [#1588](deephaven/web-client-ui#1588)
* Remove @deephaven/app-utils from @deephaven/dashboard-core-plugins dependency list ([#1596](deephaven/web-client-ui#1596)) ([7b59763](deephaven/web-client-ui@7b59763)), closes [#1593](deephaven/web-client-ui#1593)
* Tab in console input triggers autocomplete instead of indent ([#1591](deephaven/web-client-ui#1591)) ([fbe1e70](deephaven/web-client-ui@fbe1e70))


### Features

* Theming - Spectrum Provider ([#1582](deephaven/web-client-ui#1582)) ([a4013c0](deephaven/web-client-ui@a4013c0)), closes [#1543](deephaven/web-client-ui#1543)
* Theming Iris Grid ([#1568](deephaven/web-client-ui#1568)) ([ed8f4b7](deephaven/web-client-ui@ed8f4b7))
* web-client-ui changes required for deephaven.ui ([#1567](deephaven/web-client-ui#1567)) ([94ab25c](deephaven/web-client-ui@94ab25c))
* Widget plugins ([#1564](deephaven/web-client-ui#1564)) ([94cc82c](deephaven/web-client-ui@94cc82c)), closes [#1455](deephaven/web-client-ui#1455) [#1167](deephaven/web-client-ui#1167)


### BREAKING CHANGES

- `usePlugins` and `PluginsContext` were moved from
`@deephaven/app-utils` to `@deephaven/plugin`.
- `useLoadTablePlugin` was moved from `@deephaven/app-utils` to
`@deephaven/dashboard-core-plugins`.
- `useConnection` and `ConnectionContext` were moved from
`@deephaven/app-utils` to `@deephaven/jsapi-components`.
- `DeephavenPluginModuleMap` was removed from `@deephaven/redux`. Use
`PluginModuleMap` from `@deephaven/plugin` instead.
* Enterprise will need ThemeProvider for the css
variables to be available





Co-authored-by: deephaven-internal <deephaven-internal@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request triage
Projects
None yet
Development

No branches or pull requests

1 participant