Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to DataFusion 36 #457

Merged
merged 46 commits into from
Feb 26, 2024
Merged

Update to DataFusion 36 #457

merged 46 commits into from
Feb 26, 2024

Conversation

jonmmease
Copy link
Collaborator

@jonmmease jonmmease commented Feb 12, 2024

This ended up being a fairly large PR to update DataFusion to version 36.0.0 and arrow-rs to version 50.0.0. We're updating from version 32.0.0, and a lot has changed!

@jonmmease jonmmease marked this pull request as draft February 12, 2024 13:46
@jonmmease
Copy link
Collaborator Author

jonmmease commented Feb 13, 2024

VegaFusion tests are passing now. There are a couple of CI issues:

  1. The python duckdb tests are failing due to a failure to load an extension. Duckdb 0.10.0 was released today, so this will likely get ironed out.
  2. Building wheel fails on linux aarch64, with a failure to build ring. This ring GitHub issue might be relevant: Cannot cross-compile to x86_64-unknown-linux-gnu from Apple M1  briansmith/ring#1605. There is a suggestion for some compilation flags to use, and a report that it works with version 0.17.5 (currently trying to use v 0.17.3).

Prior to this PR, we were on ring 0.16.20. Oh, this looks directly relevant: briansmith/ring#1728. We may need to bump manylinux past 2014.

@jonmmease jonmmease changed the title WIP update to DataFusion 35+ WIP update to DataFusion 36 Feb 17, 2024
Copy link

semgrep-app bot commented Feb 24, 2024

Legal Risk
The following dependencies were released under a license that is currently prohibited by your organization. Merging is blocked until this is resolved.
Recommendation:
Reach out to your security team or Semgrep admin to address this issue. In special cases, exceptions may be made for dependencies with violating licenses, however, the general recommendation is to avoid using a dependency under such a license

AGPL-3.0
dssim 3.3.1
dssim-core 3.2.8

Copy link

semgrep-app bot commented Feb 24, 2024

Legal Risk
The following dependencies were released under a license that has been flagged by your organization for consideration.
Recommendation:
While merging is not directly blocked, it's best to pause and consider what it means to use this license before continuing. If you are unsure, reach out to your security team or Semgrep admin to address this issue.

MPL-2.0
webpki-roots 0.25.4

Copy link

semgrep-app bot commented Feb 24, 2024

Legal Risk
The following dependencies were released under a license that is currently prohibited by your organization. Merging is blocked until this is resolved.
Recommendation:
Reach out to your security team or Semgrep admin to address this issue. In special cases, exceptions may be made for dependencies with violating licenses, however, the general recommendation is to avoid using a dependency under such a license

AGPL-3.0
dssim 3.3.1
dssim-core 3.2.8

Copy link

semgrep-app bot commented Feb 26, 2024

Legal Risk
The following dependencies were released under a license that is currently prohibited by your organization. Merging is blocked until this is resolved.
Recommendation:
Reach out to your security team or Semgrep admin to address this issue. In special cases, exceptions may be made for dependencies with violating licenses, however, the general recommendation is to avoid using a dependency under such a license

AGPL-3.0
dssim 3.3.1
dssim-core 3.2.8

Copy link

semgrep-app bot commented Feb 26, 2024

Legal Risk
The following dependencies were released under a license that is currently prohibited by your organization. Merging is blocked until this is resolved.
Recommendation:
Reach out to your security team or Semgrep admin to address this issue. In special cases, exceptions may be made for dependencies with violating licenses, however, the general recommendation is to avoid using a dependency under such a license

AGPL-3.0
dssim 3.3.1
dssim-core 3.2.8

@jonmmease
Copy link
Collaborator Author

Legal Risk The following dependencies were released under a license that is currently prohibited by your organization. Merging is blocked until this is resolved. Recommendation: Reach out to your security team or Semgrep admin to address this issue. In special cases, exceptions may be made for dependencies with violating licenses, however, the general recommendation is to avoid using a dependency under such a license

AGPL-3.0 dssim 3.3.1 dssim-core 3.2.8

This was only used in testing (not part of the production build), but to simplify things I switch to the pixelmatch crate (with MIT license) instead of dssim for testing.

@jonmmease jonmmease marked this pull request as ready for review February 26, 2024 15:22
@jonmmease jonmmease changed the title WIP update to DataFusion 36 Update to DataFusion 36 Feb 26, 2024
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this UDF in favor of the built-in DataFusion make_list function

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this UDF in favor of DataFusion's built-in array_length and character_length functions

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced with DataFusion's built-in isnan function

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to drop our custom span UDF in favor of DataFusion transforms. I think I got it working, but DataFusion didn't seem to be able to simplify array expressions, which is what we use for scalar evaluation.

@jonmmease jonmmease merged commit 2781aad into main Feb 26, 2024
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant