-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor crate structure #241
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Remove dependency on vegafusion-sql from vegafusion-runtime
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes a significant refactor of the VegaFusion crate structure. Here are the new crates:
Crates
vegafusion-common
A subset of the functionality of
vegafusion-core
was extracted intovegafusion-common
. This includes the error struct and base data structures. This is now a dependency ofvegafusion-core
and the newvegafusion-dataframe
crates.vegafusion-dataframe
The
DataFrame
andConnection
traits have been pulled out into avegafusion-dataframe
crate. This is used as the dependency ofvegafusion-runtime
, this way there is no hard dependency onSqlDataFrame
andSqlConnection
.vegafusion-sql
This crate provides the
SqlConnection
andSqlDataFrame
structs with implement theConnection
andDataFrame
traits from thevegafusion-dataframe
crate using SQL. The functionality for generating SQL string across dialects is always available in the crate. Optional support for evaluating the queries is enabled by feature flags with a-conn
suffix.The
datafusion-conn
andsqlite-conn
feature flags are currently supported.vegafusion-datafusion-udfs
This crate contains the definitions of the DataFusion UDFs that are used to implement select Vega expression functions and transforms. These UDFs are used in two places.
DataFusionConnection
provided byvegafusion-sql
adds these UDFs to itsSessionContext
so that they are available for use in SQL querires.vegafusion-runtime
crate uses these UDFs for the evaluation of signal expressions and for simplifying expressions passed to thefilter
andformula
transforms. Note: Even when a non-DataFusion Connection is used, DataFusion is still used for signal evaluation and expression simplification.vegafusion-runtime
This crate was renamed from
vegafusion-rt-datafusion
tovegafusion-runtime
. Along with this, theTaskGraphRuntime
struct was renamed toVegaFusionRuntime
.Other refactoring
The
VegaFusionRuntime
struct now accepts aArc<dyn Connection>
in the constructor. This defines the connection that the runtime uses by default. Currentlyvegafusion-python-embed
sets this toDataFusionConnection
.This change made it possible to remove the dependency on
datafusion-core
fromvegafusion-runtime
. Now, the only time we depend on all of datafusion is when thedatafusion-conn
feature flag is enabled on thevegafusion-sql
crate.Motivation
This work lays the foundation for several exciting possibilities:
vegafusion-sql
, and then used invegafusion-runtime
with no changes to that crate.DataFrame
andConnection
traits can be developed. e.g.vegafusion-polars
,vegafusion-substrait
, etc.