Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: handle user-defined types in Isthmus #149

Merged

Conversation

vbarua
Copy link
Member

@vbarua vbarua commented May 23, 2023

BREAKING CHANGE: TypeConverter no longer uses static methods
BREAKING CHANGE: SimpleExtension.MAPPER has been replaced
with SimpleExtension.objectMapper(String namespace)

@vbarua vbarua force-pushed the vbarua/isthmus-user-defined-type-handling branch 2 times, most recently from ffbbd4f to b8a4561 Compare May 23, 2023 21:43
@vbarua
Copy link
Member Author

vbarua commented May 23, 2023

Change Summary

Updated the TypeConverter to accept a UserTypeMapper, an interface which implements type conversions between Substrait and Calcite. The existing code made heavy use of the static TypeConverter.convert method, which didn't work well for allowing users to include their own UserTypeMapper. I've updated all call sites to use an instance of TypeConverter, and most of the this code in this PR is for propagating the TypeConverter through the system.

As part of this change, I renamed convert to toSubstrait and toCalcite for improved clarity (the original method had an overload for each side of the conversion).

Additionally, I've simplified the resolution of URIs for types by injecting the URI directly into the ObjectMapper. This came up because types in FuncArgs were being created without URIs, and propagating the resolution to them was harder than just getting rid of it entirely.

See CustomFunctionTest.java for an example of how all the parts fits together.

@vbarua vbarua changed the title feat: handle user-defined types in Isthmus feat!: handle user-defined types in Isthmus May 23, 2023
@vbarua vbarua force-pushed the vbarua/isthmus-user-defined-type-handling branch from b8a4561 to 1cdd1d1 Compare May 23, 2023 22:00
.addAllTypes(
extensionSignatures.types().stream()
.map(t -> t.resolve(namespace))
.collect(java.util.stream.Collectors.toList()))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eventually, I would like to remove the need for function resolution as well.

return convert(type, typeExpression, null);
public RelDataType toCalcite(
RelDataTypeFactory relDataTypeFactory, TypeExpression typeExpression) {
return toCalcite(relDataTypeFactory, typeExpression, null);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This takes a RelDataTypeFactory, rather than bundling it in the TypeConverter itself, because not everywhere where you would want to build a TypeConverter has easy access to one of these factories, but everywhere you want to call toCalcite does.

* Utility factory for creating user-defined types implementing {@link RelDataTypeImpl}, correctly.
* Used for creating types as part of testing.
*/
public class UserTypeFactory {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a lot of boilerplate to implementing a RelDataType in Calcite, so I made a factory for them (for testing only).

@vbarua vbarua marked this pull request as ready for review May 23, 2023 22:18
Copy link
Collaborator

@jacques-n jacques-n left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confession, this is a bit of a rubberstamp. Feels like it is better to get this in and iterate if we figure out it needs improvements later. It has been hanging open too long.

@jacques-n jacques-n merged commit 7d7acf8 into substrait-io:main Jun 6, 2023
@vbarua
Copy link
Member Author

vbarua commented Jun 6, 2023

Feels like it is better to get this in and iterate if we figure out it needs improvements later.

On my end, I'm more than happy to make any fixes and improvements as the need arises 🙇

ajegou pushed a commit to ajegou/substrait-java that referenced this pull request Mar 29, 2024
* feat: user-defined type handling
* test: user-defined type conversion
* refactor: inject URI into extension parser
* refactor: inject URI into type string parser
* test: verify custom types in functions roundtrip

BREAKING CHANGE: TypeConverter no longer uses static methods
BREAKING CHANGE: SimpleExtension.MAPPER has been replaced with SimpleExtension.objectMapper(String namespace)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants