Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Core: Add View support for REST catalog #7913

Merged
merged 6 commits into from
Dec 5, 2023
Merged

Conversation

nastra
Copy link
Contributor

@nastra nastra commented Jun 26, 2023

No description provided.

@nastra nastra marked this pull request as draft June 26, 2023 13:12
@nastra nastra force-pushed the rest-view-support branch 3 times, most recently from 16eb260 to d2d5be6 Compare June 26, 2023 14:52
@nastra nastra force-pushed the rest-view-support branch 2 times, most recently from 3d01bb9 to 626dedc Compare November 2, 2023 10:02
@nastra nastra force-pushed the rest-view-support branch 5 times, most recently from 9905345 to 9fb62ee Compare November 8, 2023 15:14
@@ -257,8 +258,8 @@ public Builder addVersion(ViewVersion version) {
return this;
}

private int addVersionInternal(ViewVersion version) {
int newVersionId = reuseOrCreateNewViewVersionId(version);
private int addVersionInternal(ViewVersion newVersion) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've renamed this parameter to make the code slightly easier to understand, because we have a few cases in this method where we re-assign either the schema id or the view version id

@@ -247,8 +247,9 @@ public void createViewErrorCases() {
.withQuery(trino.dialect(), trino.sql())
.withQuery(trino.dialect(), trino.sql())
.create())
.isInstanceOf(IllegalArgumentException.class)
.hasMessage("Invalid view version: Cannot add multiple queries for dialect trino");
.isInstanceOf(Exception.class)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: I think I pointed this out before and may have forgotten the answer, but shouldn't this be a consistent exception class every time?

Copy link
Contributor Author

@nastra nastra Dec 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember I replied to this but somehow can't find where it was.
The issue with the REST version is unfortunately that it fails with org.apache.iceberg.exceptions.BadRequestException: Malformed request: Invalid view version: Cannot add multiple queries for dialect trino because we're re-applying all changes to the Builder in CatalogHandlers.createView(...):

request.viewVersion().representations().stream()
.filter(SQLViewRepresentation.class::isInstance)
.map(SQLViewRepresentation.class::cast)
.forEach(r -> viewBuilder.withQuery(r.dialect(), r.sql()));
View view = viewBuilder.create();

That IAE is then converted to a BadRequestException in

case 400:
throw new BadRequestException("Malformed request: %s", error.message());

So I think what we could do is to have

if (IllegalArgumentException.class.getSimpleName().equals(error.type())) {
        throw new IllegalArgumentException(error.message());
}

in the DefaultErrorHandler. Thoughts?

Copy link
Contributor

@rdblue rdblue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is nearly ready but the concurrent test case actually refreshes correctly and doesn't test the concurrent case with server-side retry and ID reassignment. I made a suggestion in the comment that I think will fix the test issue.

This will probably also require a flag in ViewCatalogTests for when server-side retry is possible like in table tests. If we're testing it correctly, then the InMemoryCatalog will fail.

@rdblue rdblue merged commit f19643a into apache:main Dec 5, 2023
47 checks passed
@rdblue
Copy link
Contributor

rdblue commented Dec 5, 2023

Thanks @nastra! Great to have this in.

@nastra nastra deleted the rest-view-support branch December 5, 2023 17:03
lisirrx pushed a commit to lisirrx/iceberg that referenced this pull request Jan 4, 2024
devangjhabakh added a commit to cdouglas/iceberg that referenced this pull request Apr 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

5 participants