Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-43680: [Integration] Unskip nanoarrow in IPC integration tests #43715

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

bkietz
Copy link
Member

@bkietz bkietz commented Aug 15, 2024

Rationale for this change

Nanoarrow can now read and write IPC files as of apache/arrow-nanoarrow#585 so it should no longer be skipped as a producer/consumer

What changes are included in this PR?

Nanoarrow's tester is updated to point to the new integration executable and to report nanoarrow as a consumer/producer of IPC files.

Notably the null_trivial case is skipped even though nanoarrow nominally supports it since it represents a corner case in which nanoarrow's flatbuffers library will not accept some vectors produced by other flatbuffers libraries dvidelabs/flatcc#287

Are these changes tested?

Yes

Are there any user-facing changes?

No

@github-actions github-actions bot added the awaiting committer review Awaiting committer review label Aug 15, 2024
Copy link
Member

@pitrou pitrou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like there are a lot of nanoarrow-related failures in the Integration build

@@ -1876,7 +1876,8 @@ def _temp_path():

generate_null_case([10, 0]),

generate_null_trivial_case([0, 0]),
generate_null_trivial_case([0, 0])
.skip_tester('nanoarrow'),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add issue references for these skips?

@bkietz bkietz force-pushed the nanoarrow-integration-tests branch from 8cce89d to 94dd868 Compare August 30, 2024 19:56
@paleolimbot
Copy link
Member

Just a few notes from some investigating:

  • I reran the workflow since we've fixed the Go nullability issue and at least one issue with big endian footers since the last run
  • The Python errors for nanoarrow produce/consume are in archery and can be fixed with
--- a/dev/archery/archery/integration/tester_nanoarrow.py
+++ b/dev/archery/archery/integration/tester_nanoarrow.py
@@ -56,8 +56,8 @@ class NanoarrowTester(Tester):
             'JSON_PATH': json_path,
             'COMMAND': command,
             **{
-                f'QUIRK_{q}': 1
-                for q in quirks or []
+                f'QUIRK_{q}': "1"
+                for q in quirks or ""
             },
  • After that, nanoarrow produce/consume with itself passes golden file tests for 1.0.0-littleendian, but not any of the others.
  • nanoarrow's reader seems to choke on certain types of offset buffers exported by JS and csharp
  • All other failures seem related to nanoarrow's stream-to-file conversion

@paleolimbot
Copy link
Member

nanoarrow's reader seems to choke on certain types of offset buffers exported by JS and csharp

apache/arrow-nanoarrow#626

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting committer review Awaiting committer review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants