Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-37173: [C++][Go][Format] C-export/import Run-End Encoded Arrays #37174

Merged
merged 16 commits into from
Aug 24, 2023
Merged

GH-37173: [C++][Go][Format] C-export/import Run-End Encoded Arrays #37174

merged 16 commits into from
Aug 24, 2023

Conversation

felipecrv
Copy link
Contributor

@felipecrv felipecrv commented Aug 15, 2023 •
edited
Loading

Rationale for this change

All array types should be C-exportable.

It uses "+r" as the format string. + because it's a nested format and r was not taken and seems like the obvious choice.

What changes are included in this PR?

  • Ability to C-import/expor REE arrays.
  • Reference update

Are these changes tested?

Yes.

@felipecrv felipecrv mentioned this pull request Aug 15, 2023
Closed
5 tasks
@zeroshade
Copy link
Member

@felipecrv Should I make the equivalent Go side changes and push them to this PR/branch? Or should I make a separate PR? Thoughts?

@felipecrv
Copy link
Contributor Author

@felipecrv Should I make the equivalent Go side changes and push them to this PR/branch? Or should I make a separate PR? Thoughts?

If you need my changes to test the Go implementation, you should wait a bit cause I'm still work on this one. I should even mark this as a draft.

@felipecrv felipecrv marked this pull request as draft August 15, 2023 15:12
@zeroshade
Copy link
Member

Unfortunately at the moment there isn't yet thorough c-data integration testing (but we want to add some). so I don't need your changes to add any testing, though I'd still want to do so manually at least. Do you have a preference for me keeping the Go changes separate from this PR or contributing it to this PR?

@felipecrv felipecrv changed the title GH-37173: [C++] C-export/import Run-End Encoded Arrays GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays Aug 15, 2023
@felipecrv felipecrv marked this pull request as ready for review August 15, 2023 16:19
@felipecrv
Copy link
Contributor Author

Unfortunately at the moment there isn't yet thorough c-data integration testing (but we want to add some). so I don't need your changes to add any testing, though I'd still want to do so manually at least. Do you have a preference for me keeping the Go changes separate from this PR or contributing it to this PR?

Feel free to push to this PR. I think I'm done now with the changes after fixing the buffer count issues.

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Aug 15, 2023
bkietz
bkietz requested changes Aug 16, 2023
View reviewed changes
Copy link
Member

@bkietz bkietz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly this looks great, just a couple nits:

docs/source/format/CDataInterface.rst Outdated Show resolved Hide resolved
cpp/src/arrow/c/bridge_test.cc Outdated Show resolved Hide resolved
pitrou
pitrou reviewed Aug 16, 2023
View reviewed changes
cpp/src/arrow/c/bridge_test.cc Outdated Show resolved Hide resolved
cpp/src/arrow/c/bridge_test.cc Show resolved Hide resolved
@github-actions github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Aug 16, 2023
@felipecrv felipecrv requested a review from pitrou August 18, 2023 16:24
pitrou
pitrou reviewed Aug 21, 2023
View reviewed changes
cpp/src/arrow/c/bridge.cc Outdated Show resolved Hide resolved
@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Aug 21, 2023
@github-actions github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Aug 22, 2023
pitrou
pitrou approved these changes Aug 22, 2023
View reviewed changes
Copy link
Member

@pitrou pitrou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, thank you @felipecrv

@pitrou
Copy link
Member

pitrou commented Aug 22, 2023

@alippai @roee88 @tustvold Does the C Data Interface specification addition look fine here?

@zeroshade
Copy link
Member

@felipecrv I've pushed the Go implementation for REE with c-export/import

@ianmcook ianmcook changed the title GH-37173: [C++][Format] C-export/import Run-End Encoded Arrays GH-37173: [C++][Go][Format] C-export/import Run-End Encoded Arrays Aug 22, 2023
@pitrou
Copy link
Member

pitrou commented Aug 23, 2023

@zeroshade Would you like to add REE to the roundtripped types in test_export_to_cgo.py?

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Aug 23, 2023
@github-actions github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Aug 23, 2023
@zeroshade
Copy link
Member

@bkietz is out for now but it looks like both nits he had requested have been addressed. If there's no objections by tomorrow I'll merge this in.

Thanks everyone!

@zeroshade zeroshade merged commit 9958896 into apache:main Aug 24, 2023
49 of 51 checks passed
@zeroshade zeroshade removed the awaiting change review Awaiting change review label Aug 24, 2023
@conbench-apache-arrow
Copy link

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 9958896.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.

@felipecrv felipecrv deleted the ree_c_api branch August 28, 2023 13:18
loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
@felipecrv @zeroshade @loicalleyne
apacheGH-37173: [C++][Go][Format] C-export/import Run-End Encoded Arr…
aeda02e 
…ays (apache#37174)

### Rationale for this change

All array types should be C-exportable.

It uses `"+r"` as the format string. `+` because it's a nested format and `r` was not taken and seems like the obvious choice.

### What changes are included in this PR?

- [x] Ability to C-import/expor REE arrays.
- [x] Reference update

### Are these changes tested?

Yes.
* Closes: apache#37173

Lead-authored-by: Felipe Oliveira Carvalho <felipekde@gmail.com>
Co-authored-by: Matt Topol <zotthewizard@gmail.com>
Signed-off-by: Matt Topol <zotthewizard@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pitrou pitrou pitrou approved these changes

@paleolimbot paleolimbot paleolimbot left review comments

@zeroshade zeroshade Awaiting requested review from zeroshade zeroshade is a code owner

@bkietz bkietz Awaiting requested review from bkietz

Assignees
No one assigned
Labels
Component: C++ Component: Documentation Component: Go
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[C++] REE arrays aren't supported in the C bridge API
5 participants
@felipecrv @zeroshade @pitrou @bkietz @paleolimbot