Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[VL] Explicit Arrow transitions that are viewable in query plan and Spark SQL UI #7313

Closed
zhztheplayer opened this issue Sep 23, 2024 · 0 comments · Fixed by #7436
Closed

[VL] Explicit Arrow transitions that are viewable in query plan and Spark SQL UI #7313

zhztheplayer opened this issue Sep 23, 2024 · 0 comments · Fixed by #7436
Labels
enhancement New feature or request

Comments

@zhztheplayer
Copy link
Member

zhztheplayer commented Sep 23, 2024 •
edited
Loading

Description

In Gluten's query processing framework, we have applied the following Arrow-related data transitions that are actually implicit to users:

  1. Spark-Arrow to Native-Arrow through Java API ColumnarBatches#ensureOffloaded
  2. Native-Arrow / Native-Velox to Spark-Arrow through Java API ColumnarBatches#ensureLoaded
  3. Native-Arrow to Native-Velox though C++ API VeloxColumnarBatch::from

These implicit transitions are developer-friendly but make performance tunning hard. As we have adopted transition API in Gluten's query planner, now it becomes possible to pull up the work of adding these transitions to query planning phase.
@zhztheplayer zhztheplayer added the enhancement New feature or request label Sep 23, 2024
This was referenced Sep 24, 2024
Merged
Merged
This was referenced Sep 27, 2024
Merged
Merged
@zhztheplayer zhztheplayer mentioned this issue Sep 30, 2024
Merged
@zhztheplayer zhztheplayer mentioned this issue Oct 8, 2024
Merged
@zhztheplayer zhztheplayer mentioned this issue Oct 16, 2024
Merged
sharkdtu pushed a commit to sharkdtu/gluten that referenced this issue Nov 11, 2024
@zhztheplayer
[GLUTEN-7313][VL] Explicit Arrow transitions, part 5: extra code clea…
cdf4d48 
…nups (apache#7436)

Closes apache#7313
@zhztheplayer zhztheplayer mentioned this issue Jan 8, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[GLUTEN-7313][VL] Explicit Arrow transitions, part 5: extra code cleanups zhztheplayer/gluten
1 participant
@zhztheplayer