Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(execution): NativeExecutor refactor #3689

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

universalmind303
Copy link
Contributor

Small refactor to NativeExecutor to make it a bit more flexible for spark connect.

some changes made

  • make the following configurable:
    • runtime
    • progress_bar_manager
    • enable_explain_analyze
  • refactor the engine to take in the plan when performing .run not at creation.
    • this is useful as we can now reuse an engine instead of having to create a new one for every plan.
    • Note: we could probably follow up with a refactor the python code to have a single "native" engine instead of creating it every time

The runtime configuration is especially important for spark connect as previously we were using spawn_blocking when running a plan, but now we can leverage the existing compute runtime instead of creating a new one.

src/daft-connect/src/execute.rs Show resolved Hide resolved
src/daft-connect/src/execute.rs Outdated Show resolved Hide resolved
daft/runners/native_runner.py Show resolved Hide resolved
Copy link

codspeed-hq bot commented Jan 15, 2025

CodSpeed Performance Report

Merging #3689 will improve performances by 55.47%

Comparing universalmind303:execution-refactor (6f02729) with main (34d2036)

Summary

⚡ 1 improvements
✅ 26 untouched benchmarks

Benchmarks breakdown

Benchmark main universalmind303:execution-refactor Change
test_iter_rows_first_row[100 Small Files] 197.1 ms 126.8 ms +55.47%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants