Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dask runner windowing #469

Open
wants to merge 179 commits into
base: master
Choose a base branch
from
Open

Dask runner windowing #469

wants to merge 179 commits into from

Conversation

alxmrs
Copy link
Owner

@alxmrs alxmrs commented Oct 24, 2024

Testing the build.

alxmrs and others added 30 commits September 19, 2022 16:34
@alxmrs
WIP: Created a skeleton dask runner implementation.
bc6b525
@alxmrs
WIP: Idea for a translation evaluator.
665ee61
@alxmrs
Added overrides and a visitor that translates operations.
d1095c7
@alxmrs
Fixed a dataclass typo.
fd13d40
@alxmrs
Expanded translations.
e3d0c8a
@alxmrs
Core idea seems to be kinda working...
69a660f
@alxmrs
First iteration on DaskRunnerResult (keep track of pipeline state).
ab58334
@alxmrs
Added minimal set of DaskRunner options.
5391cd6
@alxmrs
WIP: Alllmost got asserts to work! The current status is:
5deb598 
- CoGroupByKey is broken due to how tags are used with GroupByKey
- GroupByKey should output `[('0', None), ('1', 1)]`, however it actually outputs: [(None, ('1', 1)), (None, ('0', None))]
- Once that is fixed, we may have test pipelines work on Dask.
@alxmrs
With a great 1-liner from @pabloem, groupby is fixed! Now, all three …
1768e47 
…initial tests pass.
@alxmrs
Self-review: Cleaned up dask runner impl.
e6c7106
@alxmrs
Self-review: Remove TODOs, delete commented out code, other cleanup.
1b58d4c
@alxmrs
First pass at linting rules.
5fe9372
@alxmrs
WIP, include dask dependencies + test setup.
b98330e
@alxmrs
WIP: maybe better dask deps?
e48780a
@alxmrs
Skip dask tests depending on successful import.
0dc9e23
@alxmrs
Fixed setup.py (missing ,).
326d3a3
@alxmrs
Added an additional comma.
b4cc408
@alxmrs
Moved skipping logic to be above dask import.
40a6ebe
@alxmrs
Fix lint issues with dask runner tests.
3c4204d
@alxmrs
Adding destination for client address.
41623ec
@alxmrs
Changing to async produces a timeout error instead of stuck in infini…
676d752 
…te loop.
@alxmrs
Close client during wait_until_finish; rm async.
09365f6
@pabloem
Revert "Close client during wait_until_finish; rm async."
c6ba4ba 
This reverts commit 09365f6.
@pabloem
Revert "Changing to async produces a timeout error instead of stuck i…
a325356 
…n infinite loop."

This reverts commit 676d752.
@pabloem
Adding -dask tox targets onto the gradle build
ea13125
@alxmrs
Supporting side-inputs for ParDo.
f855ffc
@alxmrs
Merge remote-tracking branch 'origin/dask-runner-mvp' into dask-runne…
3fd966e 
…r-mvp
@alxmrs
wip - added print stmt.
173d79b
@alxmrs
wip - prove side inputs is set.
dd2d15c
cisaacstern and others added 25 commits October 12, 2023 16:52
@cisaacstern
Merge branch 'master' into dask-runner-windowing
b1226fa
@cisaacstern
Merge remote-tracking branch 'upstream/master' into dask-runner-windo…
ccbccc9 
…wing
@cisaacstern
Merge remote-tracking branch 'upstream/master' into dask-runner-windo…
0a7f59a 
…wing
@cisaacstern
revert casting bag vals to list
9557e3a
@cisaacstern
cast and unpack list
899fe08
@cisaacstern
for serialization, move apply_do_fn_to_bundle to module level
c398a21
@cisaacstern
gather futures to prevent silent errors
bfbbf1c
@cisaacstern
gather and wait with as_completed to reduce memory footprint of results
f287a46
@cisaacstern
install distributed from github, temporarily
83fbfc5
@cisaacstern
client.wait_for_workers requires n_workers arg; use as_completed instead
a106dec
@cisaacstern
add test replicating apache#29365 bug
9067a54
@cisaacstern
for development, re-pin to dask pr branch
d6c35eb
@cisaacstern
Merge branch 'fix-29365' into dask-runner-windowing
8c5d8d4
@cisaacstern
side inputs for side_input test
3c0fbc6
@cisaacstern
SideInputMap WIP
8dc8971
@cisaacstern
add DaskBagWindowedIterator, and side_inputs to apply signature
327ba0d
@cisaacstern
in visitor, apply side_inputs
21573a3
@cisaacstern
add ray runner side inputs tests
9bf08a7
@cisaacstern
fix unfusable test, and single-element flattens
b8c1aaf
@cisaacstern
xfail windowed side inputs test
7932802
@cisaacstern
get pylint to pass
07e4274
@cisaacstern
latest dask doesnt support py38, so mock it for autodoc
4067c9a
@cisaacstern
reenable tests, no py38, temporary commands_pre installs
7d04b16
@cisaacstern
seems like pull_request_target is standard for other workflows
0ed78d5
@alxmrs
Merge branch 'master' of github.com:apache/beam into dask-runner-wind…
6be7621 
…owing
@github-actions github-actions bot added python Pull requests that update Python code build runners labels Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
build python Pull requests that update Python code runners
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@alxmrs @pabloem @cisaacstern