Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add example of using GNU parallel to run tasks in parallel #1915

Merged
merged 5 commits into from
May 15, 2024

Conversation

amarao
Copy link
Contributor

@amarao amarao commented Feb 20, 2024

I found amazing synergy between GNU Parallel and just (basically, use parallel in --shebang mode), adding example into examples/.

@casey
Copy link
Owner

casey commented Feb 20, 2024

This is great! I think we should actually add this to the readme, since it seems super useful.

A few things:

  • There's a num_cpus() function, which can be used to set the number of jobs appropriately.
  • I think it's a good idea to use long flags instead of short flags, so --jobs instead of -j.
  • With --ungroup, parallel prints job output right away, instead of waiting for the job to finish, which lets you see when jobs start up.
  • /usr/bin/env will run parellel regardless of where it is.
  • I think we should call the recipe parallel, since churn is a bit uninformative.

With these changes, the recipe looks like this:

parallel:
  #!/usr/bin/env parallel --shebang --ungroup --jobs {{ num_cpus() }}
  echo task 1 start; sleep 1; echo task 1 done
  echo task 2 start; sleep 2; echo task 2 done
  echo task 3 start; sleep 3; echo task 3 done
  echo task 4 start; sleep 4; echo task 4 done

@amarao amarao force-pushed the add_parallel_example branch from c3fa54b to b51e9bb Compare February 21, 2024 11:58
@amarao
Copy link
Contributor Author

amarao commented Feb 21, 2024

I've updated the example.

@amarao amarao force-pushed the add_parallel_example branch from b51e9bb to 7a35b1d Compare February 21, 2024 12:02
@amarao
Copy link
Contributor Author

amarao commented Feb 21, 2024

Oh, and I found how to force env and parallel to work together in shebang mode.

Thanks for feedback.

@amarao amarao force-pushed the add_parallel_example branch from 7a35b1d to 547b0c3 Compare February 21, 2024 12:02
@amarao amarao force-pushed the add_parallel_example branch from 547b0c3 to 61f7953 Compare February 23, 2024 12:40
@casey casey enabled auto-merge (squash) May 15, 2024 01:11
@casey casey disabled auto-merge May 15, 2024 01:13
@casey casey enabled auto-merge (squash) May 15, 2024 01:14
@casey
Copy link
Owner

casey commented May 15, 2024

I moved this into the readme, since I think it'll be more discoverable there. Thanks for the PR!

@casey casey merged commit 1b62918 into casey:master May 15, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants