Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make parallelisation more efficient per not having more jobs than processors #43

Closed
takusuman opened this issue Dec 27, 2024 · 2 comments · Fixed by #44
Closed

Make parallelisation more efficient per not having more jobs than processors #43

takusuman opened this issue Dec 27, 2024 · 2 comments · Fixed by #44

Comments

@takusuman
Copy link
Contributor

takusuman commented Dec 27, 2024

G'day (again)!
As I stated at dslm4515/CMLFS#41, back in September 2021 --- Lord, that's a damn amount of time! ---, citing Gentoo developer Agostino Sarubbo's blog post, using more than one job per processor isn't ideal --- but mussel is doing not just two jobs, but three jobs per processor core ---, because it makes the build slower since the processor suffers from such overload.
I've got the data: I've ran current main branch (10a8a05) mussel along with a quick-n-dirty hack that defines $JOBS just as getnproc(), without triplicating it, for building a toolchain with Linux kernel headers (-l), OpenMP (-o) and pkg-config (-k) support, and the results were a difference of around 15 minutes --- w/o the fix: real 1h11m28,67s, with the fix: real 56m49,27s ---, plus the fact that the machine was also capable of doing another tasks because there wasn't such overload.
I don't know what was the reason for doing it before, but, at least in the most cases, it is, indeed, the most efficient way.

@firasuke
Copy link
Owner

I agree.

I don't know what was the reason for doing it before, but, at least in the most cases, it is, indeed, the most efficient way.

This was an old thing when it was recommended to have jobs equal to the number of logical processors plus one (to keep it busy building), but this did not age well as it now causes freezes and thrashes several systems when building decently sized packages.

@takusuman
Copy link
Contributor Author

This was an old thing when it was recommended to have jobs equal to the number of logical processors plus one (to keep it busy building), but this did not age well as it now causes freezes and thrashes several systems when building decently sized packages.

My machine froze several times already and I didn't knew the exact reason; but nice to know that in the past it was more commmon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants