[build] Use os.sched_getaffinity only if available #1727
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rossberg
approved these changes
Feb 15, 2024
|
Hm, CI does not like the Python, though. |
Yeah I just noticed https://github.com/WebAssembly/spec/actions/runs/7914013430/job/21602858413?pr=1727 I’ll attempt to fix it right now |
This change un-breaks the build in macOS and Windows environments, by only calling the Python os.sched_getaffinity() function where it’s available (which seems to be only on Linux) — and where it’s not available, using multiprocessing.cpu_count() instead. Otherwise, the build fails in macOS and Windows environment, because the document/core/util/mathjax2katex.py program fails — and fails the build — with a “module 'os' has no attribute 'sched_getaffinity'” message (because os.sched_getaffinity(0) is hardcoded into the program).
sideshowbarker
force-pushed
the
sideshowbarker/os-sched_getaffinity-workaround
branch
from
35be9ce to
165f9f2
Compare
|
OK, CI all green now |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change un-breaks the build in macOS and Windows environments, by only calling the Python
os.sched_getaffinity()function where it’s available (which seems to be only on Linux) — and where it’s not available, usingmultiprocessing.cpu_count()instead.Otherwise, the build fails in macOS and Windows environment, because the
document/core/util/mathjax2katex.pyprogram fails — and fails the build — with a “module 'os' has no attribute 'sched_getaffinity'” message (becauseos.sched_getaffinity(0)is hardcoded into the program).This patch will have no effect on CI (which runs under a Linux environment and is working fine) but should have the effect of enabling anybody in macOS and Windows environments to actually run the build successfully.
I’ve tested in my macOS 14.4 environment with Python 3.11.7, and the docs say that
multiprocessing.cpu_count()is supported in all environments all the way back to Python 2.6 — so I can’t see any reason why it wouldn’t always work.All that said, there’s a big difference between
os.sched_getaffinity()andmultiprocessing.cpu_count(); which is:os.sched_getaffinity()returns “the set of CPUs the calling thread of the current process is restricted to” whilemultiprocessing.cpu_count()simply returns “the number of CPUs in the system” — some of which may not actually be usable by the current process.But I think for the purposes of this particular build, the difference between the two isn’t critical. I believe that what probably will happen if that part of the
mathjax2katex.pyprogram ends up being given a number of CPUs that exceeds the actually-usable number of CPUs is: that program will anyway end up spawning only the number of threads equal to the number of actually-usable CPUs — and if it tries to spawn any threads beyond that number, those threads just don’t spawn. I think.Finally, for the record here: There is a
os.cpu_count()that does apparently the same thing asmultiprocessing.cpu_count()— but the docs indicate thatos.cpu_count()is available only in Python 3.4+. So it seems safer to instead usemultiprocessing.cpu_count()— which as noted above, is supported all the way back to Python 2.6.