gh-127586: properly restore blocked signals in resource_tracker.py #127587

stephen-hansen · 2024-12-04T01:38:11Z

Closes #127586. Instead of using SIG_UNBLOCK to undo the blocked "ignored signals" by resource_manager when creating a child process, we instead will save the blocked signal mask as it was prior to the call to SIG_BLOCK and restore from there using SIG_SETMASK. This has the intended effect of preserving the SIG_BLOCK on either SIGTERM or SIGINT if either was already blocked when entering this method in the parent thread. Existing behavior where either signal was not blocked prior to entering the resource tracker is still preserved since the SIG_SETMASK will undo the SIG_BLOCKs anyway if those signals were previously unblocked.

Please let me know if anything else is needed here, e.g. if a unit test is worthwhile here, I could take a stab tomorrow at writing one.

Issue: multiprocessing.Pool does not properly restore blocked signals #127586

cpython-cla-bot · 2024-12-04T01:38:15Z

All commit authors signed the Contributor License Agreement.

Using SIG_UNBLOCK to remove blocked "ignored signals" may accidentally cause side effects if the calling parent already had said signals blocked to begin with and did not intend to unblock them when creating a pool. Use SIG_SETMASK instead with the previous mask of blocked signals to restore the original blocked set.

Misc/NEWS.d/next/Library/2024-12-03-20-28-08.gh-issue-127586.zgotYF.rst

ZeroIntensity · 2024-12-10T03:27:57Z

Please let me know if anything else is needed here, e.g. if a unit test is worthwhile here, I could take a stab tomorrow at writing one.

That would be great. We generally require tests for all bugfixes.

…gotYF.rst Co-authored-by: Peter Bierma <zintensitydev@gmail.com>

stephen-hansen · 2024-12-12T18:53:32Z

Please let me know if anything else is needed here, e.g. if a unit test is worthwhile here, I could take a stab tomorrow at writing one.

That would be great. We generally require tests for all bugfixes.

perfect, thanks. I've added a blocked signals test for resource tracker which should cover the issue.

One weird thing I've noticed is that on some of my build pipelines, I'm seeing random test failures for idlelib on macos-13, e.g. https://github.com/python/cpython/actions/runs/12301168597/job/34331141506?pr=127587. Some of the test failures appear similar to what was reported on #76152. Not sure if there is an implicit dependency between resource_tracker and those unit tests, that I broke, or if this is unrelated to my changes. I looked for a while but couldn't find any explanation. I also just don't know idlelib too well.

ZeroIntensity · 2024-12-13T13:28:07Z

Yeah, those don't look related. Sometimes we get intermittent failures on some tests, generally not because of a bug in the code but because of a missing platform constraint on that test. I'm not sure if you have permissions to rerun failed CI jobs, so tag me if it happens and I'll do it.

Lib/test/_test_multiprocessing.py

pthread_sigmask is not available on some platforms Co-authored-by: Peter Bierma <zintensitydev@gmail.com>

Lib/test/_test_multiprocessing.py

miss-islington-app · 2024-12-15T19:53:26Z

Thanks @stephen-hansen for the PR, and @gpshead for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

….py (pythonGH-127587) * Correct pthread_sigmask in resource_tracker to restore old signals Using SIG_UNBLOCK to remove blocked "ignored signals" may accidentally cause side effects if the calling parent already had said signals blocked to begin with and did not intend to unblock them when creating a pool. Use SIG_SETMASK instead with the previous mask of blocked signals to restore the original blocked set. * Adding resource_tracker blocked signals test (cherry picked from commit 46006a1) Co-authored-by: Stephen Hansen <stephen.paul.hansen@gmail.com> Co-authored-by: Peter Bierma <zintensitydev@gmail.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>

miss-islington-app · 2024-12-15T19:53:42Z

Sorry, @stephen-hansen and @gpshead, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 46006a1b355f75d06c10e7b8086912c483b34487 3.12

bedevere-app · 2024-12-15T19:53:45Z

GH-127973 is a backport of this pull request to the 3.13 branch.

bedevere-bot · 2024-12-15T21:40:00Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL9 Refleaks 3.x has failed when building commit 46006a1.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1589/builds/630) and take a look at the build logs.
Check if the failure is related to this commit (46006a1) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1589/builds/630

Failed tests:

test.test_multiprocessing_fork.test_misc
test.test_multiprocessing_spawn.test_misc
test.test_multiprocessing_forkserver.test_misc

Failed subtests:

test_resource_tracker - test.test_multiprocessing_fork.test_misc.TestResourceTracker.test_resource_tracker
test_resource_tracker - test.test_multiprocessing_spawn.test_misc.TestResourceTracker.test_resource_tracker
test_resource_tracker - test.test_multiprocessing_forkserver.test_misc.TestResourceTracker.test_resource_tracker

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/test/_test_multiprocessing.py", line 5925, in test_resource_tracker
    self.assertRegex(err, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: Regex didn't match: 'resource_tracker: There appear to be 2 leaked semaphore objects' not found in 'Traceback (most recent call last):\n  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/multiprocessing/util.py", line 292, in _run_finalizers\n    finalizer()\n    ~~~~~~~~~^^\n  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/multiprocessing/util.py", line 216, in __call__\n    res = self._callback(*self._args, **self._kwargs)\n  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/multiprocessing/synchronize.py", line 86, in _cleanup\n    sem_unlink(name)\n    ~~~~~~~~~~^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory\n/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/multiprocessing/resource_tracker.py:277: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {\'/mp-h604h0vv\'}\n  warnings.warn(\n/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/multiprocessing/resource_tracker.py:293: UserWarning: resource_tracker: \'/mp-h604h0vv\': [Errno 2] No such file or directory\n  warnings.warn(\'resource_tracker: %r: %s\' % (name, e))\n'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/test/_test_multiprocessing.py", line 5925, in test_resource_tracker
    self.assertRegex(err, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: Regex didn't match: 'resource_tracker: There appear to be 2 leaked semaphore objects' not found in 'Traceback (most recent call last):\n  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/multiprocessing/util.py", line 292, in _run_finalizers\n    finalizer()\n    ~~~~~~~~~^^\n  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/multiprocessing/util.py", line 216, in __call__\n    res = self._callback(*self._args, **self._kwargs)\n  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/multiprocessing/synchronize.py", line 86, in _cleanup\n    sem_unlink(name)\n    ~~~~~~~~~~^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory\n/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/multiprocessing/resource_tracker.py:277: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {\'/mp-4r4kmi3r\'}\n  warnings.warn(\n/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/multiprocessing/resource_tracker.py:293: UserWarning: resource_tracker: \'/mp-4r4kmi3r\': [Errno 2] No such file or directory\n  warnings.warn(\'resource_tracker: %r: %s\' % (name, e))\n'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/test/_test_multiprocessing.py", line 5925, in test_resource_tracker
    self.assertRegex(err, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: Regex didn't match: 'resource_tracker: There appear to be 2 leaked semaphore objects' not found in 'Traceback (most recent call last):\n  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/multiprocessing/util.py", line 292, in _run_finalizers\n    finalizer()\n    ~~~~~~~~~^^\n  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/multiprocessing/util.py", line 216, in __call__\n    res = self._callback(*self._args, **self._kwargs)\n  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/multiprocessing/synchronize.py", line 86, in _cleanup\n    sem_unlink(name)\n    ~~~~~~~~~~^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory\n/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/multiprocessing/resource_tracker.py:277: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {\'/mp-cchcz5dh\'}\n  warnings.warn(\n/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.refleak/build/Lib/multiprocessing/resource_tracker.py:293: UserWarning: resource_tracker: \'/mp-cchcz5dh\': [Errno 2] No such file or directory\n  warnings.warn(\'resource_tracker: %r: %s\' % (name, e))\n'

bedevere-bot · 2024-12-15T21:52:39Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL8 Refleaks 3.x has failed when building commit 46006a1.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/259/builds/1850) and take a look at the build logs.
Check if the failure is related to this commit (46006a1) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/259/builds/1850

Failed tests:

test.test_multiprocessing_fork.test_misc
test.test_multiprocessing_spawn.test_misc
test.test_multiprocessing_forkserver.test_misc

Failed subtests:

test_resource_tracker - test.test_multiprocessing_fork.test_misc.TestResourceTracker.test_resource_tracker
test_resource_tracker - test.test_multiprocessing_forkserver.test_misc.TestResourceTracker.test_resource_tracker
test_resource_tracker - test.test_multiprocessing_spawn.test_misc.TestResourceTracker.test_resource_tracker

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/_test_multiprocessing.py", line 5925, in test_resource_tracker
    self.assertRegex(err, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: Regex didn't match: 'resource_tracker: There appear to be 2 leaked semaphore objects' not found in 'Traceback (most recent call last):\n  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/util.py", line 292, in _run_finalizers\n    finalizer()\n    ~~~~~~~~~^^\n  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/util.py", line 216, in __call__\n    res = self._callback(*self._args, **self._kwargs)\n  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/synchronize.py", line 86, in _cleanup\n    sem_unlink(name)\n    ~~~~~~~~~~^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory\n/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/resource_tracker.py:277: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {\'/mp-xh99_f9n\'}\n  warnings.warn(\n/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/resource_tracker.py:293: UserWarning: resource_tracker: \'/mp-xh99_f9n\': [Errno 2] No such file or directory\n  warnings.warn(\'resource_tracker: %r: %s\' % (name, e))\n'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/_test_multiprocessing.py", line 5925, in test_resource_tracker
    self.assertRegex(err, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: Regex didn't match: 'resource_tracker: There appear to be 2 leaked semaphore objects' not found in 'Traceback (most recent call last):\n  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/util.py", line 292, in _run_finalizers\n    finalizer()\n    ~~~~~~~~~^^\n  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/util.py", line 216, in __call__\n    res = self._callback(*self._args, **self._kwargs)\n  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/synchronize.py", line 86, in _cleanup\n    sem_unlink(name)\n    ~~~~~~~~~~^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory\n/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/resource_tracker.py:277: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {\'/mp-_fx_cbhe\'}\n  warnings.warn(\n/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/resource_tracker.py:293: UserWarning: resource_tracker: \'/mp-_fx_cbhe\': [Errno 2] No such file or directory\n  warnings.warn(\'resource_tracker: %r: %s\' % (name, e))\n'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/_test_multiprocessing.py", line 5925, in test_resource_tracker
    self.assertRegex(err, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: Regex didn't match: 'resource_tracker: There appear to be 2 leaked semaphore objects' not found in 'Traceback (most recent call last):\n  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/util.py", line 292, in _run_finalizers\n    finalizer()\n    ~~~~~~~~~^^\n  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/util.py", line 216, in __call__\n    res = self._callback(*self._args, **self._kwargs)\n  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/synchronize.py", line 86, in _cleanup\n    sem_unlink(name)\n    ~~~~~~~~~~^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory\n/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/resource_tracker.py:277: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {\'/mp-qy0i0jcq\'}\n  warnings.warn(\n/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/multiprocessing/resource_tracker.py:293: UserWarning: resource_tracker: \'/mp-qy0i0jcq\': [Errno 2] No such file or directory\n  warnings.warn(\'resource_tracker: %r: %s\' % (name, e))\n'

bedevere-bot · 2024-12-15T21:54:48Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Fedora Stable Refleaks 3.x has failed when building commit 46006a1.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/320/builds/1699) and take a look at the build logs.
Check if the failure is related to this commit (46006a1) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/320/builds/1699

Failed tests:

test.test_multiprocessing_fork.test_misc
test.test_multiprocessing_spawn.test_misc
test.test_multiprocessing_forkserver.test_misc

Failed subtests:

test_resource_tracker - test.test_multiprocessing_fork.test_misc.TestResourceTracker.test_resource_tracker
test_resource_tracker - test.test_multiprocessing_forkserver.test_misc.TestResourceTracker.test_resource_tracker
test_resource_tracker - test.test_multiprocessing_spawn.test_misc.TestResourceTracker.test_resource_tracker

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/test/_test_multiprocessing.py", line 5925, in test_resource_tracker
    self.assertRegex(err, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: Regex didn't match: 'resource_tracker: There appear to be 2 leaked semaphore objects' not found in 'Traceback (most recent call last):\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/multiprocessing/util.py", line 292, in _run_finalizers\n    finalizer()\n    ~~~~~~~~~^^\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/multiprocessing/util.py", line 216, in __call__\n    res = self._callback(*self._args, **self._kwargs)\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/multiprocessing/synchronize.py", line 86, in _cleanup\n    sem_unlink(name)\n    ~~~~~~~~~~^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory\n/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/multiprocessing/resource_tracker.py:277: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {\'/mp-diuavsej\'}\n  warnings.warn(\n/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/multiprocessing/resource_tracker.py:293: UserWarning: resource_tracker: \'/mp-diuavsej\': [Errno 2] No such file or directory\n  warnings.warn(\'resource_tracker: %r: %s\' % (name, e))\n'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/test/_test_multiprocessing.py", line 5925, in test_resource_tracker
    self.assertRegex(err, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: Regex didn't match: 'resource_tracker: There appear to be 2 leaked semaphore objects' not found in 'Traceback (most recent call last):\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/multiprocessing/util.py", line 292, in _run_finalizers\n    finalizer()\n    ~~~~~~~~~^^\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/multiprocessing/util.py", line 216, in __call__\n    res = self._callback(*self._args, **self._kwargs)\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/multiprocessing/synchronize.py", line 86, in _cleanup\n    sem_unlink(name)\n    ~~~~~~~~~~^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory\n/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/multiprocessing/resource_tracker.py:277: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {\'/mp-_idjhg2j\'}\n  warnings.warn(\n/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/multiprocessing/resource_tracker.py:293: UserWarning: resource_tracker: \'/mp-_idjhg2j\': [Errno 2] No such file or directory\n  warnings.warn(\'resource_tracker: %r: %s\' % (name, e))\n'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/test/_test_multiprocessing.py", line 5925, in test_resource_tracker
    self.assertRegex(err, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: Regex didn't match: 'resource_tracker: There appear to be 2 leaked semaphore objects' not found in 'Traceback (most recent call last):\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/multiprocessing/util.py", line 292, in _run_finalizers\n    finalizer()\n    ~~~~~~~~~^^\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/multiprocessing/util.py", line 216, in __call__\n    res = self._callback(*self._args, **self._kwargs)\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/multiprocessing/synchronize.py", line 86, in _cleanup\n    sem_unlink(name)\n    ~~~~~~~~~~^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory\n/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/multiprocessing/resource_tracker.py:277: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {\'/mp-wv8gyh8g\'}\n  warnings.warn(\n/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.refleak/build/Lib/multiprocessing/resource_tracker.py:293: UserWarning: resource_tracker: \'/mp-wv8gyh8g\': [Errno 2] No such file or directory\n  warnings.warn(\'resource_tracker: %r: %s\' % (name, e))\n'

bedevere-bot · 2024-12-15T22:47:48Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 RHEL8 Refleaks 3.x has failed when building commit 46006a1.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/551/builds/470) and take a look at the build logs.
Check if the failure is related to this commit (46006a1) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/551/builds/470

Failed tests:

test.test_multiprocessing_fork.test_misc
test.test_multiprocessing_spawn.test_misc
test.test_multiprocessing_forkserver.test_misc

Failed subtests:

test_resource_tracker - test.test_multiprocessing_fork.test_misc.TestResourceTracker.test_resource_tracker
test_resource_tracker - test.test_multiprocessing_forkserver.test_misc.TestResourceTracker.test_resource_tracker

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.refleak/build/Lib/test/_test_multiprocessing.py", line 5925, in test_resource_tracker
    self.assertRegex(err, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: Regex didn't match: 'resource_tracker: There appear to be 2 leaked semaphore objects' not found in 'Traceback (most recent call last):\n  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.refleak/build/Lib/multiprocessing/util.py", line 292, in _run_finalizers\n    finalizer()\n    ~~~~~~~~~^^\n  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.refleak/build/Lib/multiprocessing/util.py", line 216, in __call__\n    res = self._callback(*self._args, **self._kwargs)\n  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.refleak/build/Lib/multiprocessing/synchronize.py", line 86, in _cleanup\n    sem_unlink(name)\n    ~~~~~~~~~~^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory\n/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.refleak/build/Lib/multiprocessing/resource_tracker.py:277: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {\'/mp-5ksl4jwh\'}\n  warnings.warn(\n/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.refleak/build/Lib/multiprocessing/resource_tracker.py:293: UserWarning: resource_tracker: \'/mp-5ksl4jwh\': [Errno 2] No such file or directory\n  warnings.warn(\'resource_tracker: %r: %s\' % (name, e))\n'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.refleak/build/Lib/test/_test_multiprocessing.py", line 5925, in test_resource_tracker
    self.assertRegex(err, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
AssertionError: Regex didn't match: 'resource_tracker: There appear to be 2 leaked semaphore objects' not found in 'Traceback (most recent call last):\n  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.refleak/build/Lib/multiprocessing/util.py", line 292, in _run_finalizers\n    finalizer()\n    ~~~~~~~~~^^\n  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.refleak/build/Lib/multiprocessing/util.py", line 216, in __call__\n    res = self._callback(*self._args, **self._kwargs)\n  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.refleak/build/Lib/multiprocessing/synchronize.py", line 86, in _cleanup\n    sem_unlink(name)\n    ~~~~~~~~~~^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory\n/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.refleak/build/Lib/multiprocessing/resource_tracker.py:277: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {\'/mp-xo0p7x29\'}\n  warnings.warn(\n/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.refleak/build/Lib/multiprocessing/resource_tracker.py:293: UserWarning: resource_tracker: \'/mp-xo0p7x29\': [Errno 2] No such file or directory\n  warnings.warn(\'resource_tracker: %r: %s\' % (name, e))\n'

hugovk · 2024-12-16T11:44:08Z

The next 3.14 alpha release is tomorrow and these tier 1 and 2 buildbot failures are blocking it:

https://buildbot.python.org/#/release_status

Do you have an idea how to fix it or shall we revert this for now?

…_tracker.py (pythonGH-127587)" This reverts commit 46006a1.

hugovk · 2024-12-16T12:27:25Z

Revert PR in case we need it: #127983

ZeroIntensity · 2024-12-16T13:45:59Z

I swear that I'm a buildbot failure magnet 😄

stephen-hansen · 2024-12-16T16:49:58Z

Yeah, it looks like the cleanup test behavior might have changed. At first glance it looks like the previous behavior was that both semaphores would leak, guessing terminate() + wait() was immediate? Feels like it is now blocking on the subprocess, as name1 leaks as expected but name2 appears to be cleaned by its finalizer.. this feels correct to me but would appreciate if someone could confirm.

Guessing the fix is to change the 1 expected output line but for now happy to revert this. I’m going to investigate this more tonight.

…er.py (GH-127587)" (#127983) This reverts commit 46006a1.

gpshead · 2024-12-16T22:15:21Z

thanks!

….py (pythonGH-127587) * Correct pthread_sigmask in resource_tracker to restore old signals Using SIG_UNBLOCK to remove blocked "ignored signals" may accidentally cause side effects if the calling parent already had said signals blocked to begin with and did not intend to unblock them when creating a pool. Use SIG_SETMASK instead with the previous mask of blocked signals to restore the original blocked set. * Adding resource_tracker blocked signals test Co-authored-by: Peter Bierma <zintensitydev@gmail.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>

…_tracker.py (pythonGH-127587)" (python#127983) This reverts commit 46006a1.

stephen-hansen requested a review from gpshead as a code owner December 4, 2024 01:38

bedevere-app bot mentioned this pull request Dec 4, 2024

multiprocessing.Pool does not properly restore blocked signals #127586

Open

bedevere-app bot added the awaiting review label Dec 4, 2024

stephen-hansen added 2 commits December 3, 2024 20:43

Adding NEWS on mp.Pool signal handling

5e4797c

stephen-hansen force-pushed the gh-127586-mp-pool-sigblock-fix branch from 85370c0 to 5e4797c Compare December 4, 2024 01:44

stephen-hansen changed the title ~~gh-127586: properly restore blocked signals in resource_manager.py~~ gh-127586: properly restore blocked signals in resource_tracker.py Dec 4, 2024

Fixing mp Pool class path in NEWS

da95ed3

ZeroIntensity reviewed Dec 10, 2024

View reviewed changes

Misc/NEWS.d/next/Library/2024-12-03-20-28-08.gh-issue-127586.zgotYF.rst Outdated Show resolved Hide resolved

ZeroIntensity added needs backport to 3.12 bug and security fixes needs backport to 3.13 bugs and security fixes labels Dec 10, 2024

stephen-hansen and others added 4 commits December 9, 2024 23:07

Update Misc/NEWS.d/next/Library/2024-12-03-20-28-08.gh-issue-127586.z…

dcc52ba

…gotYF.rst Co-authored-by: Peter Bierma <zintensitydev@gmail.com>

Merge branch 'main' into pythongh-127586-mp-pool-sigblock-fix

2d30d44

Adding resource_tracker blocked signals test

41731d7

Linter fix

47e7b67

ZeroIntensity reviewed Dec 13, 2024

View reviewed changes

Lib/test/_test_multiprocessing.py Outdated Show resolved Hide resolved

Update Lib/test/_test_multiprocessing.py

070e3e8

pthread_sigmask is not available on some platforms Co-authored-by: Peter Bierma <zintensitydev@gmail.com>

gpshead approved these changes Dec 15, 2024

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting review labels Dec 15, 2024

gpshead reviewed Dec 15, 2024

View reviewed changes

Lib/test/_test_multiprocessing.py Outdated Show resolved Hide resolved

fix the test skip

82d49e3

gpshead self-assigned this Dec 15, 2024

gpshead merged commit 46006a1 into python:main Dec 15, 2024
38 checks passed

bedevere-app bot removed the awaiting merge label Dec 15, 2024

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Dec 15, 2024

hugovk added a commit to hugovk/cpython that referenced this pull request Dec 16, 2024

Revert "pythongh-127586: properly restore blocked signals in resource…

bf0ed4d

…_tracker.py (pythonGH-127587)" This reverts commit 46006a1.

hugovk mentioned this pull request Dec 16, 2024

Revert "gh-127586: properly restore blocked signals in resource_tracker.py (GH-127587)" #127983

Merged

hugovk added a commit that referenced this pull request Dec 16, 2024

Revert "gh-127586: properly restore blocked signals in resource_track…

1d276ec

…er.py (GH-127587)" (#127983) This reverts commit 46006a1.

stephen-hansen mentioned this pull request Dec 16, 2024

gh-127586: multiprocessing.Pool does not properly restore blocked signals (try 2) #128011

Merged

srinivasreddy pushed a commit to srinivasreddy/cpython that referenced this pull request Jan 8, 2025

Revert "pythongh-127586: properly restore blocked signals in resource…

3756544

…_tracker.py (pythonGH-127587)" (python#127983) This reverts commit 46006a1.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-127586: properly restore blocked signals in resource_tracker.py #127587

gh-127586: properly restore blocked signals in resource_tracker.py #127587

stephen-hansen commented Dec 4, 2024 •

edited

Loading

cpython-cla-bot bot commented Dec 4, 2024 •

edited

Loading

ZeroIntensity commented Dec 10, 2024

stephen-hansen commented Dec 12, 2024

ZeroIntensity commented Dec 13, 2024

miss-islington-app bot commented Dec 15, 2024

miss-islington-app bot commented Dec 15, 2024

bedevere-app bot commented Dec 15, 2024

bedevere-bot commented Dec 15, 2024

bedevere-bot commented Dec 15, 2024

bedevere-bot commented Dec 15, 2024

bedevere-bot commented Dec 15, 2024

hugovk commented Dec 16, 2024

hugovk commented Dec 16, 2024

ZeroIntensity commented Dec 16, 2024

stephen-hansen commented Dec 16, 2024

gpshead commented Dec 16, 2024

gh-127586: properly restore blocked signals in resource_tracker.py #127587

gh-127586: properly restore blocked signals in resource_tracker.py #127587

Conversation

stephen-hansen commented Dec 4, 2024 • edited Loading

cpython-cla-bot bot commented Dec 4, 2024 • edited Loading

ZeroIntensity commented Dec 10, 2024

stephen-hansen commented Dec 12, 2024

ZeroIntensity commented Dec 13, 2024

miss-islington-app bot commented Dec 15, 2024

miss-islington-app bot commented Dec 15, 2024

bedevere-app bot commented Dec 15, 2024

bedevere-bot commented Dec 15, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented Dec 15, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented Dec 15, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented Dec 15, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

hugovk commented Dec 16, 2024

hugovk commented Dec 16, 2024

ZeroIntensity commented Dec 16, 2024

stephen-hansen commented Dec 16, 2024

gpshead commented Dec 16, 2024

stephen-hansen commented Dec 4, 2024 •

edited

Loading

cpython-cla-bot bot commented Dec 4, 2024 •

edited

Loading