Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bpo-38502: regrtest uses process groups if available #16829

Merged
merged 1 commit into from
Oct 18, 2019
Merged

bpo-38502: regrtest uses process groups if available #16829

merged 1 commit into from
Oct 18, 2019

Conversation

vstinner
Copy link
Member

@vstinner vstinner commented Oct 16, 2019 •
edited by bedevere-bot
Loading

regrtest now uses process groups in the multiprocessing mode (-jN
command line option) if process groups are available: if os.setsid()
and os.killpg() functions are available.

https://bugs.python.org/issue38502

@vstinner
Copy link
Member Author

@nanjekyejoannah @pitrou @pablogsal @gpshead: Do you any drawback in this approach?

pitrou
pitrou reviewed Oct 17, 2019
View reviewed changes
Copy link
Member

@pitrou pitrou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good on the principle, but a couple comments.

Lib/test/libregrtest/runtest_mp.py
try:
popen.kill()
if USE_PROCESS_GROUP:
os.killpg(popen.pid, signal.SIGKILL)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't you call popen.wait() after this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fwiw, the old/existing code didn't. (popen.kill() does not call .wait())

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

popen.wait() is always called.

The main thread calls stop() which calls _kill().

wait() is called in the TestWorkerProcess thread: _run_process() ends with "finally: self._wait_completed()" to ensure that we also read the exit status to prevent leaking a zombie process.

In my experience, calling wait() from 2 different threads is causing too many troubles :-)

Lib/test/libregrtest/runtest_mp.py Show resolved Hide resolved
gpshead
gpshead reviewed Oct 17, 2019
View reviewed changes
Copy link
Member

@gpshead gpshead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall I think trying this to see how it pans out. I can't claim to understand all of the specifics of process groups and/or sessions.

Misc/NEWS.d/next/Tests/2019-10-17-00-49-38.bpo-38502.vUEic7.rst Outdated Show resolved Hide resolved
Lib/test/libregrtest/runtest_mp.py
try:
popen.kill()
if USE_PROCESS_GROUP:
os.killpg(popen.pid, signal.SIGKILL)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fwiw, the old/existing code didn't. (popen.kill() does not call .wait())

@vstinner
bpo-38502: regrtest uses process groups if available
16dd744 
test.regrtest now uses process groups in the multiprocessing mode
(-jN command line option) if process groups are available: if
os.setsid() and os.killpg() functions are available.
@vstinner vstinner merged commit ecb035c into python:master Oct 18, 2019
@vstinner vstinner deleted the regrtest_process_group branch October 18, 2019 13:49
jacobneiltaylor pushed a commit to jacobneiltaylor/cpython that referenced this pull request Dec 5, 2019
@vstinner
bpo-38502: regrtest uses process groups if available (pythonGH-16829)
53e3319 
test.regrtest now uses process groups in the multiprocessing mode
(-jN command line option) if process groups are available: if
os.setsid() and os.killpg() functions are available.
shihai1991 pushed a commit to shihai1991/cpython that referenced this pull request Jan 31, 2020
@vstinner @shihai1991
bpo-38502: regrtest uses process groups if available (pythonGH-16829)
88f1c0f 
test.regrtest now uses process groups in the multiprocessing mode
(-jN command line option) if process groups are available: if
os.setsid() and os.killpg() functions are available.
vstinner added a commit that referenced this pull request Apr 14, 2020
@vstinner @pablogsal
[3.8] Update libregrtest from master (GH-19516)
67b8a1f 
* bpo-37531: regrtest now catchs ProcessLookupError (GH-16827)

Fix a warning on a race condition on TestWorkerProcess.kill(): ignore
silently ProcessLookupError rather than logging an useless warning.

(cherry picked from commit a661392)

* bpo-38502: regrtest uses process groups if available (GH-16829)

test.regrtest now uses process groups in the multiprocessing mode
(-jN command line option) if process groups are available: if
os.setsid() and os.killpg() functions are available.

(cherry picked from commit ecb035c)

* bpo-37957: Allow regrtest to receive a file with test (and subtests) to ignore (GH-16989)

When building Python in some uncommon platforms there are some known tests that will fail. Right now, the test suite has the ability to ignore entire tests using the -x option and to receive a filter file using the --matchfile filter. The problem with the --matchfile option is that it receives a file with patterns to accept and when you want to ignore a couple of tests and subtests, is too cumbersome to lists ALL tests that are not the ones that you want to accept and he problem with -x is that is not easy to ignore just a subtests that fail and the whole test needs to be ignored.

For these reasons, add a new option to allow to ignore a list of test and subtests for these situations.

(cherry picked from commit e0cd8aa)

* regrtest: log timeout at startup (GH-19514)

Reduce also worker timeout.

(cherry picked from commit 4cf65a6)

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
vstinner added a commit that referenced this pull request Apr 14, 2020
@vstinner @zooba
Update libregrtest from master (GH-19517)
b894b66 
* bpo-36670: regrtest bug fixes (GH-16537)

* Fix TestWorkerProcess.__repr__(): start_time is only valid
  if _popen is not None.
* Fix _kill(): don't set _killed to True if _popen is None.
* _run_process(): only set _killed to False after calling
  run_test_in_subprocess().

(cherry picked from commit 2ea71a0)

* [3.8] Update libregrtest from master (GH-19516)

* bpo-37531: regrtest now catchs ProcessLookupError (GH-16827)

Fix a warning on a race condition on TestWorkerProcess.kill(): ignore
silently ProcessLookupError rather than logging an useless warning.

(cherry picked from commit a661392)

* bpo-38502: regrtest uses process groups if available (GH-16829)

test.regrtest now uses process groups in the multiprocessing mode
(-jN command line option) if process groups are available: if
os.setsid() and os.killpg() functions are available.

(cherry picked from commit ecb035c)

* bpo-37957: Allow regrtest to receive a file with test (and subtests) to ignore (GH-16989)

When building Python in some uncommon platforms there are some known tests that will fail. Right now, the test suite has the ability to ignore entire tests using the -x option and to receive a filter file using the --matchfile filter. The problem with the --matchfile option is that it receives a file with patterns to accept and when you want to ignore a couple of tests and subtests, is too cumbersome to lists ALL tests that are not the ones that you want to accept and he problem with -x is that is not easy to ignore just a subtests that fail and the whole test needs to be ignored.

For these reasons, add a new option to allow to ignore a list of test and subtests for these situations.

(cherry picked from commit e0cd8aa)

* regrtest: log timeout at startup (GH-19514)

Reduce also worker timeout.

(cherry picked from commit 4cf65a6)

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
(cherry picked from commit 67b8a1f)

* bpo-36842: Fix reference leak in tests by running out-of-proc (GH-13556)

(cherry picked from commit 9ddc416)

* Backport libregrtest changes from master

Co-authored-by: Steve Dower <steve.dower@python.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gpshead gpshead gpshead left review comments

@pitrou pitrou pitrou left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@vstinner @gpshead @pitrou @the-knights-who-say-ni @bedevere-bot