Paratest hangs indefinitely with WrapperRunner and SqliteRunner

[oojacoboo]

Please see #310 as another reported case of this. I'm unsure of why this was closed.

I've spent a little time digging further into this issue and I've identified what I think is the issue. Paratest hangs after, what seems to be all tests, having completed. Upon inspection, after the completion of these tests, it appears PHPUnit also has 3 processes still in the process queue with PID assignment. The number of phpunit processes still left dangling and the number of errors reported in the paratest output are the same.

After debugging the phpunit-wrapper bin script, I've noticed that the $lastExitCode for the command PHPUnit\TextUI\Command::main(false) is 2 in all of these cases. In these cases an Exception has been thrown.

It seems to me that, when Exceptions are being thrown and the exit status of phpunit within the phpunit-wrapper is not zero, that the processes aren't being handled properly.

I can manually add something like...

if ($lastExitCode != 0) {
    echo $infoText;
    exit($lastExitCode);
}

to the phpunit-wrapper within the while loop, but, this of course does not allow for the tests to complete before printing the error. However, it's far better than having the paratest process hang indefinitely, never printing out any error details.

The funny thing is, if I alter the settings for phpunit.xml to not stopOnError, paratest never completes and ends up hanging much earlier in the process of tests.

Firstly, what is the preferred phpunit.xml config for paratest? It doesn't seem that any configuration is entirely compatible.

And also, any insights into the process management for these phpunit-wrapper scripts? If they're terminating with an exit code, how is paratest interpreting this? It doesn't seem that the processes are exited in this case and if they are, paratest interprets this as a failure and exits itself.

[Slamdunk]

I've encountered myself this behaviour many time. Yes we need to better handle wrapper/sqlite runners errors.

[timglabisch]

we've the same issue. one worker seems to never stop running.

[Slamdunk]

Hi @timglabisch , have you tried the latest version of ParaTest, currently 5?
If yes, can you provide a minimal reproducible environment that replicate the issue?

[timglabisch]

@Slamdunk i am using ^3.1 - i'll try upgrading.

[timglabisch]

@Slamdunk seems that we've the same issue with with the latest version.
Sorry but i can't provide a minimal reproducible environment.

it seems to depend on the order of the tests and i guess it is related to error output. we only ran into this issue when an error occurred.

the problem is that the worker never stops and idl's on a wait4.

[Slamdunk]

If you could debug it a bit and pinpoint some lines, I would try to investigate it further.

[timglabisch]

@Slamdunk i think i figured out the reason. the output pipe of the child process overflows and the child could not write more to the pipe. the child process simply blocks on a print but the parent process never reads the pipe.

i used an old paratest / phpunit version to test this but i guess it's the same for the current version.
the line numers are wrong because i added a lot of trace functions to find this issue.

trace where it hangs:

#0 vendor/phpunit/phpunit/src/Util/Printer.php(125): paratest_trace()
#1 vendor/phpunit/phpunit/src/TextUI/ResultPrinter.php(384): PHPUnit\\Util\\Printer->write()
#2 vendor/phpunit/phpunit/src/TextUI/ResultPrinter.php(361): PHPUnit\\TextUI\\ResultPrinter->printDefectTrace()
#3 vendor/phpunit/phpunit/src/TextUI/ResultPrinter.php(347): PHPUnit\\TextUI\\ResultPrinter->printDefect()
#4 vendor/phpunit/phpunit/src/TextUI/ResultPrinter.php(403): PHPUnit\\TextUI\\ResultPrinter->printDefects()
#5 vendor/phpunit/phpunit/src/TextUI/ResultPrinter.php(170): PHPUnit\\TextUI\\ResultPrinter->printFailures()
#6 vendor/phpunit/phpunit/src/TextUI/TestRunner.php(645): PHPUnit\\TextUI\\ResultPrinter->printResult()
#7 vendor/phpunit/phpunit/src/TextUI/Command.php(208): PHPUnit\\TextUI\\TestRunner->doRun()
#8 vendor/phpunit/phpunit/src/TextUI/Command.php(159): PHPUnit\\TextUI\\Command->run()
#9 bin/phpunit_paratest_wrapper_without_xdebug.php(83): PHPUnit\\TextUI\\Command::main()

in the end in my case it hangs exacly here (PHPUnit\Util\Printer->write()):

i've to look at the paratest code, but i guess the solution is to consume the pipes.

[Slamdunk]

The default behavior of paratest is to wrap Phpuni execution between ob_start and ob_get_clean:

paratest/bin/phpunit-wrapper.php

Line 73 in 5dc3a09

ob_start();

What you are telling me it's weird

[timglabisch]

@Slamdunk we have a slightly modified wrapper code.
In my case adding $worker->updateStateFromAvailableOutput(); in the waitForAllToFinish function solves my issue.
We may do nasty things with the output buffer in the tests.

i have to take a closer look at it to rule out that it is due to our test setup. but basically it would make sense to read out the output buffer of the children.

private function waitForAllToFinish(): void
    {
        do {
            foreach ($this->workers as $key => $worker) {
                if (!$worker->isRunning()) {
                    unset($this->workers[$key]);
                }
            }
            $worker->updateStateFromAvailableOutput(); // <<<<<<
            usleep(10000);
            $this->printOutput();
        } while (\count($this->workers) > 0);
    }

you may could try to print huge amount of data in tests. (and may disable the output buffer).

[Slamdunk]

I have never liked the way WrapperWorker communicates with its child process via pipes and \STDIN, I wish to change it.

Nevertheless, still I can't figure out how calling $worker->updateStateFromAvailableOutput(); could help...
More likely I would blame this:

paratest/bin/phpunit-wrapper.php

Lines 45 to 57 in b2a1695

	if (feof(\STDIN)) {
	exit($exitCode);
	}
	$command = fgets(\STDIN);
	if ($command === false) {
	exit($exitCode);
	}
	if ($command === \ParaTest\Runners\PHPUnit\Worker\WrapperWorker::COMMAND_EXIT) {
	echo "EXITED\n";
	exit($exitCode);
	}

if you call fgets(\STDIN); inside your tests, you mess up everything, I assume.

In the meantime I suggest you to use SqliteWrapper, is reliable as well.

[timglabisch]

@Slamdunk seems to be reproducible now, would you takeover the issue?

[Slamdunk]

I'll dig into it 👍

[timglabisch]

@Slamdunk did you already worked on this issue?

[Slamdunk]

I'm on #526 which will some many issues, including this one