Junit result excludes certain failure types

[drobertduke]

When tests are not marked public or their matching data providers cannot be found, they are marked as failed in the command line output. However, they are not marked failed in the junit result. If tests are marked failed on the command line output, I think they should be marked failed in the junit result.

<?php

class Failure_Test extends PHPUnit_Framework_TestCase
{
    /**
     * @dataProvider dataprovider_that_does_not_exist
     */
    public function test_data_provider($param)
    {
        $this->assertFalse(true);
    }

    private function test_private_function()
    {
        $this->assertFalse(true);
    }
}

$ phpunit --log-junit failure_test.xml Failure_Test.php
PHPUnit 4.1.3 by Sebastian Bergmann.

Configuration read from [...]/phpunit.xml

FF

Time: 128 ms, Memory: 9.50Mb

There were 2 failures:

1) Warning
The data provider specified for Failure_Test::test_data_provider is invalid.
Method dataprovider_that_does_not_exist does not exist

phar:///usr/local/Cellar/phpunit/4.1.3/libexec/phpunit-4.1.3.phar/phpunit/TextUI/Command.php:179
phar:///usr/local/Cellar/phpunit/4.1.3/libexec/phpunit-4.1.3.phar/phpunit/TextUI/Command.php:132

2) Warning
Test method "test_private_function" in test class "Failure_Test" is not public.

phar:///usr/local/Cellar/phpunit/4.1.3/libexec/phpunit-4.1.3.phar/phpunit/TextUI/Command.php:179
phar:///usr/local/Cellar/phpunit/4.1.3/libexec/phpunit-4.1.3.phar/phpunit/TextUI/Command.php:132

FAILURES!
Tests: 2, Assertions: 0, Failures: 2.

$ cat failure_test.xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="Failure_Test" file="/box/www/current_local/test/php/integration/modular/ssf/ssf-legacy/legacy/helpers/URL_Test.php" tests="0" assertions="0" failures="0" errors="0" time="0.000000">
    <testsuite name="Failure_Test::test_data_provider" tests="0" assertions="0" failures="0" errors="0" time="0.000000"/>
  </testsuite>
</testsuites>

[dbellettini]

👍

[josephharrington]

👍

[tjlee0909]

👍

Having a non-zero exit code but having a success JUnit-result file seems wrong.