Parallel testing recreate-databases and seeding / setUpProcess hook

[daniloesser]

• Laravel Version: 8.25
• PHP Version: 7.4
• Database Driver & Version: MySql 8

Description:

I need to prepare the database to be used by my application's tests (create and seed). So I've added a "db:seed" call using the ParallelTesting facade, to the AppServiceProvider Class - boot method, as a hook, as documentation suggested.

I was expecting that the ParallelTesting::setUpProcess function would have been triggered during each database creation, seeding them and making them ready for tests. But I was not able to. The second hook, ParallelTesting::setUpTestCase works good, but I don't want to seed for every test, I'd like to seed once and only re-seed during the database creation process.

Steps To Reproduce:

1 - Insert some code to seed the database inside the ParallelTesting::setUpProcess hook.
1- run "php artisan test --parallel -p 2 --testsuite=Feature --runner WrapperRunner --recreate-databases tests/Feature" and notice that the code inside the setUpProcess is never called and all my tests are performed without any data from the seeder.

[nunomaduro]

The reason why "ParallelTesting::setUpProcess" is not being called is because you are overriding the runner. You should not do that, as Laravel already has his own runner.

Are you using any database testing traits?

[daniloesser]

Here's my output when I remove the "--runner WrapperRunner" parameter:
PHP Fatal error: Uncaught Error: Interface 'ParaTest\Runners\PHPUnit\RunnerInterface' not found in /var/www/html/vendor/laravel/framework/src/Illuminate/Testing/Parall elRunner.php:13 Stack trace: #0 /var/www/html/vendor/composer/ClassLoader.php(444): include() #1 /var/www/html/vendor/composer/ClassLoader.php(322): Composer\Autoload\includeFile('/var/www/html/v...') #2 [internal function]: Composer\Autoload\ClassLoader->loadClass('Illuminate\\Test...') #3 [internal function]: spl_autoload_call('Illuminate\\Test...') #4 /var/www/html/vendor/brianium/paratest/src/Console/Testers/PHPUnit.php(270): class_exists('\\Illuminate\\Tes...') #5 /var/www/html/vendor/brianium/paratest/src/Console/Testers/PHPUnit.php(107): ParaTest\Console\Testers\PHPUnit->initializeRunner(Object(Symfony\Component\Console\Inpu t\ArgvInput)) #6 /var/www/html/vendor/brianium/paratest/src/Console/Commands/ParaTestCommand.php(115): ParaTest\Console\Testers\PHPUnit->execute(Object(Symfony\Component\Console\Inpu t\ArgvInput), Object(Symfony\Component\Console\Output in /var/www/html/vendor/laravel/framework/src/Illuminate/Testing/ParallelRunner.php on line 13

[nunomaduro]

What paratest version are you using?

[daniloesser]

"brianium/paratest": "^4.0",

[daniloesser]

My bad. I was using an outdated version of paratest.

I've upgraded both paratest and phpunit libs to the latest version. Now when I run php artisan test --parallel -p 2 --testsuite=Feature --recreate-databases tests/Feature the script is trying to seed the tables before the migration process starts.

I've tried to force the tables creation and then re-run using 2 processes:
ParallelTesting::setUpProcess(function ($token) { \Artisan::call('migrate:fresh'); \Artisan::call('db:seed'); // this would call my standard DB seeder class });

By doing that, I've got my default DB wiped/recreated and it also created two new ones: DB_1 and DB_2. However, only the default one was seeded. DB_1 and DB_2 just got migrated without any data, and due to that, every test failed.

[nunomaduro]

Thanks for the detailed answer. Before we actually find a solution for this. Could you tell me how would you "seed" your testing database before without parallel testing?

[daniloesser]

Before, my setup was basically one separated DB, assigned on a .env.testing file. I have a custom command called "project:setup" that migrates and seeds using a custom seeder class.

project:setup command:

        $this->call('migrate:fresh');
        $this->call('db:seed');

the db:seed just calls a DatabaseSeeder class that does the following:

        $this->call([
            CountrySeeder::class,
            PageSeeder::class,
            etc,
            etc,
            ...]);

By doing that, we just populate a few tables with essential data needed for testing purposes, and then we were able to test using phpunit/artisan and --env=testing

[johanvanhelden]

In case it helps, I am having similar issues.

There is also a separate testing database configured in an .env.testing file.

In my case I would also need to run a custom Artisan command for each database that is being created: bootstrap:application.
This command is responsible for setting up roles, permissions and making sure at least 1 default user is set up.

I used to do it with a composer alias:

    "scripts": {
        "test": [
            "php artisan migrate --env=testing --no-ansi",
            "php artisan bootstrap:application --env=testing --no-ansi",
            "./vendor/bin/phpunit --testsuite Unit",
            "./vendor/bin/phpunit --testsuite Feature"
        ],
    }

So I would assume this would work:

    "scripts": {
        "test": [
            "php artisan migrate --env=testing --no-ansi",
            "php artisan bootstrap:application --env=testing --no-ansi",
            "@php artisan test --parallel"
        ],
    }

But it does not add the roles/permissions/default users the bootstrapper creates.

Calling the bootstrapper from within ParallelTesting::setUpProcess also does not work. And same as the OP, ParallelTesting::setUpTestCase works fine, but I don't want to execute it 80 times if there are 80 tests.

[nunomaduro]

The ParallelTesting::setUpProcess won't work for seeds, because the database does not exist yet at that time. The database is created once the first database testing trait appears.

I am assuming you all are using the "DatabaseTransactions" trait correct? If yes, would a ParallelTesting::afterTestDatabaseCreated or similar solve your problems?

public function boot()
{
    ParallelTesting:: afterTestDatabaseCreated(function () {
        Artisan::call('db:seed');
    });
}

[johanvanhelden]

@nunomaduro Yes, I am using the DatabaseTransactions trait.

An afterTestDatabaseCreated sounds to me like it would surely solve the problem! As it would only run once for each created database. However, I am worried about subsequent runs that use those created databases.

On subsequent runs, would the data still be there that was inserted during the initial run? Or would that data be cleaned up by the DatabaseTransactions trait to an empty database?

If the latter is the case, the "seeded" data would be gone, and the "bootstrapper" would need to run again. But that would not happen because the databases are already created and thus the tests would fail due to missing data. Does that question make sense?

If the data would be persisted for future runs, I don't see any issues and that would be a great solution. But if we would have to use the --recreate-databases flag to make sure the "bootstrapper" runs, that would slow down the test suite to much and would negate the positive effects of the parallel testing.

[nunomaduro]

The seeded data would be there, as the afterTestDatabaseCreated hook would run after creating the database, but before the "transaction".

On the next php artisan test --parallel calls, the afterTestDatabaseCreated won't be called - as the database already exists.

If you use, php artisan test --parallel --recreate-databases, the afterTestDatabaseCreated will be called - as the database needs to be recreated.

[johanvanhelden]

In that case it really sounds like a great solution! It will only run when needed and ensure the data is present. Great thinking @nunomaduro !

[johanvanhelden]

@driesvints, correct me if I am wrong, but this is not yet actually implemented in the framework yet, so should the issue not remain open for now?

[driesvints]

@johanvanhelden this issue tracker is mainly for tracking bugs, not new functionality.

[BertvanHoekelen]

Temporary workaround for everybody running into the same problem. Create a migration with:

class SeedTestDatabase extends Migration
{
    public function up(): void
    {
        if (App::environment('testing') && ParallelTesting::token() !== 1) {
            Artisan::call('db:seed');
        }
    }
}

[introwit]

@nunomaduro any updates on this?

[driesvints]

@introwit not sure what updates you're expecting?

[johanvanhelden]

@driesvints My guess is the suggested fix Nuno mentioned here: #36068 (comment)

I was also actually hoping it is something that was being worked on as it is the only thing preventing me from using parallel testing.

[introwit]

yep what @johanvanhelden said 👆🏽

[nunomaduro]

@introwit @johanvanhelden @daniloesser Let me know what do you think: #36301.

[daniloesser]

@johanvanhelden, @introwit I've tested and it's working as expected. @nunomaduro , Thank you again for this improvement!

[akalongman]

@nunomaduro one thing I did not understand, on each run of php artisan test --parallel, migrations will run automatically? Call with --recreate-databases flag is not option, its very slow