Fix MariaDB list columns performance #6202

ausi · 2023-10-20T09:47:01Z

Q	A
Type	bug
Fixed issues	contao/contao#6409

Summary

Starting with version 3.7.0 we noticed that the listTableColumns() method got very slow on some systems, our migrate command went from taking around 2 seconds with version 3.6 to 260 seconds with version3.7.

This is due to the LEFT JOIN information_schema.CHECK_CONSTRAINTS not getting optimized by MariaDB which causes the database server to scan every table of every database.

This pull request fixes that by using a subquery that is limited by WHERE CONSTRAINT_SCHEMA = $databaseName. This brings the performance back to where it was with version 3.6.

src/Platforms/MariaDb1043Platform.php

fritzmg · 2023-10-20T10:02:00Z

Just to add more notes: this is especially noticeable in local development environments where you may have a lot of databases with lots of tables from various projects - as the unoptimized information_schema.CHECK_CONSTRAINTS query will then scan every table of every database every time, which is very slow. Thus exacerbating the issue when listTableColumns() is executed multiple times during database migrations as previously mentioned (hence the operation time sky rocketing from just 2 seconds to 260 seconds for example).

derrabus

Sorry for picking this PR up so late.

src/Platforms/AbstractMySQLPlatform.php

tests/Functional/Schema/MySQLSchemaManagerTest.php

ausi · 2023-11-02T22:04:21Z

Sorry for picking this PR up so late.

Nothing to be sorry about! Thank you for all your work on doctrine!

ausi · 2023-11-05T19:16:44Z

This should be ready now.

Followup pull requests:

Deprecation in 3.8: Deprecate AbstractMySQLPlatform::getColumnTypeSQLSnippets() #6213
Removal in 4.0: Remove AbstractMySQLPlatform::getColumnTypeSQLSnippets() #6214

* 3.7.x: Fix MariaDB list columns performance (#6202)

| Q | A |------------- | ----------- | Type | improvement | Fixed issues | #6202 (comment) #### Summary Deprecates `AbstractMySQLPlatform::getColumnTypeSQLSnippets()` ~~Needs to be rebased after #6202 got merged upstream.~~ Done. --------- Co-authored-by: Alexander M. Turek <me@derrabus.de>

* 3.8.x: Deprecate AbstractMySQLPlatform::getColumnTypeSQLSnippets() (doctrine#6213) Fix MariaDB list columns performance (doctrine#6202)

* 3.8.x: Enable skipping locked rows in QueryBuilder Fix coding standard violation Remove unused unit test fixtures Add Oracle v23 to CI testing (doctrine#6217) Extend `AbstractMySQLPlatform::getColumnTypeSQLSnippets()` deprecation layer (doctrine#6215) Deprecate AbstractMySQLPlatform::getColumnTypeSQLSnippets() (doctrine#6213) Fix MariaDB list columns performance (doctrine#6202)

| Q | A |------------- | ----------- | Type | feature | Fixed issues | #6202 (comment) #### Summary Removes `AbstractMySQLPlatform::getColumnTypeSQLSnippets()` Needs to be rebased after #6202 and #6213 got merged upstream.

| Q | A |------------- | ----------- | Type | bug | Fixed issues | contao/contao#6409 #### Summary Starting with version 3.7.0 we noticed that the `listTableColumns()` method got very slow on some systems, our migrate command went from taking around 2 seconds with version 3.6 to 260 seconds with version3.7. This is due to the `LEFT JOIN information_schema.CHECK_CONSTRAINTS` not getting optimized by MariaDB which causes the database server to scan every table of every database. This pull request fixes that by using a subquery that is limited by `WHERE CONSTRAINT_SCHEMA = $databaseName`. This brings the performance back to where it was with version 3.6.

Doctrine DBAL 3.7.0 introduced a performance issue [1] which has been already fixed by the Doctrine Team [2]. This change updates the `doctrine/dbal` composer constraint to ensure that this performance issue is gone. Note: The monorepo composer.lock already containted the 3.7.2 bugfis release, therefore this has not been detected in casual core development. This update is to mitigate the performance issue at all costs for TYPO3 users. As a sideeffect, this should put `composerInstallMin` nightly function tests on speed again. Used command(s): > composer require --no-update --no-install \ -d typo3/sysext/redirects \ "doctrine/dbal":"^3.7.2" ; \ composer require --no-update --no-install \ -d typo3/sysext/core \ "doctrine/dbal":"^3.7.2" ; \ composer require --no-update --no-install \ -d typo3/sysext/install \ "doctrine/dbal":"^3.7.2" ; \ composer require --no-update \ "doctrine/dbal":"^3.7.2" ; \ composer update --lock [1] contao/contao#6409 [2] doctrine/dbal#6202 Resolves: #102830 Releases: main, 12.4 Change-Id: If8b09f9556f11cc0fda4690f81343fc6c5d5e476 Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/82449 Reviewed-by: Benni Mack <benni@typo3.org> Reviewed-by: Stefan Bürk <stefan@buerk.tech> Tested-by: Stefan Bürk <stefan@buerk.tech> Reviewed-by: Oliver Klee <typo3-coding@oliverklee.de> Tested-by: core-ci <typo3@b13.com> Reviewed-by: Garvin Hicking <gh@faktor-e.de> Tested-by: Benni Mack <benni@typo3.org> Tested-by: Garvin Hicking <gh@faktor-e.de> Tested-by: Oliver Klee <typo3-coding@oliverklee.de>

Doctrine DBAL 3.7.0 introduced a performance issue [1] which has been already fixed by the Doctrine Team [2]. This change updates the `doctrine/dbal` composer constraint to ensure that this performance issue is gone. Note: The monorepo composer.lock already containted the 3.7.2 bugfis release, therefore this has not been detected in casual core development. This update is to mitigate the performance issue at all costs for TYPO3 users. As a sideeffect, this should put `composerInstallMin` nightly function tests on speed again. Used command(s): > composer require --no-update --no-install \ -d typo3/sysext/redirects \ "doctrine/dbal":"^3.7.2" ; \ composer require --no-update --no-install \ -d typo3/sysext/core \ "doctrine/dbal":"^3.7.2" ; \ composer require --no-update --no-install \ -d typo3/sysext/install \ "doctrine/dbal":"^3.7.2" ; \ composer require --no-update \ "doctrine/dbal":"^3.7.2" ; \ composer update --lock [1] contao/contao#6409 [2] doctrine/dbal#6202 Resolves: #102830 Releases: main, 12.4 Change-Id: If8b09f9556f11cc0fda4690f81343fc6c5d5e476 Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/82448 Tested-by: Markus Klein <markus.klein@typo3.org> Tested-by: Garvin Hicking <gh@faktor-e.de> Reviewed-by: Oliver Klee <typo3-coding@oliverklee.de> Reviewed-by: Stefan Bürk <stefan@buerk.tech> Tested-by: core-ci <typo3@b13.com> Reviewed-by: Benni Mack <benni@typo3.org> Reviewed-by: Markus Klein <markus.klein@typo3.org> Tested-by: Oliver Klee <typo3-coding@oliverklee.de> Tested-by: Benni Mack <benni@typo3.org> Reviewed-by: Garvin Hicking <gh@faktor-e.de> Tested-by: Stefan Bürk <stefan@buerk.tech>

Doctrine DBAL 3.7.0 introduced a performance issue [1] which has been already fixed by the Doctrine Team [2]. This change updates the `doctrine/dbal` composer constraint to ensure that this performance issue is gone. Note: The monorepo composer.lock already containted the 3.7.2 bugfis release, therefore this has not been detected in casual core development. This update is to mitigate the performance issue at all costs for TYPO3 users. As a sideeffect, this should put `composerInstallMin` nightly function tests on speed again. Used command(s): > composer require --no-update --no-install \ -d typo3/sysext/redirects \ "doctrine/dbal":"^3.7.2" ; \ composer require --no-update --no-install \ -d typo3/sysext/core \ "doctrine/dbal":"^3.7.2" ; \ composer require --no-update --no-install \ -d typo3/sysext/install \ "doctrine/dbal":"^3.7.2" ; \ composer require --no-update \ "doctrine/dbal":"^3.7.2" ; \ composer update --lock [1] contao/contao#6409 [2] doctrine/dbal#6202 Resolves: #102830 Releases: main, 12.4 Change-Id: If8b09f9556f11cc0fda4690f81343fc6c5d5e476 Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/82449 Reviewed-by: Benni Mack <benni@typo3.org> Reviewed-by: Stefan Bürk <stefan@buerk.tech> Tested-by: Stefan Bürk <stefan@buerk.tech> Reviewed-by: Oliver Klee <typo3-coding@oliverklee.de> Tested-by: core-ci <typo3@b13.com> Reviewed-by: Garvin Hicking <gh@faktor-e.de> Tested-by: Benni Mack <benni@typo3.org> Tested-by: Garvin Hicking <gh@faktor-e.de> Tested-by: Oliver Klee <typo3-coding@oliverklee.de>

Doctrine DBAL 3.7.0 introduced a performance issue [1] which has been already fixed by the Doctrine Team [2]. This change updates the `doctrine/dbal` composer constraint to ensure that this performance issue is gone. Note: The monorepo composer.lock already containted the 3.7.2 bugfis release, therefore this has not been detected in casual core development. This update is to mitigate the performance issue at all costs for TYPO3 users. As a sideeffect, this should put `composerInstallMin` nightly function tests on speed again. Used command(s): > composer require --no-update --no-install \ -d typo3/sysext/redirects \ "doctrine/dbal":"^3.7.2" ; \ composer require --no-update --no-install \ -d typo3/sysext/core \ "doctrine/dbal":"^3.7.2" ; \ composer require --no-update --no-install \ -d typo3/sysext/install \ "doctrine/dbal":"^3.7.2" ; \ composer require --no-update \ "doctrine/dbal":"^3.7.2" ; \ composer update --lock [1] contao/contao#6409 [2] doctrine/dbal#6202 Resolves: #102830 Releases: main, 12.4 Change-Id: If8b09f9556f11cc0fda4690f81343fc6c5d5e476 Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/82448 Tested-by: Markus Klein <markus.klein@typo3.org> Tested-by: Garvin Hicking <gh@faktor-e.de> Reviewed-by: Oliver Klee <typo3-coding@oliverklee.de> Reviewed-by: Stefan Bürk <stefan@buerk.tech> Tested-by: core-ci <typo3@b13.com> Reviewed-by: Benni Mack <benni@typo3.org> Reviewed-by: Markus Klein <markus.klein@typo3.org> Tested-by: Oliver Klee <typo3-coding@oliverklee.de> Tested-by: Benni Mack <benni@typo3.org> Reviewed-by: Garvin Hicking <gh@faktor-e.de> Tested-by: Stefan Bürk <stefan@buerk.tech>

Fix MariaDB list columns performance

f921da9

ausi commented Oct 20, 2023

View reviewed changes

src/Platforms/MariaDb1043Platform.php Show resolved Hide resolved

ausi mentioned this pull request Oct 20, 2023

doctrine/dbal: 3.7.0 causes performance issue with contao:migrate contao/contao#6409

Closed

fritzmg approved these changes Oct 20, 2023

View reviewed changes

m-vo approved these changes Oct 20, 2023

View reviewed changes

derrabus reviewed Nov 2, 2023

View reviewed changes

src/Platforms/AbstractMySQLPlatform.php Outdated Show resolved Hide resolved

tests/Functional/Schema/MySQLSchemaManagerTest.php Show resolved Hide resolved

derrabus added MariaDB Performance Schema Introspection Bug labels Nov 2, 2023

ausi added 2 commits November 3, 2023 11:45

Fix backwards compatibility

d0ac1da

Fix PHPStan and Psalm

b7017b1

derrabus added this to the 3.7.2 milestone Nov 5, 2023

Deprecate and replace getColumnTypeSQLSnippets()

82d8266

ausi force-pushed the fix/mariadb-list-columns-json branch from 3a01396 to 82d8266 Compare November 5, 2023 18:14

Revert deprecation

f2758ff

This was referenced Nov 5, 2023

Deprecate AbstractMySQLPlatform::getColumnTypeSQLSnippets() #6213

Merged

Remove AbstractMySQLPlatform::getColumnTypeSQLSnippets() #6214

Merged

ausi requested a review from derrabus November 5, 2023 19:17

derrabus approved these changes Nov 5, 2023

View reviewed changes

derrabus merged commit 1f1aad9 into doctrine:3.7.x Nov 5, 2023
82 checks passed

derrabus added a commit that referenced this pull request Nov 5, 2023

Merge branch '3.7.x' into 3.8.x

706007b

* 3.7.x: Fix MariaDB list columns performance (#6202)

ausi deleted the fix/mariadb-list-columns-json branch November 5, 2023 19:34

derrabus added a commit to derrabus/dbal that referenced this pull request Nov 5, 2023

Merge branch '3.8.x' into 4.0.x

b8c6007

* 3.8.x: Deprecate AbstractMySQLPlatform::getColumnTypeSQLSnippets() (doctrine#6213) Fix MariaDB list columns performance (doctrine#6202)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix MariaDB list columns performance #6202

Fix MariaDB list columns performance #6202

ausi commented Oct 20, 2023

fritzmg commented Oct 20, 2023 •

edited

Loading

derrabus left a comment

ausi commented Nov 2, 2023

ausi commented Nov 5, 2023

Fix MariaDB list columns performance #6202

Fix MariaDB list columns performance #6202

Conversation

ausi commented Oct 20, 2023

Summary

fritzmg commented Oct 20, 2023 • edited Loading

derrabus left a comment

Choose a reason for hiding this comment

ausi commented Nov 2, 2023

ausi commented Nov 5, 2023

fritzmg commented Oct 20, 2023 •

edited

Loading