Skip to content

Commit

Permalink
SyntaxConformanceTest::testSqlOperators - Skip "Dedupe" entity on MyS…
Browse files Browse the repository at this point in the history
…QL 5.5

Overview
--------

This disables a particularly flaky test-scenario.

Technical Details
-----------------

The `SyntaxConformanceTest::testSqlOperators()` has been rather flaky for a
while.  (I think this goes back either to the creation of the test or the
expansion of Dedupe API -- in other words, the test of this scenario has
always been flaky.)

When the test fails, it looks like this:

```
api_v3_SyntaxConformanceTest::testSqlOperators with data set #28 ('Dedupe')
incorrect count returned from Dedupe getcount
Failed asserting that 0 matches expected 2.

/home/jenkins/bknix-min/build/build-1/web/sites/all/modules/civicrm/Civi/Test/Api3TestTrait.php:191
/home/jenkins/bknix-min/build/build-1/web/sites/all/modules/civicrm/tests/phpunit/api/v3/SyntaxConformanceTest.php:1131
/home/jenkins/bknix-min/build/build-1/web/sites/all/modules/civicrm/tests/phpunit/CiviTest/CiviUnitTestCase.php:215
/home/jenkins/bknix-min/civicrm-buildkit/extern/phpunit6/phpunit6.phar:570
```

I've spent a chunk of time investigating the flakiness and found:

1. The test usually (possibly always?) passes on MySQL 5.7.
2. The test often (but not always) fails on MySQL 5.5.
3. The version of PHP (php70 vs php72) does not make much difference.
4. The test never seems to fail when run by itself. The smallest
   scenario which seemd to reliably produce the failure was:
   ```
    env
      SYNTAX_CONFORMANCE_ENTITIES='ContactType Country Dedupe Domain StateProvince' \
      CIVICRM_UF=UnitTests \
      phpunit5 \
      --stop-on-failure --tap \
      tests/phpunit/api/v3/SyntaxConformanceTest.php \
      --filter '(testCustomDataGet|testLimit|testSqlOperators)'
    ```
5. The reproducibility differed on two laptops.  The laptops were largely
   identical wrt PHP/MySQL binaries+configuration (macOS,
   bknix@master-loco, `min` cfg). They differed in OS version
   and hardware specs. The slower machine (low-wattage 2-core CPU; 8gb; Mojave)
   produced test-failures much more reliably than the faster machine
   (high-wattage 4-core CPU; 16gb; Sierra).

Rationalizations for why it's OK to sip:

* The test will continue to be monitored on MySQL 5.7+.
* The test *does* pass reliably when run by itself on MySQL 5.5.
* [Civi doesn't officially support MySQL 5.5](https://docs.civicrm.org/sysadmin/en/latest/requirements/#mysql-version),
so we're not required to faff-about over a failure that only occurs on MySQL 5.5.

See also: https://chat.civicrm.org/civicrm/pl/s7nbkro8yjfgzk5z4epo8g9jph
  • Loading branch information
totten committed Sep 6, 2019
1 parent 804ce13 commit 382c48e
Showing 1 changed file with 9 additions and 0 deletions.
9 changes: 9 additions & 0 deletions tests/phpunit/api/v3/SyntaxConformanceTest.php
Original file line number Diff line number Diff line change
Expand Up @@ -583,6 +583,15 @@ public static function toBeSkipped_getSqlOperators() {
//a bit of a pseudoapi - keys by domain
'Setting',
];

// The testSqlOperators fails sporadically on MySQL 5.5, which is deprecated anyway.
// Test data providers should be able to run in pre-boot environment, so we connect directly to SQL server.
require_once 'DB.php';
$db = DB::connect(CIVICRM_DSN);
if ($db->connection instanceof mysqli && $db->connection->server_version < 50600) {
$entitiesWithout[] = 'Dedupe';
}

return $entitiesWithout;
}

Expand Down

0 comments on commit 382c48e

Please sign in to comment.