Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SyntaxConformanceTest::testSqlOperators - Skip "Dedupe" entity on MySQL 5.5 #15234

Merged
merged 1 commit into from
Sep 7, 2019

Conversation

totten
Copy link
Member

@totten totten commented Sep 6, 2019

Overview

This disables a particularly flaky test-scenario.

Technical Details

The SyntaxConformanceTest::testSqlOperators() has been rather flaky for a while. (I think this goes back either to the creation of the test or the expansion of Dedupe API -- in other words, the test of this scenario has always been flaky.)

When the test fails, it looks like this:

api_v3_SyntaxConformanceTest::testSqlOperators with data set #28 ('Dedupe')
incorrect count returned from Dedupe getcount
Failed asserting that 0 matches expected 2.

/home/jenkins/bknix-min/build/build-1/web/sites/all/modules/civicrm/Civi/Test/Api3TestTrait.php:191
/home/jenkins/bknix-min/build/build-1/web/sites/all/modules/civicrm/tests/phpunit/api/v3/SyntaxConformanceTest.php:1131
/home/jenkins/bknix-min/build/build-1/web/sites/all/modules/civicrm/tests/phpunit/CiviTest/CiviUnitTestCase.php:215
/home/jenkins/bknix-min/civicrm-buildkit/extern/phpunit6/phpunit6.phar:570

I've spent a chunk of time investigating the flakiness and found:

  1. The version of PHP (php70 vs php72) did not make much difference.
  2. The test usually (possibly always?) passed on MySQL 5.7.
  3. The test often (but not always) failed on MySQL 5.5.
  4. The test never failed when run by itself. The smallest scenario which seemed to reliably produce the failure was:
     env
       SYNTAX_CONFORMANCE_ENTITIES='ContactType Country Dedupe Domain StateProvince' \
       CIVICRM_UF=UnitTests \
       phpunit5 \
       --stop-on-failure --tap \
       tests/phpunit/api/v3/SyntaxConformanceTest.php \
       --filter '(testCustomDataGet|testLimit|testSqlOperators)'
    
  5. The reproducibility differed on two laptops. The laptops were largely identical wrt PHP/MySQL binaries+configuration (macOS, bknix@master-loco, min cfg). They differed (nominally) in host OS and (significantly) in hardware specs. The slower machine (low-wattage 2-core CPU; 8gb; Mojave) produced test-failures much more reliably than the faster machine (high-wattage 4-core CPU; 16gb; Sierra).

Rationalizations for why it's OK to skip:

  • The test will continue to be monitored on MySQL 5.7+.
  • The test does pass reliably when run by itself on MySQL 5.5.
  • Civi doesn't officially support MySQL 5.5, so we're not required to faff-about over a failure that only occurs on MySQL 5.5.

See also: https://chat.civicrm.org/civicrm/pl/s7nbkro8yjfgzk5z4epo8g9jph

…QL 5.5

Overview
--------

This disables a particularly flaky test-scenario.

Technical Details
-----------------

The `SyntaxConformanceTest::testSqlOperators()` has been rather flaky for a
while.  (I think this goes back either to the creation of the test or the
expansion of Dedupe API -- in other words, the test of this scenario has
always been flaky.)

When the test fails, it looks like this:

```
api_v3_SyntaxConformanceTest::testSqlOperators with data set civicrm#28 ('Dedupe')
incorrect count returned from Dedupe getcount
Failed asserting that 0 matches expected 2.

/home/jenkins/bknix-min/build/build-1/web/sites/all/modules/civicrm/Civi/Test/Api3TestTrait.php:191
/home/jenkins/bknix-min/build/build-1/web/sites/all/modules/civicrm/tests/phpunit/api/v3/SyntaxConformanceTest.php:1131
/home/jenkins/bknix-min/build/build-1/web/sites/all/modules/civicrm/tests/phpunit/CiviTest/CiviUnitTestCase.php:215
/home/jenkins/bknix-min/civicrm-buildkit/extern/phpunit6/phpunit6.phar:570
```

I've spent a chunk of time investigating the flakiness and found:

1. The test usually (possibly always?) passes on MySQL 5.7.
2. The test often (but not always) fails on MySQL 5.5.
3. The version of PHP (php70 vs php72) does not make much difference.
4. The test never seems to fail when run by itself. The smallest
   scenario which seemd to reliably produce the failure was:
   ```
    env
      SYNTAX_CONFORMANCE_ENTITIES='ContactType Country Dedupe Domain StateProvince' \
      CIVICRM_UF=UnitTests \
      phpunit5 \
      --stop-on-failure --tap \
      tests/phpunit/api/v3/SyntaxConformanceTest.php \
      --filter '(testCustomDataGet|testLimit|testSqlOperators)'
    ```
5. The reproducibility differed on two laptops.  The laptops were largely
   identical wrt PHP/MySQL binaries+configuration (macOS,
   bknix@master-loco, `min` cfg). They differed in OS version
   and hardware specs. The slower machine (low-wattage 2-core CPU; 8gb; Mojave)
   produced test-failures much more reliably than the faster machine
   (high-wattage 4-core CPU; 16gb; Sierra).

Rationalizations for why it's OK to sip:

* The test will continue to be monitored on MySQL 5.7+.
* The test *does* pass reliably when run by itself on MySQL 5.5.
* [Civi doesn't officially support MySQL 5.5](https://docs.civicrm.org/sysadmin/en/latest/requirements/#mysql-version),
so we're not required to faff-about over a failure that only occurs on MySQL 5.5.

See also: https://chat.civicrm.org/civicrm/pl/s7nbkro8yjfgzk5z4epo8g9jph
@civibot
Copy link

civibot bot commented Sep 6, 2019

(Standard links)

@civibot civibot bot added the 5.18 label Sep 6, 2019
@eileenmcnaughton eileenmcnaughton merged commit 5c5cee8 into civicrm:5.18 Sep 7, 2019
@totten totten deleted the 5.18-testsqlop branch September 9, 2019 20:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants