Simplenews and Admin Toolbar tests are failing

[gitressa]

After following the steps in the README and installing DDEV Selenium Standalone Chrome, there are few failing tests in the Simplenews and Admin Toolbar modules.

I am not sure if DDEV Drupal Contrib, or tests in the contrib module need to be updated? I can create issues in the module issue queues, if that's where they belong.

Result of SimpleNews test

user:~/dev/simplenews$ ddev phpunit
PHPUnit 9.6.15 by Sebastian Bergmann and contributors.

Simplenews Demo (Drupal\Tests\simplenews_demo\Functional\SimplenewsDemo)
 ✔ Installed

Simplenews Administration (Drupal\Tests\simplenews\Functional\SimplenewsAdministration)
 ✘ Newsletter settings
   │
   │ Behat\Mink\Exception\ResponseTextException: The text "Simplenews adds elements to the newsletter node add/edit" was not found anywhere in the text of the current page.
   │
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:907
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:293
   │ /var/www/html/tests/src/Functional/SimplenewsAdministrationTest.php:271
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

 ✔ Subscription management
 ✘ Content types
   │
   │ Behat\Mink\Exception\ElementNotFoundException: Button with id|name|label|value "Save content type" not found.
   │
   │ /var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:144
   │ /var/www/html/web/core/tests/Drupal/Tests/UiHelperTrait.php:78
   │ /var/www/html/tests/src/Functional/SimplenewsAdministrationTest.php:725
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

 ✔ Subscriber status filter
 ✔ Newsletter issues overview
 ✔ Access

Simplenews Field Ui (Drupal\Tests\simplenews\Functional\SimplenewsFieldUi)
 ✔ Content type creation

Simplenews I18n (Drupal\Tests\simplenews\Functional\SimplenewsI18n)
 ✘ Newsletter issue translation
   │
   │ Behat\Mink\Exception\ElementNotFoundException: Button with id|name|label|value "Save content type" not found.
   │
   │ /var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:144
   │ /var/www/html/web/core/tests/Drupal/Tests/UiHelperTrait.php:78
   │ /var/www/html/tests/src/Functional/SimplenewsI18nTest.php:125
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

Simplenews Personalization Forms (Drupal\Tests\simplenews\Functional\SimplenewsPersonalizationForms)
 ✔ Synchronize subscribe register
 ✘ Synchronize register subscribe
   │
   │ Behat\Mink\Exception\ElementNotFoundException: Button with id|name|label|value "Confirm" not found.
   │
   │ /var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:144
   │ /var/www/html/web/core/tests/Drupal/Tests/UiHelperTrait.php:78
   │ /var/www/html/tests/src/Functional/SimplenewsPersonalizationFormsTest.php:87
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

 ✔ Subscribe request password
 ✔ Disable account
 ✔ Delete account
 ✘ Blocked subscribe
   │
   │ Behat\Mink\Exception\DriverException: Only string values can be used for a radio input.
   │
   │ /var/www/html/vendor/behat/mink-browserkit-driver/src/BrowserKitDriver.php:415
   │ /var/www/html/vendor/behat/mink/src/Element/NodeElement.php:118
   │ /var/www/html/web/core/tests/Drupal/Tests/UiHelperTrait.php:96
   │ /var/www/html/tests/src/Functional/SimplenewsPersonalizationFormsTest.php:171
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

Simplenews Recipient Handler (Drupal\Tests\simplenews\Functional\SimplenewsRecipientHandler)
 ✔ Site mail
 ✔ New users
 ✔ Subscribers by role

Simplenews Send (Drupal\Tests\simplenews\Functional\SimplenewsSend)
 ✔ Programmatic newsletter
 ✔ Send now no cron
 ✔ Send multiple no cron
 ✔ Send now cron throttle
 ✔ Send now cron
 ✔ Send publish no cron
 ✔ Update newsletter
 ✔ Send fail
 ✔ Reason
 ✔ Delete
 ✔ Impersonation
 ✔ Newsletter theme
 ✔ Html escaping

Simplenews Source (Drupal\Tests\simplenews\Functional\SimplenewsSource)
 ✔ Send minimal source implementation
 ✔ Send caching
 ✔ Send html
 ✔ Send hidden
 ✔ Send no caching
 ✔ Send missing node
 ✔ Send missing subscriber
 ✔ Skip

Simplenews Subscribe (Drupal\Tests\simplenews\Functional\SimplenewsSubscribe)
 ✘ Subscribe multiple
   │
   │ Behat\Mink\Exception\ExpectationException: The string "Are you sure you want to confirm your subscription for <em class="placeholder">m*****@e*****.com</em>?" was not found anywhere in the HTML response of the current page.
   │
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:888
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:363
   │ /var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:540
   │ /var/www/html/tests/src/Functional/SimplenewsSubscribeTest.php:68
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

 ✘ Confirm immediate
   │
   │ Behat\Mink\Exception\ExpectationException: The string "Subscription changes confirmed for <em class="placeholder">[email protected]</em>." was not found anywhere in the HTML response of the current page.
   │
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:888
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:363
   │ /var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:540
   │ /var/www/html/tests/src/Functional/SimplenewsSubscribeTest.php:171
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

 ✘ Subscribe anonymous
   │
   │ Behat\Mink\Exception\ExpectationException: The string "Are you sure you want to confirm your subscription for <em class="placeholder">d*****@e*****.com</em>?" was not found anywhere in the HTML response of the current page.
   │
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:888
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:363
   │ /var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:540
   │ /var/www/html/tests/src/Functional/SimplenewsSubscribeTest.php:260
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

 ✔ Subscribe anonymous single
 ✘ Subscribe anonymous redirect
   │
   │ Behat\Mink\Exception\ExpectationException: The string "Are you sure you want to confirm your subscription for <em class="placeholder">e*****@e*****.com</em>?" was not found anywhere in the HTML response of the current page.
   │
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:888
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:363
   │ /var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:540
   │ /var/www/html/tests/src/Functional/SimplenewsSubscribeTest.php:525
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

 ✔ Subscribe authenticated
 ✔ Simplenews subscription block
 ✔ Admin create
 ✘ Hash auth
   │
   │ Behat\Mink\Exception\ResponseTextException: The text "Subscriptions for [email protected]" was not found anywhere in the text of the current page.
   │
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:907
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:293
   │ /var/www/html/tests/src/Functional/SimplenewsSubscribeTest.php:779
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

 ✔ Formatting
 ✔ Duplicate
 ✔ Block settings

Simplenews Synchronize Fields Form (Drupal\Tests\simplenews\Functional\SimplenewsSynchronizeFieldsForm)
 ✔ Subscriber form field sync

Simplenews Uninstall (Drupal\Tests\simplenews\Functional\SimplenewsUninstall)
 ✔ Uninstall

Simplenews Kernel (Drupal\Tests\simplenews\Kernel\SimplenewsKernel)
 ✔ Masking

Simplenews Monitoring (Drupal\Tests\simplenews\Kernel\SimplenewsMonitoring)
 ✔ Sensors

Simplenews Synchronize Fields (Drupal\Tests\simplenews\Kernel\SimplenewsSynchronizeFields)
 ✔ Synchronize base fields
 ✔ Synchronize configurable fields
 ✔ Set shared field automatically
 ✔ Disable sync
 ✔ User recursion prevention

Time: 10:13.697, Memory: 6.00 MB

Summary of non-successful tests:

Simplenews Administration (Drupal\Tests\simplenews\Functional\SimplenewsAdministration)
 ✘ Newsletter settings
   │
   │ Behat\Mink\Exception\ResponseTextException: The text "Simplenews adds elements to the newsletter node add/edit" was not found anywhere in the text of the current page.
   │
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:907
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:293
   │ /var/www/html/tests/src/Functional/SimplenewsAdministrationTest.php:271
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

 ✘ Content types
   │
   │ Behat\Mink\Exception\ElementNotFoundException: Button with id|name|label|value "Save content type" not found.
   │
   │ /var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:144
   │ /var/www/html/web/core/tests/Drupal/Tests/UiHelperTrait.php:78
   │ /var/www/html/tests/src/Functional/SimplenewsAdministrationTest.php:725
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

Simplenews I18n (Drupal\Tests\simplenews\Functional\SimplenewsI18n)
 ✘ Newsletter issue translation
   │
   │ Behat\Mink\Exception\ElementNotFoundException: Button with id|name|label|value "Save content type" not found.
   │
   │ /var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:144
   │ /var/www/html/web/core/tests/Drupal/Tests/UiHelperTrait.php:78
   │ /var/www/html/tests/src/Functional/SimplenewsI18nTest.php:125
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

Simplenews Personalization Forms (Drupal\Tests\simplenews\Functional\SimplenewsPersonalizationForms)
 ✘ Synchronize register subscribe
   │
   │ Behat\Mink\Exception\ElementNotFoundException: Button with id|name|label|value "Confirm" not found.
   │
   │ /var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:144
   │ /var/www/html/web/core/tests/Drupal/Tests/UiHelperTrait.php:78
   │ /var/www/html/tests/src/Functional/SimplenewsPersonalizationFormsTest.php:87
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

 ✘ Blocked subscribe
   │
   │ Behat\Mink\Exception\DriverException: Only string values can be used for a radio input.
   │
   │ /var/www/html/vendor/behat/mink-browserkit-driver/src/BrowserKitDriver.php:415
   │ /var/www/html/vendor/behat/mink/src/Element/NodeElement.php:118
   │ /var/www/html/web/core/tests/Drupal/Tests/UiHelperTrait.php:96
   │ /var/www/html/tests/src/Functional/SimplenewsPersonalizationFormsTest.php:171
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

Simplenews Subscribe (Drupal\Tests\simplenews\Functional\SimplenewsSubscribe)
 ✘ Subscribe multiple
   │
   │ Behat\Mink\Exception\ExpectationException: The string "Are you sure you want to confirm your subscription for <em class="placeholder">m*****@e*****.com</em>?" was not found anywhere in the HTML response of the current page.
   │
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:888
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:363
   │ /var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:540
   │ /var/www/html/tests/src/Functional/SimplenewsSubscribeTest.php:68
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

 ✘ Confirm immediate
   │
   │ Behat\Mink\Exception\ExpectationException: The string "Subscription changes confirmed for <em class="placeholder">[email protected]</em>." was not found anywhere in the HTML response of the current page.
   │
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:888
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:363
   │ /var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:540
   │ /var/www/html/tests/src/Functional/SimplenewsSubscribeTest.php:171
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

 ✘ Subscribe anonymous
   │
   │ Behat\Mink\Exception\ExpectationException: The string "Are you sure you want to confirm your subscription for <em class="placeholder">d*****@e*****.com</em>?" was not found anywhere in the HTML response of the current page.
   │
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:888
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:363
   │ /var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:540
   │ /var/www/html/tests/src/Functional/SimplenewsSubscribeTest.php:260
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

 ✘ Subscribe anonymous redirect
   │
   │ Behat\Mink\Exception\ExpectationException: The string "Are you sure you want to confirm your subscription for <em class="placeholder">e*****@e*****.com</em>?" was not found anywhere in the HTML response of the current page.
   │
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:888
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:363
   │ /var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:540
   │ /var/www/html/tests/src/Functional/SimplenewsSubscribeTest.php:525
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

 ✘ Hash auth
   │
   │ Behat\Mink\Exception\ResponseTextException: The text "Subscriptions for [email protected]" was not found anywhere in the text of the current page.
   │
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:907
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:293
   │ /var/www/html/tests/src/Functional/SimplenewsSubscribeTest.php:779
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

ERRORS!
Tests: 60, Assertions: 1372, Errors: 10.
Failed to run phpunit : exit status 2

Result of Admin Toolbar test

user:~/dev/admin_toolbar$ ddev phpunit
PHPUnit 9.6.15 by Sebastian Bergmann and contributors.

Admin Toolbar Search Setting (Drupal\Tests\admin_toolbar_search\Functional\AdminToolbarSearchSetting)
 ✔ Toolbar search

Admin Toolbar Search (Drupal\Tests\admin_toolbar_search\FunctionalJavascript\AdminToolbarSearch)
 ✔ Toolbar search
 ✔ No access

Admin Toolbar Tools Search (Drupal\Tests\admin_toolbar_search\FunctionalJavascript\AdminToolbarToolsSearch)
 ✘ Toolbar search
   │
   │ Behat\Mink\Exception\ElementHtmlException: The string "/admin/structure/media/manage/zora/fields" appears in the HTML of the element matching css "#toolbar-administration", but it should not.
   │
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:927
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:600
   │ /var/www/html/admin_toolbar_search/tests/src/FunctionalJavascript/AdminToolbarToolsSearchTest.php:190
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

Admin Toolbar Tools Alter (Drupal\Tests\admin_toolbar_tools\Functional\AdminToolbarToolsAlter)
 ✔ Admin toolbar tools

Admin Toolbar Admin Menu (Drupal\Tests\admin_toolbar\Functional\AdminToolbarAdminMenu)
 ✔ Module status change subtrees hash cache clear
 ✔ Menu link update subtrees hash cache clear
 ✔ User role update subtrees hash cache clear
 ✔ Non current user account updates
 ✔ Locale translation subtrees hash cache clear
 ✘ Subtrees json request
   │
   │ TypeError: Behat\Mink\Session::setRequestHeader(): Argument #1 ($name) must be of type string, int given, called in /var/www/html/web/core/tests/Drupal/Tests/UiHelperTrait.php on line 235
   │
   │ /var/www/html/vendor/behat/mink/src/Session.php:199
   │ /var/www/html/web/core/tests/Drupal/Tests/UiHelperTrait.php:235
   │ /var/www/html/web/core/modules/toolbar/tests/src/Functional/ToolbarAdminMenuTest.php:387
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

 ✔ Language switching
 ✔ Back to site link
 ✔ External link

Admin Toolbar Alter (Drupal\Tests\admin_toolbar\Functional\AdminToolbarAlter)
 ✘ Admin toolbar
   │
   │ Behat\Mink\Exception\ExpectationException: The string "class="toolbar-icon toolbar-icon-user-admin-index"" was not found anywhere in the HTML response of the current page.
   │
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:888
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:363
   │ /var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:540
   │ /var/www/html/tests/src/Functional/AdminToolbarAlterTest.php:56
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

Admin Toolbar Tools Sort (Drupal\Tests\admin_toolbar\Functional\AdminToolbarToolsSort)
 ✔ Menu update
 ✔ Menu sorting

Time: 03:49.179, Memory: 6.00 MB

Summary of non-successful tests:

Admin Toolbar Tools Search (Drupal\Tests\admin_toolbar_search\FunctionalJavascript\AdminToolbarToolsSearch)
 ✘ Toolbar search
   │
   │ Behat\Mink\Exception\ElementHtmlException: The string "/admin/structure/media/manage/zora/fields" appears in the HTML of the element matching css "#toolbar-administration", but it should not.
   │
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:927
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:600
   │ /var/www/html/admin_toolbar_search/tests/src/FunctionalJavascript/AdminToolbarToolsSearchTest.php:190
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

Admin Toolbar Admin Menu (Drupal\Tests\admin_toolbar\Functional\AdminToolbarAdminMenu)
 ✘ Subtrees json request
   │
   │ TypeError: Behat\Mink\Session::setRequestHeader(): Argument #1 ($name) must be of type string, int given, called in /var/www/html/web/core/tests/Drupal/Tests/UiHelperTrait.php on line 235
   │
   │ /var/www/html/vendor/behat/mink/src/Session.php:199
   │ /var/www/html/web/core/tests/Drupal/Tests/UiHelperTrait.php:235
   │ /var/www/html/web/core/modules/toolbar/tests/src/Functional/ToolbarAdminMenuTest.php:387
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

Admin Toolbar Alter (Drupal\Tests\admin_toolbar\Functional\AdminToolbarAlter)
 ✘ Admin toolbar
   │
   │ Behat\Mink\Exception\ExpectationException: The string "class="toolbar-icon toolbar-icon-user-admin-index"" was not found anywhere in the HTML response of the current page.
   │
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:888
   │ /var/www/html/vendor/behat/mink/src/WebAssert.php:363
   │ /var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:540
   │ /var/www/html/tests/src/Functional/AdminToolbarAlterTest.php:56
   │ /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
   │

ERRORS!
Tests: 17, Assertions: 382, Errors: 3.
Failed to run phpunit : exit status 2

[weitzman]

Simplenews is failing its gitlab ci tests and admin toolbar hasn't adopted gitlab ci. I suggest working there first.

Also it is not feasible for this project to support the tests themselves that contrib modules have authored.

[gitressa]

Thanks for a fast answer, I'll create issues there.

But does this mean that Gitlab CI is a requirement? Because that's not what the README says:

This project should work for any contrib project, including those that haven't opted into Gitlab CI.

[weitzman]

Gitlab CI is not a pre-requisite. Its just that the failures likely have nothing to do this package.