Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky test: TestSMTPMailerWithNoEHLO.test_send_multiple_threads #15

Closed
mgedmin opened this issue Oct 6, 2017 · 2 comments · Fixed by #38
Closed

Flaky test: TestSMTPMailerWithNoEHLO.test_send_multiple_threads #15

mgedmin opened this issue Oct 6, 2017 · 2 comments · Fixed by #38

Comments

@mgedmin
Copy link
Member

mgedmin commented Oct 6, 2017 •
edited
Loading

Examples of failures:

log 
$ coverage run -m zope.testrunner --test-path=src
Running zope.testrunner.layer.UnitTests tests:
  Set up zope.testrunner.layer.UnitTests in 0.000 seconds.
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/zopefoundation/zope.sendmail/src/zope/sendmail/tests/test_mailer.py", line 147, in run
    self.test_send_multiple_same_mailer()
  File "/home/travis/build/zopefoundation/zope.sendmail/src/zope/sendmail/tests/test_mailer.py", line 138, in test_send_multiple_same_mailer
    self.assertEqual(2, len(smtps))
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 814, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 2 != 1
Failure in test test_send_multiple_threads (zope.sendmail.tests.test_mailer.TestSMTPMailerWithNoEHLO)
Traceback (most recent call last):
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 601, in run
    testMethod()
  File "/home/travis/build/zopefoundation/zope.sendmail/src/zope/sendmail/tests/test_mailer.py", line 163, in test_send_multiple_threads
    self.assertEqual([True for _ in threads], results)
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 1019, in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 1001, in assertSequenceEqual
    self.fail(msg)
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: Lists differ: [True, True] != [True, AssertionError('2 != 1',)]
First differing element 1:
True
AssertionError('2 != 1',)
- [True, True]
+ [True, AssertionError('2 != 1',)]
  Ran 77 tests with 1 failures, 0 errors and 3 skipped in 2.738 seconds.
Tearing down left over layers:
  Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.
log 
Running test-ztk-zope.sendmail
test-ztk-zope.sendmail failed with:
Running zope.testrunner.layer.UnitTests tests:
  Set up zope.testrunner.layer.UnitTests in 0.000 seconds.
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/opt/python/2.7.13/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/opt/python/2.7.13/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py2.7.egg/zope/sendmail/tests/test_mailer.py", line 147, in run
    self.test_send_multiple_same_mailer()
  File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py2.7.egg/zope/sendmail/tests/test_mailer.py", line 138, in test_send_multiple_same_mailer
    self.assertEqual(2, len(smtps))
  File "/opt/python/2.7.13/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/opt/python/2.7.13/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 2 != 1
Failure in test test_send_multiple_threads (zope.sendmail.tests.test_mailer.TestSMTPMailerWithNoEHLO)
Traceback (most recent call last):
  File "/opt/python/2.7.13/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py2.7.egg/zope/sendmail/tests/test_mailer.py", line 163, in test_send_multiple_threads
    self.assertEqual([True for _ in threads], results)
  File "/opt/python/2.7.13/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/opt/python/2.7.13/lib/python2.7/unittest/case.py", line 742, in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
  File "/opt/python/2.7.13/lib/python2.7/unittest/case.py", line 724, in assertSequenceEqual
    self.fail(msg)
  File "/opt/python/2.7.13/lib/python2.7/unittest/case.py", line 410, in fail
    raise self.failureException(msg)
AssertionError: Lists differ: [True, True] != [True, AssertionError('2 != 1'...
First differing element 1:
True
AssertionError('2 != 1',)
- [True, True]
+ [True, AssertionError('2 != 1',)]
  Ran 77 tests with 1 failures, 0 errors and 3 skipped in 0.107 seconds.
Tearing down left over layers:
  Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.
Running test-ztk-zope.i18n
...
log 
Running test-ztk-zope.sendmail
test-ztk-zope.sendmail failed with:
Running zope.testrunner.layer.UnitTests tests:
  Set up zope.testrunner.layer.UnitTests in 0.000 seconds.
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/threading.py", line 797, in __bootstrap_inner
    self.run()
  File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/threading.py", line 750, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py2.7.egg/zope/sendmail/tests/test_mailer.py", line 147, in run
    self.test_send_multiple_same_mailer()
  File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py2.7.egg/zope/sendmail/tests/test_mailer.py", line 138, in test_send_multiple_same_mailer
    self.assertEqual(2, len(smtps))
  File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 2 != 0
Failure in test test_send_multiple_threads (zope.sendmail.tests.test_mailer.TestSMTPMailerWithNoEHLO)
Traceback (most recent call last):
  File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py2.7.egg/zope/sendmail/tests/test_mailer.py", line 163, in test_send_multiple_threads
    self.assertEqual([True for _ in threads], results)
  File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/unittest/case.py", line 742, in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
  File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/unittest/case.py", line 724, in assertSequenceEqual
    self.fail(msg)
  File "/opt/python/pypy2.7-5.8.0/lib-python/2.7/unittest/case.py", line 410, in fail
    raise self.failureException(msg)
AssertionError: Lists differ: [True, True] != [True, AssertionError('2 != 0'...
First differing element 1:
True
AssertionError('2 != 0',)
- [True, True]
+ [True, AssertionError('2 != 0',)]
  Ran 77 tests with 1 failures, 0 errors and 3 skipped in 0.224 seconds.
Tearing down left over layers:
  Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.
Running test-ztk-zope.session
...
log 
Running test-ztk-zope.sendmail
test-ztk-zope.sendmail failed with:
Running zope.testrunner.layer.UnitTests tests:
  Set up zope.testrunner.layer.UnitTests in 0.000 seconds.
Failure in test test_send_multiple_threads (zope.sendmail.tests.test_mailer.TestSMTPMailerWithNoEHLO)
Traceback (most recent call last):
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 601, in run
    testMethod()
  File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py3.5.egg/zope/sendmail/tests/test_mailer.py", line 163, in test_send_multiple_threads
    self.assertEqual([True for _ in threads], results)
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 1019, in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 1001, in assertSequenceEqual
    self.fail(msg)
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: Lists differ: [True, True] != [True, AssertionError('2 != 1',)]
First differing element 1:
True
AssertionError('2 != 1',)
- [True, True]
+ [True, AssertionError('2 != 1',)]
  Ran 77 tests with 1 failures, 0 errors and 3 skipped in 0.263 seconds.
Tearing down left over layers:
  Tear down zope.testrunner.layer.UnitTests Exception in thread Thread-3:
Traceback (most recent call last):
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py3.5.egg/zope/sendmail/tests/test_mailer.py", line 147, in run
    self.test_send_multiple_same_mailer()
  File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-4.1.0-py3.5.egg/zope/sendmail/tests/test_mailer.py", line 138, in test_send_multiple_same_mailer
    self.assertEqual(2, len(smtps))
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/opt/python/pypy3.5-5.8.0/lib-python/3/unittest/case.py", line 814, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 2 != 1
in 0.000 seconds.
Running test-ztk-zope.session
...
@mgedmin mgedmin mentioned this issue Oct 6, 2017
Merged
@icemac
Copy link
Member

icemac commented Jul 31, 2020 •
edited by mgedmin
Loading

This problem still seems to exist:

log 
Running test-ztk-zope.sendmail
test-ztk-zope.sendmail failed with:
Running zope.testrunner.layer.UnitTests tests:
  Set up zope.testrunner.layer.UnitTests in 0.000 seconds.
Failure in test test_send_multiple_threads (zope.sendmail.tests.test_mailer.TestSMTPMailer)
Traceback (most recent call last):
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 601, in run
    testMethod()
  File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-5.0-py3.6.egg/zope/sendmail/tests/test_mailer.py", line 171, in test_send_multiple_threads
    self.assertEqual([True for _ in threads], results)
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 1019, in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 1001, in assertSequenceEqual
    self.fail(msg)
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: Lists differ: [True, True] != [True, AssertionError('2 != 1',)]
First differing element 1:
True
AssertionError('2 != 1',)
- [True, True]
+ [True, AssertionError('2 != 1',)]
Failure in test test_send_multiple_threads (zope.sendmail.tests.test_mailer.TestSMTPMailerWithNoEHLO)
Traceback (most recent call last):
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 601, in run
    testMethod()
  File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-5.0-py3.6.egg/zope/sendmail/tests/test_mailer.py", line 171, in test_send_multiple_threads
    self.assertEqual([True for _ in threads], results)
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 1019, in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 1001, in assertSequenceEqual
    self.fail(msg)
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: Lists differ: [True, True] != [True, AssertionError('2 != 1',)]
First differing element 1:
True
AssertionError('2 != 1',)
- [True, True]
+ [True, AssertionError('2 != 1',)]
  Ran 78 tests with 2 failures, 0 errors and 3 skipped in 0.156 seconds.
Tearing down left over layers:
  Tear down zope.testrunner.layer.UnitTests Exception in thread Thread-1:
Traceback (most recent call last):
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-5.0-py3.6.egg/zope/sendmail/tests/test_mailer.py", line 155, in run
    self.test_send_multiple_same_mailer()
  File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-5.0-py3.6.egg/zope/sendmail/tests/test_mailer.py", line 146, in test_send_multiple_same_mailer
    self.assertEqual(2, len(smtps))
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 814, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 2 != 1
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-5.0-py3.6.egg/zope/sendmail/tests/test_mailer.py", line 155, in run
    self.test_send_multiple_same_mailer()
  File "/home/travis/build/zopefoundation/zopetoolkit/eggs/zope.sendmail-5.0-py3.6.egg/zope/sendmail/tests/test_mailer.py", line 146, in test_send_multiple_same_mailer
    self.assertEqual(2, len(smtps))
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 821, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/opt/python/pypy3.6-7.1.1/lib-python/3/unittest/case.py", line 814, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 2 != 1
in 0.000 seconds.
Running test-ztk-zope.session
...

@mgedmin
Copy link
Member Author

mgedmin commented Dec 18, 2020

Here's the test:

zope.sendmail/src/zope/sendmail/tests/test_mailer.py

Lines 148 to 171 in 85e2af8

def test_send_multiple_threads(self):
import threading
results = []
def run():
try:
self.test_send_multiple_same_mailer()
except BaseException as e: # pragma: no cover
results.append(e)
raise
else:
results.append(True)
threads = []
for _ in range(2):
threads.append(threading.Thread(target=run))
for t in threads:
t.start()
for t in threads:
t.join()
self.assertEqual([True for _ in threads], results)

It calls self.test_send_multiple_same_mailer() concurrently, with the same self:

zope.sendmail/src/zope/sendmail/tests/test_mailer.py

Lines 126 to 146 in 85e2af8

def test_send_multiple_same_mailer(self):
# The mailer re-opens itself as needed when sending
# multiple mails.
smtps = []
def hook(smtp):
smtps.append(smtp)
self.smtp_hook = hook
for run in (1, 2):
fromaddr = '[email protected]' + str(run)
toaddrs = ('[email protected]', '[email protected]')
msgtext = 'Headers: headers\n\nbodybodybody\n-- \nsig\n'
self.mailer.send(fromaddr, toaddrs, msgtext)
self.assertEqual(self.smtp.fromaddr, fromaddr)
self.assertEqual(self.smtp.toaddrs, toaddrs)
self.assertEqual(self.smtp.msgtext, msgtext)
self.assertTrue(self.smtp.quitted)
self.assertTrue(self.smtp.closed)
self.assertEqual(2, len(smtps))

Line 134 is not thread safe!

zope.sendmail/src/zope/sendmail/tests/test_mailer.py

Line 134 in 85e2af8

self.smtp_hook = hook

mgedmin added a commit that referenced this issue Dec 18, 2020
@mgedmin
Fix race condition in test
3a5aaed 
We had a unit test spawning two threads attempting to run
self.test_send_multiple_same_mailer() twice, concurrently.  This works badly
when both instances try to mutate self.smtp_hook and verify values on
self.smtp.

Closes #15.
@mgedmin mgedmin mentioned this issue Dec 18, 2020
Merged
@mauritsvanrees mauritsvanrees mentioned this issue Dec 18, 2020
Merged
mgedmin added a commit that referenced this issue Dec 22, 2020
@mgedmin
Fix race condition in test
792a3ae 
We had a unit test spawning two threads attempting to run
self.test_send_multiple_same_mailer() twice, concurrently.  This works badly
when both instances try to mutate self.smtp_hook and verify values on
self.smtp.

Closes #15.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix race condition in test zopefoundation/zope.sendmail
2 participants
@mgedmin @icemac