Does this still exist in 1.3.3? Do you have a small test case that can reproduce this?

I've experienced the same issue. Here's a minimal test case that triggers the problem for me.

# -*- coding: utf-8 -*-
class NoseEncodingTestCase(TestCase):
    def test_crash_nose(self):
        print u'äää'
        print '\xe4'
        self.fail()

So the problem is caused by mixing unicode and 8-bit str output in a test that fails.

StringIO code contains this comment:
The StringIO object can accept either Unicode or 8-bit strings,
but mixing the two may take some care. If both are used, 8-bit
strings that cannot be interpreted as 7-bit ASCII (that use the
8th bit) will cause a UnicodeError to be raised when getvalue()
is called.

nose==1.3.4 and django-nose==1.2

What do you propose the solution to be? What should be captured? How should it be coerced? Yes, mixing non-unicode and unicode is a problem, but what do you think the correct behavior should be and why?

BTW, thanks for the test case!

I guess that a test case (especially a failing one) might output anything in stdout. Often such outputs do come from inside 3rd party code, so nose should be able to accept a mix of any outputs

I'm not very knowledgeable on this issue, but one (maybe totally unworkable) idea:

• Implement a subclass of io.TextIOBase that uses io.BytesIO to store the data. Use this to wrap stdout.
• Make TextIOBase subclass accept both str and unicode instances. Unicode is encoded with sys.stdout.encoding before being added to BytesIO.
• When the contents need to be displayed, try to decode contents of the buffer as sys.stdout.encoding, but use errors=replace so that decoding doesn't choke on invalid byte sequences.
• Maybe have a way for the end user to configure the encoding/decoding method used.

io library was added in Python 2.6, if older versions need to be supported the current StringIO solution can be preserved

None of what you proposed works with Python 2.5 or 2.4. And I'm not real interested in maintaining yet another place where everything differs. Is there a solution that works across the board?

Problem could be solved by implementing a subclass of StringIO that overrides just getvalue().
Idea is to preserve current functionality, except when encoding error is detected, force the contents of the buffer into ascii encoding.

Pseudocode something like:

def getvalue(self):
  try:
    return super(self, ...).getvalue()
  except UnicodeDecodeError:
    implement basically the same thing as StringIO.getvalue except
    for each buffer, check if it is unicode or str, and force it into ascii,
    use errors=replace to ignore invalid byte sequences.

This is not a nice and general solution (as using io.TextIOBase would be), but should be workable in all Python versions. Note: I've only checked Python 2.7 StringIO code, if internal implementation of StringIO is a lot different in earlier Python versions, this will fail. Also don't know about Python 3, possibly this bug does not even occur there.

Doesn't this approach corrupt the output?

Yes it does, if you have a list of buffers in who-knows-what encoding and you combine them into one string, that's what happens (but output containing some question marks is preferable to tests not running at all).

But there may be another option. The problem is caused by forcing a list of strings to use the same encoding, so don't do that. Skip calling getvalue() altogether, implement another method that goes through the internal StringIO buffer list and prints each buffer in turn. No need to force any encoding on them. Even then, depending on value of sys.stdout.encoding, sometimes the output will be corrupted in any case, but I don't see a way around that.