pip list |head raises BrokenPipeError

[bmaggard]

• Pip version: pip 9.0.1
• Python version: python 3.5
• Operating System: OS X 10.11.6 (15G1108)

Description:

pip list |tail
works like it should

pip list |head
... not so much

What I've run:

pip list |head

Output, for each package:

--- Logging error ---
Traceback (most recent call last):
File "/Users/.../anaconda/lib/python3.5/logging/init.py", line 984, in emit
self.flush()
File "/Users/.../anaconda/lib/python3.5/logging/init.py", line 964, in flush
self.stream.flush()
BrokenPipeError: [Errno 32] Broken pipe
Call stack:
File "/Users/.../anaconda/bin/pip", line 6, in
sys.exit(pip.main())
File "/Users/.../anaconda/lib/python3.5/site-packages/pip/init.py", line 249, in main
return command.main(cmd_args)
File "/Users/.../anaconda/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/Users/.../anaconda/lib/python3.5/site-packages/pip/commands/list.py", line 164, in run
self.output_package_listing(packages, options)
File "/Users/.../anaconda/lib/python3.5/site-packages/pip/commands/list.py", line 258, in output_package_listing
logger.info(self.output_legacy(dist))
Message: 'xlwt (1.1.2)'
Arguments: ()
Exception ignored in: <_io.TextIOWrapper name='' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

[IanLee1521]

FWIW, I do not see this in pip 9.0.1, python 2.7.12, OS X 10.11.6 --

$ pip list |head
Package    Version
---------- --------
pip        9.0.1
setuptools 30.4.0
wheel      0.30.0a0

Note, I'm also not using anaconda.

[xavfernandez]

This behavior only appears with enough output (in my case, at least 11 lines).
I'm getting the same issue but it seems unrelated to pip.

import logging.config


logging.config.dictConfig({
    "version": 1,
    "handlers": {
        "console": {
            "level": logging.INFO,
            "class": "logging.StreamHandler",
            "stream": "ext://sys.stdout",
        },
    },
    "root": {
        "level": logging.INFO,
        "handlers": ["console"],
    },
})

logger = logging.getLogger(__name__)
for i in range(11):
    logger.info('Test %s', i)

yields the same logging error:

Test 0
Test 1
Test 2
Test 3
Test 4
Test 5
Test 6
Test 7
Test 8
Test 9
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.5/logging/__init__.py", line 984, in emit
    self.flush()
  File "/usr/lib/python3.5/logging/__init__.py", line 964, in flush
    self.stream.flush()
BrokenPipeError: [Errno 32] Broken pipe
Call stack:
  File "test_log_pip.py", line 21, in <module>
    logger.info('Test %s', i)
Message: 'Test %s'
Arguments: (10,)
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

Note how it successfully work for the 10 first lines.
I'm getting the same error with python 3.5 and 3.4

[pradyunsg]

To quote from http://www.unixguide.net/unix/bash/E2.shtml

If a sequence of commands appears in a pipeline, and one of the
reading commands finishes before the writer has finished, the
writer receives a SIGPIPE signal.

As far as I can see, this also closes the pipe's stdout, so I'm not sure how to handle this.

[piotr-dobrogost]

For reference: How can I fix a Broken Pipe error? at SU.

[pradyunsg]

From the link @piotr-dobrogost posted, here's a workaround:

pip list | tail -n +1 | head -1

This "bug" is essentially how unix pipes work. pip can't do anything about this.

[pradyunsg]

Can we please close this? This is not a pip bug.

[bmaggard]

I'm not very educated on this topic, and I'm not clear on who you're asking to close it. My goal was to report it as a usability issue I was experiencing when using pip. Someone else marked it a 'bug'.

Isn't it pip's responsibility to handle the EPIPE/SIGPIPE gracefully? Or perhaps it's some underlying library's responsibility?

Do we at least agree that I should be seeing the first 10 lines of output, rather than a stack trace?

[ankostis]

Cause

As can bee seen by the stack-trace, this annoyance is caused by standard python's logging-handlers that are not guarding against BrokenPipeError exceptions (from the OP):

...
File "/Users/.../anaconda/lib/python3.5/logging/init.py", line 964, in flush
self.stream.flush()
BrokenPipeError: [Errno 32] Broken pipe

Prevention

It maybe worthy for pip to mitigate it in the code that sets up logging:

pip/src/pip/_internal/utils/logging.py

Line 67 in fc4ce3d

class ColorizedStreamHandler(logging.StreamHandler):

• either by overriding flush() in all XyzHandler classes declared there,
• or by monkey-patching the base logging.Streamhandler.flush() with:

   # Monkeypatch logging to behave when piped.
    old_flush = logging.StreamHandler.flush

    def flush(self):
        try:
            old_flush(self)
        except BrokenPipeError:
            pass  # is our stderr piped in `head` or `tail`?

    logging.StreamHandler.flush = flush

I can provide a patch if you agree (and decide which method to apply).

[pradyunsg]

I'm not sure simply covering up flush() is enough here. Could you check that applying this fix will result in unintended side-effects elsewhere?

If we do decide to go ahead with a PR here, we'd definitely be overriding flush() in our subclass.

[HaleTom]

I'm also seeing this on Arch Linux:

% pip --version
pip 18.1 from /home/ravi/.local/share/miniconda3/lib/python3.6/site-packages/pip (python 3.6)
% pip list | grep -q pip

[Lots of stack trace follows]

[cjerdonek]

Hmm, overriding flush doesn't seem quite like the right approach to me here. One concern I'd have is that swallowing BrokenPipeError could have side effects like silently masking issues (e.g. letting the program continue in a broken state). A perhaps better approach I think would be to let the BrokenPipeError error bubble up (e.g. to main()), and handle it there by writing a simple message to stderr that the pipe to stdout has been broken.

A similar issue (still open) was discussed on the CPython tracker here: https://bugs.python.org/issue11380
There @ncoghlan suggested an approach in the following comment that is similar to what I described above (indeed I might have gotten the idea from reading that comment :)):
https://bugs.python.org/issue11380#msg198349

I can post a patch in a short bit showing what I mean more specifically. It seems simpler and doesn't involve modifying any vendored libraries.

[cjerdonek]

I posted a proof-of-concept here: #5907

[pradyunsg]

I like the approach taken there in that PR.

[cjerdonek]

My PR #5907 for this issue is now ready to be reviewed. I finished adding the tests I wanted to add.

[nsoranzo]

I suppose this can be closed now that #5907 has been merged.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.