Moving off optparse

[pradyunsg]

@pypa/pip-committers Thoughts on switching over to using argparse or even click for option parsing?

[xavfernandez]

Since optparse is deprecated it would make sense to switch to argparse.
But I think @dstufft was planning to switch to click.

[pradyunsg]

+1 for click from me.

I'm pretty sure there'd be some edge case the switch would break but that's fine.

[thechief389]

I think you would be better off with argparse as it's in the standard library and you don't have to add a new dependency to it. It also has the argcomplete library which is good for tab completion.

Also, read this and see what you think: http://xion.io/post/programming/python-dont-use-click.html

[pradyunsg]

Summary of the blog post:

click has an hands-off approach and the author feels that approach isn't appropriate when the CLI is your user interface.

---

That's a fair argument, made in a fairly emotional tone IMO. (maybe that python-dev thread making me look at things this way)

I'm happy with both options and likely argparse is going to be easier to switch to anyways.

[RonnyPfannschmidt]

it should be noted that argparse has a few pretty nasty "bydesign" bugs around automatically aliasing the prefix pf an argument to the full version,
its only opt out in 3.6* and there is no fix for older pythons

[thechief389]

But nevertheless, argparse is the best choice.

[pradyunsg]

@RonnyPfannschmidt do you have any links? My Googling couldn't find anything... :/

[RonnyPfannschmidt]

https://docs.python.org/3/library/argparse.html#allow-abbrev - sorry for quoting the python version wrong, it was added in 3.5

[thechief389]

I'm working on moving to click. Is there anyone else doing it?

[pradyunsg]

I've done a bit of work in the past but ran out of time; mostly was just trying to figure out what all is worth investigating, how things could be done and potential improvements before actually starting working on code.

I do have some notes I can share -- none of what's there in the notes is set in stone but might be a good place to start from. You don't need to respond to every hypothetical in it but those should be figured out as we do this. :)

Please feel free to discuss what you're thinking here.

---

Investigate:

• Configuration interaction
  • Context Defaults?
    • Does the behavior map exactly?
• How click's unicode support affects us: https://click.palletsprojects.com/en/7.x/python3/
• Utilizing the *_option decorators: Is it worth the effort to bodge-use them when we're not doing decorators?
• Argument? Or Context.allow_extra_args? Which is better for us?
• How do we use Context in our context? ;)
• Backwards compatibility?
  • Do we want an "opt-in", "opt-out", "drop" approach here? That's a 3 release cycle removal.
  • But look at how much might break via tests to determine the exact flow.
• Using click's utilities (in place of our own/through dependencies)
  • get_terminal_size
  • prompt
  • progressbar?
  • edit? Maybe?
  • echo? I doubt.
  • style? Can do but should we?
  • get_app_dir? I doubt.
  • echo_via_pager --> should help with Hide global options for subcommands unless --verbose #5746
• click-contrib has some fancy things
  • https://github.com/click-contrib/click-completion: Yes.
  • https://github.com/click-contrib/click-man: should do
    • How to replace existing man page generation? Ping stakeholders on original PR/issue to ask for feedback.
  • https://github.com/click-contrib/click-plugins: BIG FEATURE request -- look at this if we do go that route sometime.
  • https://github.com/click-contrib/click-configfile: Is this relevant to us?
  • https://github.com/click-contrib/click-didyoumean: Replace our custom logic?

Plan:

• Rework the existing command setup to have the same "structure".
• No decorators.
  • Call a method / list of Option in pip.commands? pip.cli.cmdoptions?
• Do A/B testing: use an env var to have conditional for how they're parsed
  • need to edit tox to pass whatever env var

To be used parts of the API:

• Command
• Group
• Option
• Context? (see above)

Avoid direct use:

• ASIDE: why aren't these marked more visibly / separated better? File an issue?
• CommandCollection
• Parameter
• BaseCommand
• MultiCommand

[pradyunsg]

Hey @thechief389! Have you made any progress on this?

[thechief389]

I had trouble trying to vendor all the libraries and their dependencies for click and click-completion.

[techtonik]

Another argument against click. After spending 15 minutes I haven't found the code to implement help in this manner:

1. <command> -h
2. <command> --help
3. help <command>
4. help <command> -h
5. help <command> --help
6. help

There 4. 5. 6. are equivalent to help -h or help --help.

[davidism]

Hello, I'm one of the Pallets (Click) core maintainers. I'm happy to answer questions you have, although I'd ask that you ping me here or in our Discord rather than opening up issues for questions in the repo.

I think we're already set up to be vendorable, and if not we can fix that. If you notice any bugs while trying to integrate Click with Pip, please let us know.

[davidism]

The basic implementation of a help name command, although it should be improved:

@cli.command()
@click.argument("name")
@click.pass_context
def help(ctx, name):
    command = cli.get_command(ctx, name)
    click.echo(command.get_help(ctx))

[pradyunsg]

Thanks for pitching in here @davidsm! :)

I'll ping here if and when I work on this. :)

[cjerdonek]

Because of the size of the code base, I think it's important to have an implementation plan that would let us make progress on this incrementally.

[pradyunsg]

We can't really have half our parsing be optparse and the other half be click. I think you know that. ;) I think you mean what the plan is for separating concerns within pip's codebase about this to ease the actual "switch". My current thought process for that is:

• modify pip's Commands to stop passing "options" to other internal objects (like to pip_version_check).
  • mypy annotations should help here, though I don't think they'll cover 100%.
• create custom Option classes and adding code to convert them into concrete optparse variants would make life easier later on.
• it should be not-too-difficult to create an optparse.Values-like object using click's higher-level classes to be passed to Commands

PS: it's past 1 am. Don't quote me on these. :)

[cjerdonek]

We can't really have half our parsing be optparse and the other half be click.

That's not what I was saying.. I was saying I think it's important for us to have a way to do it that lets us make changes to the code gradually over time. There can be many possible approaches.

For example, maybe there is a way to add a translation / compatibility layer to have whatever we're migrating to (argparse, click, or whatever) accept optparse options. That would let us switch the options one at a time from optparse to the new format.

The point is to avoid a massive PR or massive amounts of code getting turned on at once. The approach should let us incrementally add new code that is getting used as we add it.

[thechief389]

Pipenv uses click and it works well.

[techtonik]

Poetry looks better and uses cleo.

[pradyunsg]

It's pretty simple to use click's underlying classes to make a CLI like pip's. The more fun part is figuring out how the transition for pip's optparse based code to have click options and making commands get Values-like objects, as @cjerdonek pointed out.

Click to reveal code-dump

import inspect
import functools

import click

from .commands import get_commands


def call_with_context(func):
    return func(click.get_current_context())


def create_click_command_from_our_command(our_command):
    # Convert the options; this bit has to be figured out.
    params = [
        click.Option(opt.param_decls, **opt.kwargs) for opt in our_command.options
    ]

    # Construct the click command
    cmd = click.Command(
        our_command.name,
        short_help=inspect.getdoc(our_command),
        callback=functools.partial(call_with_context, our_command.run),
        params=params,
    )

    return cmd


def main():
    cli = click.Group()
    for our_command in get_commands():
        cli.add_command(create_click_command_from_our_command(our_command))

    cli.main(prog_name="pip")

[techtonik]

Click doesn't support custom formatting - pallets/click#561

[davidism]

There are plenty of ways to override formatting now, adding the ability to override the full formatter class (the issue you linked) is an extension to what's already possible.

[thechief389]

Why would we need to customize formatting?

…

Sent from my iPhone

On Feb 9, 2019, at 11:02 PM, anatoly techtonik ***@***.***> wrote: Click doesn't support custom formatting - pallets/click#561 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[pradyunsg]

Here's a rough draft of what I'd imagined we'd end up with in terms of how the commands' option declaration looks like. I'm not looking for feedback / inputs on this at this time. I'm noting it here in case someone else comes around to work on this and would like to know a rough target of what I want us to achieve.

...
from pip._internal.cli.base_command import Command
from pip._internal.cli.options import common_options, Choice, IndexOptionsMixin, Option
...


class ListCommand(Command, IndexOptionsMixin):
    """
    List installed packages, including editables.

    Packages are listed in a case-insensitive sorted order.
    """

    usage = """
      %prog [options]"""

    options = [
        Option(
            "-o", "--outdated",
            is_flag=True,
            help="List outdated packages",
        ),
        Option(
            "-u", "--uptodate",
            is_flag=True,
            help="List up-to-date packages",
        ),
        Option(
            "-e", "--editable",
            is_flag=True,
            help="List editable projects.",
        ),
        Option(
            "-l",
            "--local",
            is_flag=True,
            help=(
                "If in a virtualenv that has global access, do not list "
                "globally-installed packages."
            ),
        ),
        Option(
            "--user",
            is_flag=True,
            help="Only output packages installed in user-site."
        ),
        common_options.list_path,
        Option(
            "--pre",
            is_flag=True,
            help=(
                "Include pre-release and development versions. By default, "
                "pip only finds stable versions."
            ),
        ),
        Option(
            "--format",
            type=Choice(["columns", "freeze", "json"]),
            default="columns",
            dest="list_format",
            help=(
                "Select the output format among: columns (default), freeze, or json"
            ),
        ),
        Option(
            "--not-required",
            is_flag=True,
            help="List packages that are not dependencies of installed packages.",
        ),
        Option(
            "--include-editable/--exclude-editable",
            is_flag=True,
            help="Include editable package from output.",
        ),
    ]

    # No need for wrangling with `self.cmd_opts` in `__init__`
    # Rest of the command definition follows immediately.

[sinscary]

@pradyunsg Is this still up for development?

[pradyunsg]

@sinscary certainly!

[sinscary]

@pradyunsg In that case I would like to give it a try.

[pradyunsg]

Please do! In case you want to have a voice / video chat about this, free free to reach out to me over email. :)

[janluke]

Shameless plug: in case you need additional features for Click, like argparse argument groups, you may consider Cloup. To my knowledge, it's the only package that allows to define option groups without using decorators with Click.

[pradyunsg]

Well, here's a fun thing that's going to make this a bit more painful:

I ran pip wheel setuptools --build /tmp/fooooo (we removed the --build/--build-dir options in 21.3) and optparse has been a very helpful parser and completed that to --build-option. From a breakpoint at the start of WheelCommand.run:

(Pdb) options.build_options
['/tmp/fooooo']
(Pdb) sys.argv
['/Users/pgedam/Developer/totally-normal-nothing-to-see-here-project/.venv/bin/pip', 'wheel', 'setuptools', '--build', '/tmp/fooooo']

🤦🏽

[Shivansh-007]

How about using sdispater/cleo @pradyunsg? It supports the class-based approach that we are wanting to have, along with some other pretty features like text colouring.

How poetry uses it for its CLI - https://github.com/python-poetry/poetry/blob/6485bc23d6497c7731e0f1a635f960b33f2ae99e/src/poetry/console/commands/export.py#L9-L38

[uranusjr]

Actually from the above discussion argparse is by far the best choice. The only blocker against it has been removed now pip has dropped all Python <3.6. And since it’s stdlib, nothing can really beat it at this point (plus migrating from optparse to argparse might actually be the least work due to their similarities in interface design).

[Shivansh-007]

For argparse, we would need to build some features from scratch as we have currently done for optparse, IMO command-line libraries like click and cleo are doing a much great job at the interface they are providing for building apps and have pretty good features.

Migrating to cleo won't be a big pain as they share a bit of similarity, the way we currently have them written is similar to cleo's style.

[uranusjr]

I wouldn’t mind reviewing a POC if you feel it is easy enough to do. Note that we won’t be using much of the features apart from command line parsing (e.g. colors) in either case since those are already done with other mechanisms.

[notatallshaw]

FWIW it seems that while optparse is "deprecated" it's clearly the opinion, in this discussion about an argparse bug, that optparse will never be removed and argparse development itself has largely slowed down in the face of design issues,

[slightlybelowzen]

I'd be happy to work on a small POC for this (using argparse) if there's still interest in moving off of optparse. We can decide where to go from there.

[pradyunsg]

There is, and please feel welcome to do a PoC!

[slightlybelowzen]

Also I don't know if I'm missing something but in the docs it says, the cli/ subpackage is implemented using argparse which seems off?

pip/docs/html/development/architecture/anatomy.rst

Line 75 in 51de88c

* ``cli/`` *[subpackage containing helpers & additional code for managing the command line interface. Uses argparse from stdlib]*

Hopefully after this PR is merged, we won't need to update it😁!

[uranusjr]

The doc entry is mostly likely a typo, it’s added much later than the actual implementation.