Remove a package with its dependencies that are not required by other packages

[DarkSuniuM]

What's the problem this feature will solve?
Uninstall a package with its dependencies that are not required by any other package

Describe the solution you'd like
A flag like --remove-dependencies that user can use with pip uninstall <package_name>

For example, I wrongly installed a package (Virtualenv or SystemGlobal) and then I realized I don't need that, Only footage I can see is the log of installation that says which packages are installed after I ran pip install jupyter, There is no way to remove the package and its dependencies!

[benoit-pierre]

One thing that might make implementing this functionality problematic: there's no record of the extras used when a package was installed (which is also a problem with pip check).

[uranusjr]

Another piece missing is what packages are “top-level”. Say I install Django, and then “wrongly” install django-debug-toolbar, I wouldn’t want the hypothetical pip uninstall --remove-dependencies django-debug-toolbar to remove Django (and its dependencies).

[pradyunsg]

This is something that we might want to do moving forward but we would need to first fix the issues with installed package metadata that @benoit-pierre and @uranusjr point out.

IMO The best path forward here would be to resolve those issues and then proceed with the discussion on this feature.

[DarkSuniuM]

This is something that we might want to do moving forward but we would need to first fix the issues with installed package metadata that @benoit-pierre and @uranusjr point out.

IMO The best path forward here would be to resolve those issues and then proceed with the discussion on this feature.

I'm not a professional or a well experienced developer, But I guess each package has a requirements.txt file or something like that, is that right? If so, can't we just use them to lookup the dependencies and check them against other installed packages?

[DarkSuniuM]

Another piece missing is what packages are “top-level”. Say I install Django, and then “wrongly” install django-debug-toolbar, I wouldn’t want the hypothetical pip uninstall --remove-dependencies django-debug-toolbar to remove Django (and its dependencies).

A Metadata file, can has a key like installed-by, The key can be null, or it can be the name of the package that installed the package, No one will install django-debug-toolbar to install django, users will install django, and then install other packages like django-debug-toolbar.
When django is installed first, the installed-by key in the metadata file should be null, and then if u uninstall django-debug-toolbar, then the django package which installed before and is a dependency of django-debug-toolbar won't be uninstalled, but the packages that has installed-by key and the value is django-debug-toolbar will be uninstalled.. I guess it will work well this way

[uranusjr]

I read through the relevent PEPs and realised that, although pip does not implement it, there is already a mechanism to record whether a package is “user-installed” (and should not be removed even if no-one depends on it)—namely, the REQUESTED file in PEP 376. The extras are however still difficult to handle, and would require some extra metadata. The CLI would also need some extra work to sort out edge cases like this:

pip install A[x]
pip install A[y]
pip uninstall --remove-dependencies A     # What should it uninstall?
pip uninstall --remove-dependencies A[x]  # This is not currently possible.

[pradyunsg]

Yep. I can confirm that I had the same findings earlier.

I'd started a discussion about storing the extras on distutils-sig a while back but haven't had the time to follow through on it.

[pradyunsg]

To reiterate, IMO until we store the missing bits information on the filesystem or in an internal database, this feature can't be meaningfully discussed.

[alexchandel]

Amazing how this languishes for 2 years because of bikeshedding. Despite protestations otherwise, it can indeed be meaningfully "discussed" and easily done. pip uninstall --all-dependencies.

This issue is not "out of scope," nor the job of another tool. pip has all the information to uninstall dependencies. This issue is about uninstalling leaves created by an uninstall. That's useful to many of us. It's not about uninstalling leaves that were not manually installed. It's about blindly pruning, just as pip blindly installs without recording the request in any database.

[DarkSuniuM]

Yep. I can confirm that I had the same findings earlier.

I'd started a discussion about storing the extras on distutils-sig a while back but haven't had the time to follow through on it.

I see something related to this is already implemented:

▶ pip show flask
Name: Flask
Version: 1.1.2
Summary: A simple framework for building complex web applications.
Home-page: https://palletsprojects.com/p/flask/
Author: Armin Ronacher
Author-email: [email protected]
License: BSD-3-Clause
Location: /usr/lib/python3.8/site-packages
Requires: Werkzeug, Jinja2, itsdangerous, click
Required-by: Flask-WTF, Flask-SQLAlchemy, Flask-Security-Too, Flask-Script, Flask-Principal, Flask-Paranoid, Flask-Migrate, Flask-Mail, Flask-Login, Flask-Gravatar, Flask-Compress, Flask-BabelEx

Can't we use this "Requires" and "Required-by" to implement "Dependency Removal"?
We can check for the Requires part, then check if they are required by any other packages, if not, we can safely remove them as well recursively

[Saphyel]

@DarkSuniuM I was thinking in your example if you install only django-debug-toolbar and you remove that one make sense that django is removed too, but if you install both and you unistall one of them the other should be still with all the dependencies.

I think the easiest solution for this is having a global pyproject file so you can keep track of which libraries you installed/added in the system so when you uninstall something pip should resolve which dependencies should keep and delete the orphans

[DarkSuniuM]

@DarkSuniuM I was thinking in your example if you install only django-debug-toolbar and you remove that one make sense that django is removed too, but if you install both and you unistall one of them the other should be still with all the dependencies.

I think the easiest solution for this is having a global pyproject file so you can keep track of which libraries you installed/added in the system so when you uninstall something pip should resolve which dependencies should keep and delete the orphans

In the lib/python3.8/site-packages directory, we have a dist-info dir, for each installed library, this dist-info directory, contains a file called INSTALLER, I believe we can store this info here, PACKAGE_INSTALLED_DIRECTLY with a True/False value

[pfmoore]

If anyone genuinely feels that this is a relatively easy problem to solve, and that the pip developers are simply blocking it over details, then I suggest they provide a PR that implements the solution. If it is indeed as simple as people are suggesting, then having a PR that makes it obvious that this is the case would be a very compelling argument for moving forward. On the other hand, if (as I personally believe) there are complexities that make this harder than it looks at first glance, then trying to write a PR would help whoever does so to better understand why this problem is hard.

Simply claiming that someone else "should" be able to implement this without much effort, on the other hand, isn't particularly productive.

The current discussion about ways of storing the required information is reasonable - but please realise that any new metadata that needs to be stored must be agreed as a standard - pip is not the only tool that uses or creates this data, and the standards are how we ensure that tools can work together. By all means design a proposal before proposing a standard, but pip won't accept a PR that changes metadata unless it's backed by an approved standard.

[ayroblu]

Hey, was curious about this, so my understanding is that simply copying https://github.com/enjoysoftware/pip3-autoremove in as a function under a flag is considered out of scope for pip if I'm understanding this correctly?

[RonnyPfannschmidt]

At first glance pip-autoremove is incorrect for anything but trivial cases

[pradyunsg]

I have a slightly different idea for how we can cover this usecase; where users want to ensure their environment matches a certain set of requirements and nothing else. There's a standard lock file format being discussed at https://discuss.python.org/t/11736), which enables a different avenue -- synchronising the environment with the lock file's contents, with the ability to generate that lock file from a requirements file.

[atrigent]

Hello @RonnyPfannschmidt. Can you please explain in what circumstances pip-autoremove is broken, other than issues like the missing extras information and other issues outside of its control?

[ferdnyc]

@atrigent

Can you please explain in what circumstances pip-autoremove is broken

It lacks the concept of "user-installed" packages, separate from dependencies.

Take this scenario, for example:

1. I create and activate a venv
2. I run pip install mypy
   1. The dependency typing-extensions is pulled in
   2. The dependency mypy-extensions is pulled in
3. pip3-autoremove mypy will remove mypy, along with typing-extensions and mypy-extensions.

That's all fine. But in this scenario:

1. I create and activate a venv
2. I run pip install typing-extensions
3. I run pip install mypy
   1. The dependency mypy-extensions is pulled in
4. pip3-autoremove mypy will remove mypy, along with typing-extensions and mypy-extensions.

That last step is wrong. pip3-autoremove mypy SHOULD remove mypy and mypy-extensions, but it SHOULD NOT remove typing-extensions, because in the second scenario typing-extensions was not installed as a dependency of mypy, it was installed explicitly by the user.

RedHat/Fedora's dnf package manager, for example, tracks which packages are explicitly requested by the user (as well as which are installed as part of a package group), and which were pulled in as dependencies. The dnf mark {install,remove,group} subcommands can be used manipulate these flags, marking or unmarking packages as user- or group-installed.

dnf autoremove will only remove packages which are both:

1. Not required by any installed package
2. Not marked as user- or group-installed

This avoids yanking out every package the user installed explicitly, which also happens to be a dependency of some package that's being removed.

(pip3-autoremove could implement this kind of tracking, since any explicitly-installed package will have a REQUESTED file in its .dist-info directory. Packages automatically pulled in as dependencies will lack a REQUESTED file. python3-autoremove should (at least by default, unless passed some additional flag) avoid uninstalling packages with a REQUESTED flag, even if they're explicit dependencies only of the package being uninstalled.)