More generic support for Python

[itamarst]

It seems that trivy supports poetry.lock and Pipfile.lock for Python, but not requirements.txt.

requirements.txt has dual usage:

1. For some people, it's just high-level dependencies, unpinned to any particular versions.
2. For other, it's equivalent to a lock file, transitively pinned dependencies often with hashes. For example, the pip-tools package takes an unpinned requirements.in and generates a pinned requirements.txt.

More broadly, there's the issue of people who install packages using unpinned mechanisms, e.g. RUN pip install flask in their Dockerfile.

I can imagine a number of approaches:

1. Add a parser for requirements.txt, and just give up if it's unpinned. This will miss some vulnerabilities.
2. Run pip list inside the container to get actually installed packages, and then you don't care how they were installed. The downside is that this is a big difference from your current mode of operation.
3. Do the equivalent of pip list just by inspecting the contents of any site-packages directories you find in the image. I went and asked a pip developer, and turns out all pip list does is look for <package name>-<version>.dist-info directories and uses that to get package name version. E.g. they look like paramiko-2.7.1.dist-info for package paramiko with version 2.7.1

The third option seems like the easiest and most useful: it works for any Python environment and installation mechanism (it ought to work for Conda too, which is yet another packaging tool), and it's quite simple, just listing directories and parsing their names.

[mkjpryor-stfc]

My team uses requirements.txt as a lock file, so support for that would be much appreciated.

More generally, I like the idea of just finding any site-packages directories and scanning the dist-infos.

An alternative approach could be to locate all pip executables and use the output of pip freeze to drive the analysis.

[jithurjacob]

This would be really helpful as we use conda for ML libs and we are not able to use trivy.

[github-actions]

This issue is stale because it has been labeled with inactivity.

[knqyf263]

I've added this task to our backlog.

[knqyf263]

As a first step, we'll support == only.

How it works:

1. Look for all requirementx.txt.
2. Parse requirementx.txt and extract package names and versions
3. Use them for vulnerability detection

How to implement it:

1. Add a new parser here
2. Add a new analyzer for requirements.txt here
3. Update go.mod/go.sum in Trivy

[PratikDhanave]

@knqyf263 I am working on it. work in progress

[Gerry9000]

Looks like this issue has the same goal for python and node packages, using approach #3 from this issue, and the contributor mentioned they have code they can submit:
#1039

[knqyf263]

We finally merged #1169, so Trivy will support requirements.txt in the next version. Note that it detects only == specifier.

[jerbob92]

I have tried adding more generic pip-compile support: aquasecurity/fanal#357
Please let me know what you think.