Make recursive cloning (e.g. of Git submodules) configurable

[dgutson]

We would need the ability to exclude the cloning of the (git) submodules of a given repository. I would like to be able to specify that as an exclude in the .ort.yml file. Maybe there could be a section regarding the behavior of the downloader.

[mnonnenmacher]

@dgutson Can you explain where exactly you need this option? Do you want an option for the download command to disable recursive cloning? Or do you want a way to configure the scanner to not clone a specific repository recursively?

[dgutson]

@mnonnenmacher there are large repositories included as submodules that should be skipped. Currently there is no way to avoid these to be both downloaded (cloned) and analyzed.

[sschuberth]

@dgutson, what @mnonnenmacher was asking is: Do you want this to be a new CLI option to the stand-alone ort download sub-command, or a configuration option to the scanner (which implicitly calls the downloader programmatically).

Given that you wrote

I would like to be able to specify that as an exclude in the .ort.yml file.

I understand that you only want the latter, a way for the scanner (not downloader CLI) to not download Git submodules, and thus omit them from scanning.

Maybe we could make path excludes smart to understand that a path refers to a submodule, and then not clone that submodule.

[dgutson]

@arieltorti what do you think?

[mnonnenmacher]

I understood the response differently:

Currently there is no way to avoid these to be both downloaded (cloned) and analyzed.

As you say "analyzed" my assumption was that you use the downloader to download a repository before running the analyzer. So you would need an option for the download command to not clone recursively, but this would also require changes to the scanner because currently it would still do a recursive clone. However, before we can say if and how this can be implemented we would need to understand the use-case better.

[arieltorti]

I understand that you only want the latter, a way for the scanner (not downloader CLI) to not download Git submodules, and thus omit them from scanning.

That will work.

As you say "analyzed" my assumption was that you use the downloader to download a repository before running the analyzer.

We run ort analyze directly on the repository, without ort download. The idea would be to exclude certain submodules to avoid downloading non-analyzable data or duplicated submodules (because they were already scanned in a standalone way)