Hosted Suppression File

[jeremylong]

Due to situations like #4670, #4671, #4677, #4690 - ODC needs to be able to respond faster and provide an updated suppression file when situations like this occur.

1. Implement a hosted suppression file - ODC will download the file to its local cache daily. This will be controllable by a switch so those working in an offline mode can disable the feature.
2. Update the issue-ops for false positives so that if/when a core maintainer (@aikebah, @nhumblot, or myself (@ctxt)) gives the github action bot's comment a thumbs up on a the generated suppression rule (e.g., comment on #4722) the generated suppression rule is then copied into the hosted suppression file, the updated suppression file is tested (to ensure it does not break ODC by not being parsable for some reason), and then the updated file will be automatically published.
   • Note, this will allow the team at release time to copy the suppressions from the hosted file into the release.
   • A schedule for removing the suppressions from the hosted file into the base suppression rules will need to be determined. We don't want to force someone to upgrade, but we also don't want to maintain the rules in two places.

[albuch]

Please make this resilient against network failures and unavailability of the resource (use suppression file that is bundled instead).
We already have a long enough history of data source availability impacting DependencyCheck.

[OrangeDog]

Just to note that the bot-generated suppression is often not good, as it doesn't understand which aspect is wrong, or which additional versions and/or packages would be affected.

[aikebah]

I would say it's often fine, though sometimes multiple FP reports can be combined for a single rule. Many of the reports I could, after review take over the generated suppression, some I had to modify and some were plain wrong.
Having a quick path for the approval and processing of the obviously correct ones eases the process. Those that require special handling would follow a regular code-and-PR process, but the obvious ones would be as plain and simple (or even simpler I guess) as various dependabot PR's that I was able to quickly merge in after changelog review from a mobile device by pushing a few approve/merge buttons

[jeremylong]

The generated suppression file based on approving FP reports can be found here: https://jeremylong.github.io/DependencyCheck/suppressions/publishedSuppressions.xml

[aikebah]

@jeremylong did you already start efforts on part 2 - building the usage and caching of the hosted suppression file? Or is that an activity still to be started?

[jeremylong]

I have not started building that part of the solution yet - planned for the 8.0.0 release. If you want to take that - great. Otherwise I'll get to it as I can ;)

[aikebah]

@jeremylong I'll start working on implementing that part for the 8.0.0 milestone soon... first I'll triage a few of the outstanding FP reports so that our users can already start consuming our hosted suppression file by manual configuration of an additional suppression-file.

[jeremylong]

@aikebah thanks!

[aikebah]

First draft implementation is nearing completion, expect to push the branch as a work-in-progress today, need to see whether I also manage to incorporate setting of the configurable parts (download frequency, hosted suppressions URL) into the advanced settings of the integrations, otherwise those will follow at a later time.

[aikebah]

@jeremylong feel free to already comment on my initial implementation. The pending work is adding configurability of the flags added to the Settings to the various integrations and extending the documentation to take this new 'internet required' functionality into account where applicable. The core engine integration of using the hosted suppressions file as published on the use-hosted-suppressions branch I consider complete.

[jeremylong]

@aikebah quick review - looks good. I've generally added the default values for the settings to the properties files themselves; but having the defaults in the java code is fine as well. The only other thing I might question is the use of the cached data source directly in the analyzer - especially considering how this may play out on a large multi-module maven project setup to run on all modules.

[aikebah]

The only other thing I might question is the use of the cached data source directly in the analyzer

You'd rather see it operate on a copy of the resource like the RetireJS and CVE database? Would be fine with me as well, but thought that with the one-time-init in rules loading I could spare the extra I/O and space-usage of the copy action. I don't mind adding in that defensive copy just in case, will do so after adding the configuration to the

In any case I have a separate issue/PR cooking to update the downloader to take a defensive approach for updating the web resources by not downloading in-place, but downloading to a tempfile inside the datadirectory that replaces the destination only at the end of a successful download - I think that would get rid of most risks of a corrupted web resource (I would expect most corruptions to be an empty file due to a started but failed download)