Updater feature request: verify chain of trust from bootstrapped root metadata

[joshuagl]

Description of issue or feature request:

Context:

• In order to comply with the detailed client workflow a TUF client must ship a bootstrap trusted root metadata file out-of band (i.e. with the client installer).

Observations:

• On general purpose operating systems (Linux, Windows, macOS) this bootstrap root metadata will often be installed into operating system owned storage (i.e. the Python that comes with macOS, a distro installed Python on Linux, or Python from the Microsoft Store on Windows).
• In the case of pip, and possibly other integrations, the trusted root metadata downloaded and stored during the detailed client workflow will often be persisted in user owned storage (i.e somewhere in a UNIX users home directory).
• In many cases the bootstrap root metadata will be afforded a much higher level or protection from tampering (SELinux/MAC, integrity verification with IMA/EVM, etc on some Linux distros. System integrity protection for the default Python install on recent macOS.)

With these considerations in mind, it feels like providing a way to verify the chain of trust from the boostrap root metadata to the trusted root metadata will provide some additional integrity protection for the client, to better detect tampering with trusted root metadata that isn't stored in OS protected locations.

Current behavior:

No verification of chain of trust from bootstrap trusted root metadata to current trusted root metadata.

Expected behavior:

Expose an optional method on tuf.client.updater.Updater, to be called before refresh(), which will walk the chain of trust from the bootstrap root metadata to the current trusted root metadata. This method will perform a similar sequence of steps as defined in 5.1 of the detailed client workflow, only it will prefer local copies of the intermediate trusted root metadata before reaching out to the repository to download intermediate root metadata file version.

[jku]

I'll write down the details for pip specifically as an example (like it currently works in my WIP branch):

• every pip installation includes a bootstrap root.json. In some cases these will be in 'safe' locations like /usr/lib/python3/dist-packages/pip/
• when pip starts it checks the configured repository urls (e.g. "https://pypi.org/simple/") and the runtime metadata directory (~/.local/share/pip/). Note that the runtime location is shared for all pip installs
  • If runtime metadata for a repository url does not exist, but bootstrap metadata for a repository url does exist, copy bootstrap metadata to runtime metadata directory
• during pip runtime more metadata will be downloaded to the runtime metadata directory, and stored for later use
• next pip startup will find the runtime metadata directory already populated and will use that cached metadata

So the issue here is that any application running as user can modify ~/.local/share/pip/. We can't prevent that but when pip starts, it could make sure that the boostrap metadata that was installed with pip is used. In some cases that boostrap metadata is more trustworthy than something in the home directory.

The easy solution is to always wipe the runtime metadata directory (or at least root.json) and start from bootstrap but that seems wasteful...

I'll add one caveat to this bug report: the pip design of using shared runtime metadata location has some downsides and might turn out to be wrong. The core problem is still there since we do want to use cached metadata even if it's not shared by multiple installs

[jku]

I think this changes nothing from the above descriptions, but just for the record: When the installed pip gets upgraded, the bootstrap metadata may change -- in an extreme case the new root.json may not even be in the same chain of trust as the old one.

Another thing I had not considered before: we may want to provide multiple bootstrap files (targets.json and bins.json as well as root.json). This is not really useful for verifying the chain of trust of an existing runtime metadata but the point would be to make the first run pip invocation faster: the client would not need to download the large bins.json file (assuming the included file has not expired).

[mnm678]

Another thing I had not considered before: we may want to provide multiple bootstrap files (targets.json and bins.json as well as root.json). This is not really useful for verifying the chain of trust of an existing runtime metadata but the point would be to make the first run pip invocation faster: the client would not need to download the large bins.json file (assuming the included file has not expired).

This would also provide some additional rollback protection for first time clients as they could use bootstrapped targets and maybe snapshot to ensure version numbers have not decreased. The added metadata may make the initial install much larger, but it might be useful in some cases.

[MVrachev]

I will work on this after I push some updates on some of my already opened prs.

[jku]

The added metadata may make the initial install much larger, but it might be useful in some cases.

The point about rollback protection is good... but I think including large files known to be expired (snapshot) is unappealing (In that same train of thought I'm really hoping we get some version of succinct delegation in at some point so I don't have to explain to pip folks how the pip install size just grew by 1.6MB because of bins.json 😬)

In fact, I think snapshot.json in the bootstrap does not bring additional benefits if timestamp is included: the snapshot version number included in timestamp.json should get checked for replay attack according to spec if I'm reading it right.

So for reference, my understanding of the potential bootstrap contents for pip:

1. root.json (required, small size)
2. timestamp.json (additional replay attack protection, small size, will be expired but that should not matter)
3. maybe targets.json+bins.json (huge size currently, no security benefit but including these potentially helps first run experience)

Does that seem correct? The last point is really debatable even now and succinct delegations, if it appears, will make it more so.

[jku]

The original issue (chaining root trust from bootstrap root.json) should now be fairly easy to implement in ngclient:

• Updater should take the bootstrap root file (or bytes?) as optional constructor argument, and initialize the TrustedMetadataSet with it
• Updater should be modified to store versioned valid root metadata files (<N>.root.json) in the local metadata store. Even if Updater was not itself bootstrapped, it should store versioned root files. maybe also any time a versioned root is stored, non-versioned root.json(symlink?) should be stored so non-bootstrap updater knows what to start with.
• When loading root metadata if a bootstrap root.json is given, Updater should load local metadata versions (starting from bootstrap version onwards) first, and continue with remote if the local one is not available or valid. So root update would look like:
  • initialization:
    • if boostrapped: Updater gets bootstrap root vN as constructor argument, loads into TrustedMetadataSet, stores versioned copy in cache
    • else: Updater reads non-versioned root from local cache, loads into TrustedMetadataSet (this is what currently happens)
  • Updater root update loop continues as long as root versions are found:
    • if valid root N+1 found in local cache, load into TrustedMetadataSet
    • elif valid root N+1 found on remote server, load into TrustedMetadataSet, store a versioned copy in cache
    • else break update loop, consider N the final root and continue as usual to timestamp update

[jku]

There's a first draft in https://github.com/jku/python-tuf/commits/bootstrap-root-metadata -- I don't know if I'll push this further: Feel free to take this code and turn into a PR. Needs tests (at the very least).

[jku]

Status:

• branch exists and is up-do-date at time of writing: https://github.com/jku/python-tuf/commits/bootstrap-root-metadata
• it seems to work fine based on existing and added tests as well as manual testing

TODO:

• Maybe: For safety, before this PR, add tests to make sure current version is ok with root.json being a symlink to a versioned file...
• fix existing tests that choke on the added subdirectory in the metadata cache dir
• Add tests for bootstrap feature in the PR (some have been added)
• Make sure the local metadata caching changes are not problematic:
  • The PR adds a "root_history" subdirectory in the apps local metadata cache directory: this seems ok to me but maybe someone in community can come up with a security problem here
  • non-versioned "root.json" is now a symlink to "root_history/N.root.json": this is for two reasons
    • old versions of python-tuf won't break when same cache directory is used
    • calling ngclient without bootstrap argument keeps working as it used to (just load the cached non-versioned root.json)
• Make sure tuf-conformance does not choke on this additional directory or the use of symlink (alternatively, remove the symlink/directory in the confomance client)