[release/6.0] Reduce net core app current package dependencies, increase direct update availability

[ericstj]

Customer Impact

• Customer reported
• Found internally

Customers using packages like Microsoft.Extensions.Logging.Console, System.Memory.Data, Microsoft.Extensions.Hosting, etc and targeting net6.0 need to update package references for packages like System.Text.Json, System.Text.Encodings.Web, System.Collections.Immutable, etc even though those libraries are provided by the shared framework.

We can reduce the number of places where folks need to update packages by omitting packages when the same library is provided by the framework.

Additionally we've shipped many packages over the course of net6.0 but haven't shipped every package that depends on them - causing customers to update more transitive dependencies to get updates. We have seen slower up-take for security fixes when transitive updates are required - the goal here is to ensure more direct updates are available.

Regression

• Yes
• No

Testing

Build packages. Add validation to make sure all up-stack packages ship as well. Examined package diffs.

Risk

Low. This is removing package references for a few packages and enabling more packages. The biggest risk here is that we'll be enabling a lot of packages that need to flow in servicing - a total of 55 - but the goal here is that this set of packages will help customers better get clean of CVEs.

Background

This drops package dependencies from all packages which can reference the framework copy of the same library (without downgrading the library exposed to a compatible TFM like netstandard2.0).

I also added a feature to our build that enforces transitive servicing. You can specify ServiceTransitiveDependencies to make sure that you enable all up-stack packages for shipping when enabling a single package.

I enabled all packages I changed, then all upstack packages (separate commits).

This should improve the situation where folks are asked to update just to update a package reference on the latest framework.

It will also help reduce application size since the libraries will no longer be bundled in the app.

PR detail

This is a backport of #107161.

I also reviewed all packages that ever shipped in 6.0 with a CVE to ensure that we've shipped at least one package version that references that - to ensure folks have availability of a direct update for the CVE.

The only "code changes" in this PR are to remove dependencies when the same dependency is in the shared framework. You can see the result of that diff here: ericstj/diffs@499919a

The remaining changes are infrastructure or enabling packages.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

[carlossanlop]

LGTM, just a couple of optional nits.
I'll write down the list of enabled OOB packages.

[ericstj]

Lots of System.IO.Tests failures here which appear to be #100558 but it's not being linked.

[carlossanlop]

How's this looking @ericstj? Friendly reminder that today 10/14 is Code Complete for the November Release.

[ericstj]

Approved over email.

[ericstj]

This needs to resolve conflicts with #108675

[ericstj]

@carlossanlop - Ok, this had some churn with resolving conflicts, but it should be ready to go. The tests failing earlier were all known issues but BA gave up with too many wasm failures.

[ericstj]

Both all configurations legs which build packages have completed. Going to merge this to get the 6.0 servicing release moving