feat: allow and deny list support for schema field overwriting (namely label and image fields atm)

[aaron-prindle]

fixes #6416
Related DD here: go/skaffold-field-overwriting
This PR adds the abillity to configure what K8s Group Kind's and which fields specifically are overwritten by skaffold when it renders & deploys manifests. Use cases and related issues include:

• rendering & deploying custom CRDs as part of a skaffold dev session (currently skaffold only supports a set of hardcoded CRDs here)

• rendering and deploying any schema that has immutable spec that would previously cause skaffold to fail on re-deployment

  • example: a StatefulSet w/ .spec.volumeClaimTemplates.* entry will fail on re-deploy prior to fix(labelling): hotfix for issues related to deploy erroring due to nested yaml field overriding in StatefulSet w/ PVC #7011 as skaffold would try to overwrite a StatefulSet's .spec.volumeClaimTemplates.*.metadata.labels immutable field. While fix(labelling): hotfix for issues related to deploy erroring due to nested yaml field overriding in StatefulSet w/ PVC #7011 fixed that specific issue, this PR generalizes to work with any such immutable field as it is user configurable.

• Related Issues
  Allow-List of supported k8s kinds should be used only when applying labels #4489
  Skaffold labels cause incompatibility with subsequent standard kubectl commands #6416
  Question: How to handle immutable tags? #6998
  Image not recognized in crd k8s manifest #4081

Example usage of the transform field:

deploy:
  transform:
    allow:
    - type: Function.openfaas.com
      image: [.spec.image]
      labels: [.spec.metadata.labels]

Open Questions:

• Is transform a good name. IIUC, skaffold v2 introduces a render phase which also has something like render.transform as a field. Any opinions on better alternatives for naming or if that might be a source of confusion worth addressing now once V2 is introduced?

Future Work:

[codecov]

Codecov Report

Merging #7056 (39b6c0a) into main (290280e) will decrease coverage by 2.04%.
The diff coverage is 56.96%.

@@            Coverage Diff             @@
##             main    #7056      +/-   ##
==========================================
- Coverage   70.48%   68.43%   -2.05%     
==========================================
  Files         515      560      +45     
  Lines       23150    26437    +3287     
==========================================
+ Hits        16317    18093    +1776     
- Misses       5776     7094    +1318     
- Partials     1057     1250     +193     

Impacted Files	Coverage Δ
cmd/skaffold/app/cmd/deploy.go	52.00% <ø> (-1.85%)	⬇️
cmd/skaffold/app/cmd/dev.go	84.61% <0.00%> (ø)
cmd/skaffold/app/cmd/render.go	36.66% <0.00%> (-4.72%)	⬇️
cmd/skaffold/skaffold.go	0.00% <0.00%> (ø)
cmd/skaffold/app/cmd/inspect_tests.go	62.50% <14.28%> (-1.14%)	⬇️
cmd/skaffold/app/cmd/lsp.go	28.12% <28.12%> (ø)
cmd/skaffold/app/cmd/fix.go	68.85% <40.00%> (-7.62%)	⬇️
cmd/skaffold/app/cmd/lint.go	42.85% <42.85%> (ø)
cmd/skaffold/app/cmd/find_configs.go	48.88% <50.00%> (+0.24%)	⬆️
cmd/skaffold/app/skaffold.go	76.19% <70.00%> (-8.43%)	⬇️
... and 220 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d2134aa...39b6c0a. Read the comment docs.

[aaron-prindle]

The only failing test - the kokoro test failure is a known flake - TestBuildKanikoInsecureRegistry

[tejal29]

LGTM

[gsquared94]

discussed offline with @aaron-prindle. Having resourceSelector be top level can have issues with applying profiles and config upgrades. It's safer nested under deploy.resourceSelector.
Otherwise, LGTM.

[aaron-prindle]

Current status:
field name is changed: transform -> resourceSelector
resourceSelector is nested under deploy: resourceSelector -> deploy.resourceSelector

Possible issue:
When desiging the feature and schema here, there were some UX reasons as to why the field should not be nested under deploy:

Discussed Alternatives and Tradeoffs For Schema Design
In discussion with the Cloud Deploy team it was noted that the desired CUJ might have better UX if the allow & deny lists could be shared more easily across a team which would mean making these options top level in the schema as it will allow this config to be modularized (use skaffold modules).  The CUJ (summarized) was as follows:
Company X is developing a kubernetes application that deploys custom CRDs
As part of properly configuring skaffold for deploying CRDs, Tech-Lead-1 (TL1) adds the company’s custom CRD (eg: Service.my.custom.crd) is to their allow
TL1 then wants to share their configuration across their team/company for their devs using skaffold so they can deploy their CRD application, in order to do so w/o allow being a top level field, a skaffold profile must be used and passed for each skaffold invocation (vs using a module which is more traditional and well understood by users).  

If the Allow & Deny fields are not top level in the schema (eg: deploy.resourceSelector.allow vs resourceSelector.allow) then the configuration is not sharable by modules but instead requires using profiles which can be more complicated. From the Cloud Deploy Team - “This [the original nested deploy.transform.enable alternative] complicates the Cloud Deploy experience, as modules can be referenced within the yaml (IIUC), but profiles need to be explicitly enabled per pipeline/target.”

@gsquared94 can you clarify if this field is nested if modules can still be used interchangeably for this feature? Eg: a company can have a module which specifies deploy.resourceSelector and that can be consumed easily across their company?

[aaron-prindle]

Final status, Cloud Deploy needs this field to be top level for their module usage requirments.

• field name is changed: transform -> resourceSelector
• resourceSelector is a top level field

[afirth]

@aaron-prindle I'm really excited about this feature! Are there already docs for it? If not, perhaps the DD (https://goto.google.com/skaffold-field-overwriting) could be made public?

[aaron-prindle]

Awesome @afirth! There are public docs for it here:
https://skaffold.dev/docs/tutorials/skaffold-resource-selector/

Please file any issues w/ the docs or feature that you encounter. Currently I would recommend using this for:

• Removing any immutable fields from skaffold's overwriting to avoid deployment errors
• Adding custom CRDs or possibly currently not-status-checked resources to skaffold's status checking.