kustomize fallback can cause version mismatch problems

[nicks]

From the Tilt channel:

I did not have a kustomize binary, and tilt appears to fall back on kubectl kustomize. This call failed, because kubectl (v1.17) uses kustomize v2 and is incompatible with CAPI's kustomizations, which need kustomize v3.

Currently Tilt prints:

Successfully loaded Tiltfile (6.567930778s)
Traceback (most recent call last):
  /home/dlipovetsky/projects/cluster-api/Tiltfile:217:17: in <toplevel>
  /home/dlipovetsky/projects/cluster-api/Tiltfile:205:24: in enable_providers
  /home/dlipovetsky/projects/cluster-api/Tiltfile:170:25: in enable_provider
  <builtin>: in kustomize
Error: command ["kubectl" "kustomize" "/home/dlipovetsky/projects/cluster-api/config/default"] failed.
error: exit status 1
stdout: ""
stderr: "Error: AccumulateTarget: couldn't find target apps_v1_Deployment|system|~P|controller-manager|~S for json patch\n\n\n\nExamples:\n  # Use the current working directory\n  kubectl kustomize .\n  \n  # Use some shared configuration directory\n  kubectl kustomize /home/configuration/production\n  \n  # Use a URL\n  kubectl kustomize github.com/kubernetes-sigs/kustomize.git/examples/helloWorld?ref=v1.0.6\n\nUsage:\n  kubectl kustomize <dir> [flags] [options]\n\nUse \"kubectl options\" for a list of global command-line options (applies to all commands).\n\nAccumulateTarget: couldn't find target apps_v1_Deployment|system|~P|controller-manager|~S for json patch\n\n"

And there's probably an opportunity here for better hints what's wrong. They might include:

• more explicitly pointing out that we're falling back to the kubectl kustomize
• adding min_version parameters to the built-in customize

[dlipovetsky]

• adding min_version parameters to the built-in customize

It is awkward to determine the version of kustomize used by kubectl kustomize. There is no way to get the version, i.e. kubectl kustomize version is not available. So tilt would need to map kubectl to kustomize versions.

Makes we wonder if kubectl kustomize version would be generally useful...

[nicks]

ya, i think that's a great idea, i filed kubernetes/kubectl#797 upstream

[dlipovetsky]

Thanks for filing that issue. Someone tagged it in another issue (kubernetes/kubernetes#85146), and my reading of that one is that the future of kubectl kustomize is unclear.

What do you think of removing the fallback from tilt?

[jazzdan]

Thanks for the pointer @dlipovetsky, I think we should make it more clear when the fallback is happening but wait until a decision is made upstream before removing the fallback altogether. If it continues to be confusing after such a change is made, we can revisit.

Regardless, I think the kustomize v2 vs v3 issue is going to bite people regardless what we do with 'kubectl kustomize', so we may need a versioning primitive either way. (This may also come up with our helm primitive)