Migrate to external cloud controller managers for providers with in-tree implementation

[xmudrii]

Kubernetes has announced the removal of in-tree cloud controller managers (CCMs) from the code base and migrating to the external CCMs for all providers.

More details can be found in the following KEP kubernetes/enhancements#669

It's unclear when the in-tree implementations will be removed and replaced with external, but we can assume not before 1.17.

Decision/research items:

• Find out what's the timeline and when we need to/can switch to external implementations
  • Most providers have external implementations by now, but many of them are not stable or don't have any release.
    • OpenStack has several releases, vSphere two beta releases, and Azure has 2 alpha releases.
    • Others providers still don't have releases or are tight with the in-tree implementations
    • Eventually we should only switch if there are (more) stable implementations (e.g. at least a beta release)
• Find out the version compatibility for each external CCM
  • It seems like various CCMs can have different release cycles. We should find out the version compatibility between CCMs and Kubernetes.
  • It may happen that some external CCMs support only newer k8s versions, so we would have to switch between in-tree and external implementations depending on the version.
• Decide how do we want to handle the switch to external CCMs
  • Should we automatically use the external CCM if it is supported or should it be up to the user

Once we decide to switch to external CCMs, we need to:

• Refactor how we handle CCMs
  • Currently, we have the InTreeCloudProvider() function that we use to find out should components be configured to work with the in-tree provider, external provider, or no provider.
  • This is not anymore the case, as in-tree providers have their external implementations
  • This is required, so we pass correct flags to kubelet and other components.
• Add manifests for external CCMs
• Find out how are we going to handle cluster upgrades and what are we going to do with CCMs in that case

[xmudrii]

The upstream will start building Kubernetes without in-tree CCMs for testing purposes as of 1.16. More details can be found in the following KEP kubernetes/enhancements#1179

[xmudrii]

#615 is going to migrate OpenStack to the external implementation.

[kubermatic-bot]

Issues go stale after 90d of inactivity.
After a furter 30 days, they will turn rotten.
Mark the issue as fresh with /remove-lifecycle stale.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kron4eg]

/remove-lifecycle stale

[shaase-ctrl]

/reopen