Skip to content

Latest commit

 

History

History
1519 lines (1339 loc) · 198 KB

CHANGELOG-1.17.md

File metadata and controls

1519 lines (1339 loc) · 198 KB
Raw

v1.17.0

Documentation

Downloads for v1.17.0

filename sha512 hash
kubernetes.tar.gz 68d5af15901281954de01164426cfb5ca31c14341387fad34d0cb9aa5f40c932ad44f0de4f987caf2be6bdcea2051e589d25878cf4f9ac0ee73048029a11825f
kubernetes-src.tar.gz 5424576d7f7936df15243fee0036e7936d2d6224e98ac805ce96cdf7b83a7c5b66dfffc8823d7bc0c17c700fa3c01841208e8cf89be91d237d12e18f3d2f307c

Client Binaries

filename sha512 hash
kubernetes-client-darwin-386.tar.gz 4c9a06409561b8ecc8901d0b88bc955ab8b8c99256b3f6066811539211cff5ba7fb9e3802ac2d8b00a14ce619fa82aeebe83eae9f4b0774bedabd3da0235b78b
kubernetes-client-darwin-amd64.tar.gz 78ce6875c5f5a03bc057e7194fd1966beb621f825ba786d35a9921ab1ae33ed781d0f93a473a6b985da1ba4fbe95c15b23cdca9e439dfd653dbcf5a2b23d1a73
kubernetes-client-linux-386.tar.gz 7a4bcd7d06d0f4ba929451f652c92a3c4d428f9b38ed83093f076bb25699b9c4e82f8f851ab981e68becbf10b148ddab4f7dce3743e84d642baa24c00312a2aa
kubernetes-client-linux-amd64.tar.gz 7f9fc9ac07e9acbf12b58ae9077a8ce1f7fb4b5ceccd3856b55d2beb5e435d4fd27884c10ffdf3e2e18cafd4acc001ed5cf2a0a9a5b0545d9be570f63012d9c0
kubernetes-client-linux-arm.tar.gz 8f74fff80a000cfaefa2409bdce6fd0d546008c7942a7178a4fa88a9b3ca05d10f34352e2ea2aec5297aa5c630c2b9701b507273c0ed0ddc0c297e57b655d62e
kubernetes-client-linux-arm64.tar.gz 18d92b320f138f5080f98f1ffee20e405187549ab3aad55b7f60f02e3b7f5a44eb9826098576b42937fd0aac01fe6bcae36b5a8ee52ddde3571a1281b279c114
kubernetes-client-linux-ppc64le.tar.gz fd9b15a88b3d5a506a84ebfb56de291b85978b14f61a2c05f4bdb6a7e45a36f92af5a024a6178dbebd82a92574ec6d8cf9d8ac912f868f757649a2a8434011fe
kubernetes-client-linux-s390x.tar.gz ae3b284a78975cbfccaac04ea802085c31fd75cccf4ece3a983f44faf755dd94c43833e60f52c5ea57bc462cb24268ef4b7246876189113f588a012dd58e9630
kubernetes-client-windows-386.tar.gz 4ba83b068e7f4a203bcc5cc8bb2c456a6a9c468e695f86f69d8f2ac81be9a1ce156f9a2f28286cb7eb0480faac397d964821c009473bdb443d84a30b6d020551
kubernetes-client-windows-amd64.tar.gz fc79b0e926a823c7d8b9010dee0c559587b7f97c9290b2126d517c4272891ce36e310a64c85f3861a1c951da8dc21f46244a59ff9d52b7b7a3f84879f533e6aa

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz 28b2703c95894ab0565e372517c4a4b2c33d1be3d778fae384a6ab52c06cea7dd7ec80060dbdba17c8ab23bbedcde751cccee7657eba254f7d322cf7c4afc701
kubernetes-server-linux-arm.tar.gz b36a9f602131dba23f267145399aad0b19e97ab7b5194b2e3c01c57f678d7b0ea30c1ea6b4c15fd87b1fd3bf06abd4ec443bef5a3792c0d813356cdeb3b6a935
kubernetes-server-linux-arm64.tar.gz 42adae077603f25b194e893f15e7f415011f25e173507a190bafbee0d0e86cdd6ee8f11f1bcf0a5366e845bd968f92e5bf66785f20c1125c801cf3ec9850d0bd
kubernetes-server-linux-ppc64le.tar.gz 7e72d4255e661e946203c1c0c684cd0923034eb112c35e3ba08fbf9d1ef5e8bb291840c6ff99aea6180083846f9a9ba88387e176ee7a5def49e1d19366e2789f
kubernetes-server-linux-s390x.tar.gz 00bc634654ec7d1ec2eca7a3e943ac287395503a06c8da22b7efb3a35435ceb323618c6d9931d6693bfb19f2b8467ae8f05f98392df8ee4954556c438409c8d4

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 49ef6a41c65b3f26a4f3ffe63b92c8096c26aa27a89d227d935bc06a497c97505ad8bc215b4c5d5ad3af6489c1366cd26ecc8e2781a83f46a91503678abba71b
kubernetes-node-linux-arm.tar.gz 21a213fd572200998bdd71f5ebbb96576fc7a7e7cfb1469f028cc1a310bc2b5c0ce32660629beb166b88f54e6ebecb2022b2ed1fdb902a9b9d5acb193d76fa0f
kubernetes-node-linux-arm64.tar.gz 3642ee5e7476080a44005db8e7282fdbe4e4f220622761b95951c2c15b3e10d7b70566bfb7a9a58574f3fc385d5aae80738d88195fa308a07f199cee70f912f4
kubernetes-node-linux-ppc64le.tar.gz 99687088be50a794894911d43827b7e1125fbc86bfba799f77c096ddaa5b2341b31d009b8063a177e503ce2ce0dafbda1115216f8a5777f34e0e2d81f0114104
kubernetes-node-linux-s390x.tar.gz 73b9bc356de43fbed7d3294be747b83e0aac47051d09f1df7be52c33be670b63c2ea35856a483ebc2f57e30a295352b77f1b1a6728afa10ec1f3338cafbdb2bb
kubernetes-node-windows-amd64.tar.gz 2fbc80f928231f60a5a7e4f427953ef17244b3a8f6fdeebcbfceb05b0587b84933fa723898c64488d94b9ce180357d6d4ca1505ca3c3c7fb11067b7b3bf6361b

Changes

A complete changelog for the release notes is now hosted in a customizable format at relnotes.k8s.io. Check it out and please give us your feedback!

What’s New (Major Themes)

Cloud Provider Labels reach General Availability

Added as a beta feature way back in v1.2, v1.17 sees the general availability of cloud provider labels.

Volume Snapshot Moves to Beta

The Kubernetes Volume Snapshot feature is now beta in Kubernetes v1.17. It was introduced as alpha in Kubernetes v1.12, with a second alpha with breaking changes in Kubernetes v1.13.

CSI Migration Beta

The Kubernetes in-tree storage plugin to Container Storage Interface (CSI) migration infrastructure is now beta in Kubernetes v1.17. CSI migration was introduced as alpha in Kubernetes v1.14.

Known Issues

  • volumeDevices mapping ignored when container is privileged
  • The Should recreate evicted statefulset conformance test fails because Pod ss-0 expected to be re-created at least once. This was caused by the Predicate PodFitsHostPorts failed scheduling error. The root cause was a host port conflict for port 21017. This port was in-use as an ephemeral port by another application running on the node. This will be looked at for the 1.18 release.
  • client-go discovery clients constructed using NewDiscoveryClientForConfig or NewDiscoveryClientForConfigOrDie default to rate limits that cause normal discovery request patterns to take several seconds. This is fixed in https://issue.k8s.io/86168 and will be resolved in v1.17.1. As a workaround, the Burst value can be adjusted higher in the rest.Config passed into NewDiscoveryClientForConfig or NewDiscoveryClientForConfigOrDie.

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

Cluster Lifecycle

  • Kubeadm: add a new kubelet-finalize phase as part of the init workflow and an experimental sub-phase to enable automatic kubelet client certificate rotation on primary control-plane nodes. Prior to 1.17 and for existing nodes created by kubeadm init where kubelet client certificate rotation is desired, you must modify /etc/kubernetes/kubelet.conf to point to the PEM symlink for rotation: client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem and client-key: /var/lib/kubelet/pki/kubelet-client-current.pem, replacing the embedded client certificate and key. (#84118, @neolit123)

Network

  • EndpointSlices: If upgrading a cluster with EndpointSlices already enabled, any EndpointSlices that should be managed by the EndpointSlice controller should have a http://endpointslice.kubernetes.io/managed-by label set to endpointslice-controller.k8s.io.

Scheduling

  • Kubeadm: when adding extra apiserver authorization-modes, the defaults Node,RBAC are no longer prepended in the resulting static Pod manifests and a full override is allowed. (#82616, @ghouscht)

Storage

  • All nodes need to be drained before upgrading Kubernetes cluster, because paths used for block volumes are changed in this release, so on-line upgrade of nodes aren't allowed. (#74026, @mkimuram)

Windows

  • The Windows containers RunAsUsername feature is now beta.
  • Windows worker nodes in a Kubernetes cluster now support Windows Server version 1903 in addition to the existing support for Windows Server 2019
  • The RuntimeClass scheduler can now simplify steering Linux or Windows pods to appropriate nodes
  • All Windows nodes now get the new label node.kubernetes.io/windows-build that reflects the Windows major, minor, and build number that are needed to match compatibility between Windows containers and Windows worker nodes.

Deprecations and Removals

  • kubeadm.k8s.io/v1beta1 has been deprecated, you should update your config to use newer non-deprecated API versions. (#83276, @Klaven)
  • The deprecated feature gates GCERegionalPersistentDisk, EnableAggregatedDiscoveryTimeout and PersistentLocalVolumes are now unconditionally enabled and can no longer be specified in component invocations. (#82472, @draveness)
  • Deprecate the default service IP CIDR. The previous default was 10.0.0.0/24 which will be removed in 6 months/2 releases. Cluster admins must specify their own desired value, by using --service-cluster-ip-range on kube-apiserver. (#81668, @darshanime)
  • Remove deprecated "include-uninitialized" flag. (#80337, @draveness)
  • All resources within the rbac.authorization.k8s.io/v1alpha1 and rbac.authorization.k8s.io/v1beta1 API groups are deprecated in favor of rbac.authorization.k8s.io/v1, and will no longer be served in v1.20. (#84758, @liggitt)
  • The certificate signer no longer accepts ca.key passwords via the CFSSL_CA_PK_PASSWORD environment variable. This capability was not prompted by user request, never advertised, and recommended against in the security audit. (#84677, @mikedanese)
  • Deprecate the instance type beta label (beta.kubernetes.io/instance-type) in favor of its GA equivalent: node.kubernetes.io/instance-type (#82049, @andrewsykim)
  • The built-in system:csi-external-provisioner and system:csi-external-attacher cluster roles are removed as of 1.17 release (#84282, @tedyu)
  • The in-tree GCE PD plugin kubernetes.io/gce-pd is now deprecated and will be removed in 1.21. Users that self-deploy Kubernetes on GCP should enable CSIMigration + CSIMigrationGCE features and install the GCE PD CSI Driver (https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver) to avoid disruption to existing Pod and PVC objects at that time. Users should start using the GCE PD CSI CSI Driver directly for any new volumes. (#85231, @davidz627)
  • The in-tree AWS EBS plugin kubernetes.io/aws-ebs is now deprecated and will be removed in 1.21. Users that self-deploy Kubernetes on AWS should enable CSIMigration + CSIMigrationAWS features and install the AWS EBS CSI Driver (https://github.com/kubernetes-sigs/aws-ebs-csi-driver) to avoid disruption to existing Pod and PVC objects at that time. Users should start using the AWS EBS CSI CSI Driver directly for any new volumes. (#85237, @leakingtapan)
  • The CSINodeInfo feature gate is deprecated and will be removed in a future release. The storage.k8s.io/v1beta1 CSINode object is deprecated and will be removed in a future release. (#83474, @msau42)
  • Removed Alpha feature MountContainers (#84365, @codenrhoden)
  • Removed plugin watching of the deprecated directory {kubelet_root_dir}/plugins and CSI V0 support in accordance with deprecation announcement in https://v1-13.docs.kubernetes.io/docs/setup/release/notes (#84533, @davidz627)
  • kubeadm deprecates the use of the hyperkube image (#85094, @rosti)

Metrics Changes

Added metrics

  • Add scheduler_goroutines metric to track number of kube-scheduler binding and prioritizing goroutines (#83535, @wgliang)
  • Adding initial EndpointSlice metrics. (#83257, @robscott)
  • Adds a metric apiserver_request_error_total to kube-apiserver. This metric tallies the number of request_errors encountered by verb, group, version, resource, subresource, scope, component, and code. (#83427, @logicalhan)
  • A new kubelet_preemptions metric is reported from Kubelets to track the number of preemptions occuring over time, and which resource is triggering those preemptions. (#84120, @smarterclayton)
  • Kube-apiserver: Added metrics authentication_latency_seconds that can be used to understand the latency of authentication. (#82409, @RainbowMango)
  • Add plugin_execution_duration_seconds metric for scheduler framework plugins. (#84522, @liu-cong)
  • Add permit_wait_duration_seconds metric to the scheduler. (#84011, @liu-cong)

Deprecated/changed metrics

  • etcd version monitor metrics are now marked as with the ALPHA stability level. (#83283, @RainbowMango)
  • Change pod_preemption_victims metric from Gauge to Histogram. (#83603, @Tabrizian)
  • Following metrics from kubelet are now marked as with the ALPHA stability level: kubelet_container_log_filesystem_used_bytes kubelet_volume_stats_capacity_bytes kubelet_volume_stats_available_bytes kubelet_volume_stats_used_bytes kubelet_volume_stats_inodes kubelet_volume_stats_inodes_free kubelet_volume_stats_inodes_used plugin_manager_total_plugins volume_manager_total_volumes (#84907, @RainbowMango)
  • Deprecated metric rest_client_request_latency_seconds has been turned off. (#83836, @RainbowMango)
  • Following metrics from kubelet are now marked as with the ALPHA stability level: node_cpu_usage_seconds_total node_memory_working_set_bytes container_cpu_usage_seconds_total container_memory_working_set_bytes scrape_error (#84987, @RainbowMango)
  • Deprecated prometheus request meta-metrics have been removed http_request_duration_microseconds http_request_duration_microseconds_sum http_request_duration_microseconds_count http_request_size_bytes http_request_size_bytes_sum http_request_size_bytes_count http_requests_total, http_response_size_bytes http_response_size_bytes_sum http_response_size_bytes_count due to removal from the prometheus client library. Prometheus http request meta-metrics are now generated from promhttp.InstrumentMetricHandler instead.
  • Following metrics from kube-controller-manager are now marked as with the ALPHA stability level: storage_count_attachable_volumes_in_use attachdetach_controller_total_volumes pv_collector_bound_pv_count pv_collector_unbound_pv_count pv_collector_bound_pvc_count pv_collector_unbound_pvc_count (#84896, @RainbowMango)
  • Following metrics have been turned off: apiserver_request_count apiserver_request_latencies apiserver_request_latencies_summary apiserver_dropped_requests etcd_request_latencies_summary apiserver_storage_transformation_latencies_microseconds apiserver_storage_data_key_generation_latencies_microseconds apiserver_storage_transformation_failures_total (#83837, @RainbowMango)
  • Following metrics have been turned off: scheduler_scheduling_latency_seconds scheduler_e2e_scheduling_latency_microseconds scheduler_scheduling_algorithm_latency_microseconds scheduler_scheduling_algorithm_predicate_evaluation scheduler_scheduling_algorithm_priority_evaluation scheduler_scheduling_algorithm_preemption_evaluation scheduler_scheduling_binding_latency_microseconds ([#83838](https://github.com/kubernetes/kubernetes/pull/83838), @RainbowMango)
  • Deprecated metric kubeproxy_sync_proxy_rules_latency_microseconds has been turned off. (#83839, @RainbowMango)

Notable Features

Stable

  • Graduate ScheduleDaemonSetPods to GA. (feature gate will be removed in 1.18) (#82795, @draveness)
  • Graduate TaintNodesByCondition to GA in 1.17. (feature gate will be removed in 1.18) (#82703, @draveness)
  • The WatchBookmark feature is promoted to GA. With WatchBookmark feature, clients are able to request watch events with BOOKMARK type. See https://kubernetes.io/docs/reference/using-api/api-concepts/#watch-bookmarks for more details. (#83195, @wojtek-t)
  • Promote NodeLease feature to GA. The feature make Lease object changes an additional healthiness signal from Node. Together with that, we reduce frequency of NodeStatus updates to 5m by default in case of no changes to status itself (#84351, @wojtek-t)
  • CSI Topology feature is GA. (#83474, @msau42)
  • The VolumeSubpathEnvExpansion feature is graduating to GA. The VolumeSubpathEnvExpansion feature gate is unconditionally enabled, and will be removed in v1.19. (#82578, @kevtaylor)
  • Node-specific volume limits has graduated to GA. (#83568, @bertinatto)
  • The ResourceQuotaScopeSelectors feature has graduated to GA. The ResourceQuotaScopeSelectors feature gate is now unconditionally enabled and will be removed in 1.18. (#82690, @draveness)

Beta

  • The Kubernetes Volume Snapshot feature has been moved to beta. The VolumeSnapshotDataSource feature gate is on by default in this release. This feature enables you to take a snapshot of a volume (if supported by the CSI driver), and use the snapshot to provision a new volume, pre-populated with data from the snapshot.
  • Feature gates CSIMigration to Beta (on by default) and CSIMigrationGCE to Beta (off by default since it requires installation of the GCE PD CSI Driver) (#85231, @davidz627)
  • EndpointSlices are now beta but not yet enabled by default. Use the EndpointSlice feature gate to enable this feature. (#85365, @robscott)
  • Promote CSIMigrationAWS to Beta (off by default since it requires installation of the AWS EBS CSI Driver) (#85237, @leakingtapan)
  • Moving Windows RunAsUserName feature to beta (#84882, @marosset)

CLI Improvements

  • The kubectl's api-resource command now has a --sort-by flag to sort resources by name or kind. (#81971, @laddng)
  • A new --prefix flag added into kubectl logs which prepends each log line with information about it's source (pod name and container name) (#76471, @m1kola)

API Changes

  • CustomResourceDefinitions now validate documented API semantics of x-kubernetes-list-type and x-kubernetes-map-type atomic to reject non-atomic sub-types. (#84722, @sttts)
  • Kube-apiserver: The AdmissionConfiguration type accepted by --admission-control-config-file has been promoted to apiserver.config.k8s.io/v1 with no schema changes. (#85098, @liggitt)
  • Fixed EndpointSlice port name validation to match Endpoint port name validation (allowing port names longer than 15 characters) (#84481, @robscott)
  • CustomResourceDefinitions introduce x-kubernetes-map-type annotation as a CRD API extension. Enables this particular validation for server-side apply. (#84113, @enxebre)

Other notable changes

API Machinery

  • kube-apiserver: the --runtime-config flag now supports an api/beta=false value which disables all built-in REST API versions matching v[0-9]+beta[0-9]+. (#84304, @liggitt) The --feature-gates flag now supports an AllBeta=false value which disables all beta feature gates. (#84304, @liggitt)
  • New flag --show-hidden-metrics-for-version in kube-apiserver can be used to show all hidden metrics that deprecated in the previous minor release. (#84292, @RainbowMango)
  • kube-apiserver: Authentication configuration for mutating and validating admission webhooks referenced from an --admission-control-config-file can now be specified with apiVersion: apiserver.config.k8s.io/v1, kind: WebhookAdmissionConfiguration. (#85138, @liggitt)
  • kube-apiserver: The ResourceQuota admission plugin configuration referenced from --admission-control-config-file admission config has been promoted to apiVersion: apiserver.config.k8s.io/v1, kind: ResourceQuotaConfiguration with no schema changes. (#85099, @liggitt)
  • kube-apiserver: fixed a bug that could cause a goroutine leak if the apiserver encountered an encoding error serving a watch to a websocket watcher (#84693, @tedyu)
  • Fix the bug that EndpointSlice for masters wasn't created after enabling EndpointSlice feature on a pre-existing cluster. (#84421, @tnqn)
  • Switched intstr.Type to sized integer to follow API guidelines and improve compatibility with proto libraries (#83956, @liggitt)
  • Client-go: improved allocation behavior of the delaying workqueue when handling objects with far-future ready times. (#83945, @barkbay)
  • Fixed an issue with informers missing an Added event if a recently deleted object was immediately recreated at the same time the informer dropped a watch and relisted. (#83911, @matte21)
  • Fixed panic when accessing CustomResources of a CRD with x-kubernetes-int-or-string. (#83787, @sttts)
  • The resource version option, when passed to a list call, is now consistently interpreted as the minimum allowed resource version. Previously when listing resources that had the watch cache disabled clients could retrieve a snapshot at that exact resource version. If the client requests a resource version newer than the current state, a TimeoutError is returned suggesting the client retry in a few seconds. This behavior is now consistent for both single item retrieval and list calls, and for when the watch cache is enabled or disabled. (#72170, @jpbetz)
  • Fixes a goroutine leak in kube-apiserver when a request times out. (#83333, @lavalamp)
  • Fixes the bug in informer-gen that it produces incorrect code if a type has nonNamespaced tag set. (#80458, @tatsuhiro-t)
  • Resolves bottleneck in internal API server communication that can cause increased goroutines and degrade API Server performance (#80465, @answer1991)
  • Resolves regression generating informers for packages whose names contain . characters (#82410, @nikhita)
  • Resolves issue with /readyz and /livez not including etcd and kms health checks (#82713, @logicalhan)
  • Fixes regression in logging spurious stack traces when proxied connections are closed by the backend (#82588, @liggitt)
  • Kube-apiserver now reloads serving certificates from disk every minute to allow rotation without restarting the server process (#84200, @jackkleeman)
  • Client-ca bundles for the all generic-apiserver based servers will dynamically reload from disk on content changes (#83579, @deads2k)
  • Client-go: Clients can request protobuf and json and correctly negotiate with the server for JSON for CRD objects, allowing all client libraries to request protobuf if it is available. If an error occurs negotiating a watch with the server, the error is immediately return by the client Watch() method instead of being sent as an Error event on the watch stream. (#84692, @smarterclayton) Renamed FeatureGate RequestManagement to APIPriorityAndFairness. This feature gate is an alpha and has not yet been associated with any actual functionality. (#85260, @MikeSpreitzer)
  • Filter published OpenAPI schema by making nullable, required fields non-required in order to avoid kubectl to wrongly reject null values. (#85722, @sttts)
  • kube-apiserver: fixed a conflict error encountered attempting to delete a pod with gracePeriodSeconds=0 and a resourceVersion precondition (#85516, @michaelgugino)
  • Use context to check client closed instead of http.CloseNotifier in processing watch request which will reduce 1 goroutine for each request if proto is HTTP/2.x . (#85408, @answer1991)
  • Reload apiserver SNI certificates from disk every minute (#84303, @jackkleeman)
  • The mutating and validating admission webhook plugins now read configuration from the admissionregistration.k8s.io/v1 API. (#80883, @liggitt)
  • kube-proxy: a configuration file specified via --config is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. (#82927, @obitech)
  • When registering with a 1.17+ API server, MutatingWebhookConfiguration and ValidatingWebhookConfiguration objects can now request that only v1 AdmissionReview requests be sent to them. Previously, webhooks were required to support receiving v1beta1 AdmissionReview requests as well for compatibility with API servers <= 1.15.
    • When registering with a 1.17+ API server, a CustomResourceDefinition conversion webhook can now request that only v1 ConversionReview requests be sent to them. Previously, conversion webhooks were required to support receiving v1beta1 ConversionReview requests as well for compatibility with API servers <= 1.15. (#82707, @liggitt)
  • OpenAPI v3 format in CustomResourceDefinition schemas are now documented. (#85381, @sttts)
  • kube-apiserver: Fixed a regression accepting patch requests > 1MB (#84963, @liggitt)
  • The example API server has renamed its wardle.k8s.io API group to wardle.example.com (#81670, @liggitt)
  • CRDs defaulting is promoted to GA. Note: the feature gate CustomResourceDefaulting will be removed in 1.18. (#84713, @sttts)
  • Restores compatibility with <=1.15.x custom resources by not publishing OpenAPI for non-structural custom resource definitions (#82653, @liggitt)
  • If given an IPv6 bind-address, kube-apiserver will now advertise an IPv6 endpoint for the kubernetes.default service. (#84727, @danwinship)
  • Add table convertor to component status. (#85174, @zhouya0)
  • Scale custom resource unconditionally if resourceVersion is not provided (#80572, @knight42)
  • When the go-client reflector relists, the ResourceVersion list option is set to the reflector's latest synced resource version to ensure the reflector does not "go back in time" and reprocess events older than it has already processed. If the server responds with an HTTP 410 (Gone) status code response, the relist falls back to using resourceVersion="". (#83520, @jpbetz)
  • Fix unsafe JSON construction in a number of locations in the codebase (#81158, @zouyee)
  • Fixes a flaw (CVE-2019-11253) in json/yaml decoding where large or malformed documents could consume excessive server resources. Request bodies for normal API requests (create/delete/update/patch operations of regular resources) are now limited to 3MB. (#83261, @liggitt)
  • CRDs can have fields named type with value array and nested array with items fields without validation to fall over this. (#85223, @sttts)

Apps

  • Support Service Topology (#72046, @m1093782566)
  • Finalizer Protection for Service LoadBalancers is now in GA (enabled by default). This feature ensures the Service resource is not fully deleted until the correlating load balancer resources are deleted. (#85023, @MrHohn)
  • Pod process namespace sharing is now Generally Available. The PodShareProcessNamespace feature gate is now deprecated and will be removed in Kubernetes 1.19. (#84356, @verb)
  • Fix handling tombstones in pod-disruption-budged controller. (#83951, @zouyee)
  • Fixed the bug that deleted services were processed by EndpointSliceController repeatedly even their cleanup were successful. (#82996, @tnqn)
  • Add RequiresExactMatch for label.Selector (#85048, @shaloulcy)
  • Adds a new label to indicate what is managing an EndpointSlice. (#83965, @robscott)
  • Fix handling tombstones in pod-disruption-budged controller. (#83951, @zouyee)
  • Fixed the bug that deleted services were processed by EndpointSliceController repeatedly even their cleanup were successful. (#82996, @tnqn)
  • An end-user may choose to request logs without confirming the identity of the backing kubelet. This feature can be disabled by setting the AllowInsecureBackendProxy feature-gate to false. (#83419, @deads2k)
  • When scaling down a ReplicaSet, delete doubled up replicas first, where a "doubled up replica" is defined as one that is on the same node as an active replica belonging to a related ReplicaSet. ReplicaSets are considered "related" if they have a common controller (typically a Deployment). (#80004, @Miciah)
  • Kube-controller-manager: Fixes bug setting headless service labels on endpoints (#85361, @liggitt)
  • People can see the right log and note. (#84637, @zhipengzuo)
  • Clean duplicate GetPodServiceMemberships function (#83902, @gongguan)

Auth

  • K8s docker config json secrets are now compatible with docker config desktop authentication credentials files (#82148, @bbourbie)
  • Kubelet and aggregated API servers now use v1 TokenReview and SubjectAccessReview endpoints to check authentication/authorization. (#84768, @liggitt)
  • Kube-apiserver can now specify --authentication-token-webhook-version=v1 or --authorization-webhook-version=v1 to use v1 TokenReview and SubjectAccessReview API objects when communicating with authentication and authorization webhooks. (#84768, @liggitt)
  • Authentication token cache size is increased (from 4k to 32k) to support clusters with many nodes or many namespaces with active service accounts. (#83643, @lavalamp)
  • Apiservers based on k8s.io/apiserver with delegated authn based on cluster authentication will automatically update to new authentication information when the authoritative configmap is updated. (#85004, @deads2k)
  • Configmaps/extension-apiserver-authentication in kube-system is continuously updated by kube-apiservers, instead of just at apiserver start (#82705, @deads2k)

CLI

  • Fixed kubectl endpointslice output for get requests (#82603, @robscott)
  • Gives the right error message when using kubectl delete a wrong resource. (#83825, @zhouya0)
  • If a bad flag is supplied to a kubectl command, only a tip to run --help is printed, instead of the usage menu. Usage menu is printed upon running kubectl command --help. (#82423, @sallyom)
  • Commands like kubectl apply now return errors if schema-invalid annotations are specified, rather than silently dropping the entire annotations section. (#83552, @liggitt)
  • Fixes spurious 0 revisions listed when running kubectl rollout history for a StatefulSet (#82643, @ZP-AlwaysWin)
  • Correct a reference to a not/no longer used kustomize subcommand in the documentation (#82535, @demobox)
  • Kubectl set resources will no longer return an error if passed an empty change for a resource. kubectl set subject will no longer return an error if passed an empty change for a resource. (#85490, @sallyom)
  • Kubectl: --resource-version now works properly in label/annotate/set selector commands when racing with other clients to update the target object (#85285, @liggitt)
  • The --certificate-authority flag now correctly overrides existing skip-TLS or CA data settings in the kubeconfig file (#83547, @liggitt)

Cloud Provider

  • Azure: update disk lock logic per vm during attach/detach to allow concurrent updates for different nodes. (#85115, @aramase)
  • Fix vmss dirty cache issue in disk attach/detach on vmss node (#85158, @andyzhangx)
  • Fix race condition when attach/delete azure disk in same time (#84917, @andyzhangx)
  • Change GCP ILB firewall names to contain the k8s-fw- prefix like the rest of the firewall rules. This is needed for consistency and also for other components to identify the firewall rule as k8s/service-controller managed. (#84622, @prameshj)
  • Ensure health probes are created for local traffic policy UDP services on Azure (#84802, @feiskyer)
  • Openstack: Do not delete managed LB in case of security group reconciliation errors (#82264, @multi-io)
  • Fix aggressive VM calls for Azure VMSS (#83102, @feiskyer)
  • Fix: azure disk detach failure if node not exists (#82640, @andyzhangx)
  • Add azure disk encryption(SSE+CMK) support (#84605, @andyzhangx)
  • Update Azure SDK versions to v35.0.0 (#84543, @andyzhangx)
  • Azure: Add allow unsafe read from cache (#83685, @aramase)
  • Reduces the number of calls made to the Azure API when requesting the instance view of a virtual machine scale set node. (#82496, @hasheddan)
  • Added cloud operation count metrics to azure cloud controller manager. (#82574, @kkmsft)
  • On AWS nodes with multiple network interfaces, kubelet should now more reliably report the same primary node IP. (#80747, @danwinship)
  • Update Azure load balancer to prevent orphaned public IP addresses (#82890, @chewong)

Cluster Lifecycle

  • Kubeadm alpha certs command now skip missing files (#85092, @fabriziopandini)

  • Kubeadm: the command "kubeadm token create" now has a "--certificate-key" flag that can be used for the formation of join commands for control-planes with automatic copy of certificates (#84591, @TheLastProject)

  • Kubeadm: Fix a bug where kubeadm cannot parse kubelet's version if the latter dumps logs on the standard error. (#85351, @rosti)

  • Kubeadm: added retry to all the calls to the etcd API so kubeadm will be more resilient to network glitches (#85201, @fabriziopandini)

  • Fixes a bug in kubeadm that caused init and join to hang indefinitely in specific conditions. (#85156, @chuckha)

  • Kubeadm now includes CoreDNS version 1.6.5

    • kubernetes plugin adds metrics to measure kubernetes control plane latency.
    • the health plugin now includes the lameduck option by default, which waits for a duration before shutting down. (#85109, @rajansandeep)

  • Fixed bug when using kubeadm alpha certs commands with clusters using external etcd (#85091, @fabriziopandini)

  • Kubeadm no longer defaults or validates the component configs of the kubelet or kube-proxy (#79223, @rosti)

  • Kubeadm: remove the deprecated --cri-socket flag for kubeadm upgrade apply. The flag has been deprecated since v1.14. (#85044, @neolit123)

  • Kubeadm: prevent potential hanging of commands such as "kubeadm reset" if the apiserver endpoint is not reachable. (#84648, @neolit123)

  • Kubeadm: fix skipped etcd upgrade on secondary control-plane nodes when the command kubeadm upgrade node is used. (#85024, @neolit123)

  • Kubeadm: fix an issue with the kube-proxy container env. variables (#84888, @neolit123)

  • Utilize diagnostics tool to dump GKE windows test logs (#83517, @YangLu1031)

  • Kubeadm: always mount the kube-controller-manager hostPath volume that is given by the --flex-volume-plugin-dir flag. (#84468, @neolit123)

  • Update Cluster Autoscaler version to 1.16.2 (CA release docs: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.16.2) (#84038, @losipiuk)

  • Kubeadm no longer removes /etc/cni/net.d as it does not install it. Users should remove files from it manually or rely on the component that created them (#83950, @yastij)

  • Kubeadm: fix wrong default value for the upgrade node --certificate-renewal flag. (#83528, @neolit123)

  • Bump metrics-server to v0.3.5 (#83015, @olagacek)

  • Dashboard: disable the dashboard Deployment on non-Linux nodes. This step is required to support Windows worker nodes. (#82975, @wawa0210)

  • Fixes a panic in kube-controller-manager cleaning up bootstrap tokens (#82887, @tedyu)

  • Kubeadm: add a new kubelet-finalize phase as part of the init workflow and an experimental sub-phase to enable automatic kubelet client certificate rotation on primary control-plane nodes.

    Prior to 1.17 and for existing nodes created by kubeadm init where kubelet client certificate rotation is desired, you must modify "/etc/kubernetes/kubelet.conf" to point to the PEM symlink for rotation: client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem and client-key: /var/lib/kubelet/pki/kubelet-client-current.pem, replacing the embedded client certificate and key. (#84118, @neolit123)

  • Kubeadm: add a upgrade health check that deploys a Job (#81319, @neolit123)

  • Kubeadm now supports automatic calculations of dual-stack node cidr masks to kube-controller-manager. (#85609, @Arvinderpal)

  • Kubeadm: reset raises warnings if it cannot delete folders (#85265, @SataQiu)

  • Kubeadm: enable the usage of the secure kube-scheduler and kube-controller-manager ports for health checks. For kube-scheduler was 10251, becomes 10259. For kube-controller-manager was 10252, becomes 10257. (#85043, @neolit123)

  • A new kubelet command line option, --reserved-cpus, is introduced to explicitly define the CPU list that will be reserved for system. For example, if --reserved-cpus=0,1,2,3 is specified, then cpu 0,1,2,3 will be reserved for the system. On a system with 24 CPUs, the user may specify isolcpus=4-23 for the kernel option and use CPU 4-23 for the user containers. (#83592, @jianzzha)

  • Kubelet: a configuration file specified via --config is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. (#83204, @obitech)

  • Kubeadm now propagates proxy environment variables to kube-proxy (#84559, @yastij)

  • Update the latest validated version of Docker to 19.03 (#84476, @neolit123)

  • Update to Ingress-GCE v1.6.1 (#84018, @rramkumar1)

  • Kubeadm: enhance certs check-expiration to show the expiration info of related CAs (#83932, @SataQiu)

  • Kubeadm: implemented structured output of 'kubeadm token list' in JSON, YAML, Go template and JsonPath formats (#78764, @bart0sh)

  • Kubeadm: add support for 127.0.0.1 as advertise address. kubeadm will automatically replace this value with matching global unicast IP address on the loopback interface. (#83475, @fabriziopandini)

  • Kube-scheduler: a configuration file specified via --config is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. (#83030, @obitech)

  • Kubeadm: use the --service-cluster-ip-range flag to init or use the ServiceSubnet field in the kubeadm config to pass a comma separated list of Service CIDRs. (#82473, @Arvinderpal)

  • Update crictl to v1.16.1. (#82856, @Random-Liu)

  • Bump addon-resizer to 1.8.7 to fix issues with using deprecated extensions APIs (#85864, @liggitt)

  • Simple script based hyperkube image that bundles all the necessary binaries. This is an equivalent replacement for the image based on the go based hyperkube command + image. (#84662, @dims)

  • Hyperkube will now be available in a new Github repository and will not be included in the kubernetes release from 1.17 onwards (#83454, @dims)

  • Remove prometheus cluster monitoring addon from kube-up (#83442, @serathius)

  • SourcesReady provides the readiness of kubelet configuration sources such as apiserver update readiness. (#81344, @zouyee)

  • This PR sets the --cluster-dns flag value to kube-dns service IP whether or not NodeLocal DNSCache is enabled. NodeLocal DNSCache will listen on both the link-local as well as the service IP. (#84383, @prameshj)

  • kube-dns add-on:

    • All containers are now being executed under more restrictive privileges.
    • Most of the containers now run as non-root user and has the root filesystem set as read-only.
    • The remaining container running as root only has the minimum Linux capabilities it requires to run.
    • Privilege escalation has been disabled for all containers. (#82347, @pjbgf)

  • Kubernetes no longer monitors firewalld. On systems using firewalld for firewall maintenance, kube-proxy will take slightly longer to recover from disruptive firewalld operations that delete kube-proxy's iptables rules.

    As a side effect of these changes, kube-proxy's sync_proxy_rules_last_timestamp_seconds metric no longer behaves the way it used to; now it will only change when services or endpoints actually change, rather than reliably updating every 60 seconds (or whatever). If you are trying to monitor for whether iptables updates are failing, the sync_proxy_rules_iptables_restore_failures_total metric may be more useful. (#81517, @danwinship)

Instrumentation

Network

  • The official kube-proxy image (used by kubeadm, among other things) is now compatible with systems running iptables 1.8 in "nft" mode, and will autodetect which mode it should use. (#82966, @danwinship)

  • Kubenet: added HostPort IPv6 support. HostPortManager: operates only with one IP family, failing if receives port mapping entries with different IP families. HostPortSyncer: operates only with one IP family, skipping portmap entries with different IP families (#80854, @aojea)

  • Kube-proxy now supports DualStack feature with EndpointSlices and IPVS. (#85246, @robscott)

  • Remove redundant API validation when using Service Topology with externalTrafficPolicy=Local (#85346, @andrewsykim)

  • Update github.com/vishvananda/netlink to v1.0.0 (#83576, @andrewsykim)

  • -- kube-controller-manager --node-cidr-mask-size-ipv4 int32 Default: 24. Mask size for IPv4 node-cidr in dual-stack cluster. --node-cidr-mask-size-ipv6 int32 Default: 64. Mask size for IPv6 node-cidr in dual-stack cluster.

    These 2 flags can be used only for dual-stack clusters. For non dual-stack clusters, continue to use --node-cidr-mask-size flag to configure the mask size.

    The default node cidr mask size for IPv6 was 24 which is now changed to 64. (#79993, @aramase)

  • deprecate cleanup-ipvs flag (#83832, @gongguan)

  • Kube-proxy: emits a warning when a malformed component config file is used with v1alpha1. (#84143, @phenixblue)

  • Set config.BindAddress to IPv4 address 127.0.0.1 if not specified (#83822, @zouyee)

  • Updated kube-proxy ipvs README with correct grep argument to list loaded ipvs modules (#83677, @pete911)

  • The userspace mode of kube-proxy no longer confusingly logs messages about deleting endpoints that it is actually adding. (#83644, @danwinship)

  • Kube-proxy iptables probabilities are now more granular and will result in better distribution beyond 319 endpoints. (#83599, @robscott)

  • Significant kube-proxy performance improvements for non UDP ports. (#83208, @robscott)

  • Improved performance of kube-proxy with EndpointSlice enabled with more efficient sorting. (#83035, @robscott)

  • EndpointSlices are now beta for better Network Endpoint performance at scale. (#84390, @robscott)

  • Updated EndpointSlices to use PublishNotReadyAddresses from Services. (#84573, @robscott)

  • When upgrading to 1.17 with a cluster with EndpointSlices enabled, the endpointslice.kubernetes.io/managed-by label needs to be set on each EndpointSlice. (#85359, @robscott)

  • Adds FQDN addressType support for EndpointSlice. (#84091, @robscott)

  • Fix incorrect network policy description suggesting that pods are isolated when a network policy has no rules of a given type (#84194, @jackkleeman)

  • Fix bug where EndpointSlice controller would attempt to modify shared objects. (#85368, @robscott)

  • Splitting IP address type into IPv4 and IPv6 for EndpointSlices (#84971, @robscott)

  • Added appProtocol field to EndpointSlice Port (#83815, @howardjohn)

  • The docker container runtime now enforces a 220 second timeout on container network operations. (#71653, @liucimin)

  • Fix panic in kubelet when running IPv4/IPv6 dual-stack mode with a CNI plugin (#82508, @aanm)

  • EndpointSlice hostname is now set in the same conditions Endpoints hostname is. (#84207, @robscott)

  • Improving the performance of Endpoint and EndpointSlice controllers by caching Service Selectors (#84280, @gongguan)

  • Significant kube-proxy performance improvements when using Endpoint Slices at scale. (#83206, @robscott)

Node

  • Mirror pods now include an ownerReference for the node that created them. (#84485, @tallclair)

  • Fixed a bug in the single-numa-policy of the TopologyManager. Previously, best-effort pods would result in a terminated state with a TopologyAffinity error. Now they will run as expected. (#83777, @lmdaly)

  • Fixed a bug in the single-numa-node policy of the TopologyManager. Previously, pods that only requested CPU resources and did not request any third-party devices would fail to launch with a TopologyAffinity error. Now they will launch successfully. (#83697, @klueska)

  • Fix error where metrics related to dynamic kubelet config isn't registered (#83184, @odinuge)

  • If container fails because ContainerCannotRun, do not utilize the FallbackToLogsOnError TerminationMessagePolicy, as it masks more useful logs. (#81280, @yqwang-ms)

  • Use online nodes instead of possible nodes when discovering available NUMA nodes (#83196, @zouyee)

  • Use IPv4 in wincat port forward. (#83036, @liyanhui1228)

  • Single static pod files and pod files from http endpoints cannot be larger than 10 MB. HTTP probe payloads are now truncated to 10KB. (#82669, @rphillips)

  • Limit the body length of exec readiness/liveness probes. remote CRIs and Docker shim read a max of 16MB output of which the exec probe itself inspects 10kb. (#82514, @dims)

  • Kubelet: Added kubelet serving certificate metric server_rotation_seconds which is a histogram reporting the age of a just rotated serving certificate in seconds. (#84534, @sambdavidson)

  • Reduce default NodeStatusReportFrequency to 5 minutes. With this change, periodic node status updates will be send every 5m if node status doesn't change (otherwise they are still send with 10s).

    Bump NodeProblemDetector version to v0.8.0 to reduce forced NodeStatus updates frequency to 5 minutes. (#84007, @wojtek-t)

  • The topology manager aligns resources for pods of all QoS classes with respect to NUMA locality, not just Guaranteed QoS pods. (#83492, @ConnorDoyle)

  • Fix a bug that a node Lease object may have been created without OwnerReference. (#84998, @wojtek-t)

  • External facing APIs in plugin registration and device plugin packages are now available under k8s.io/kubelet/pkg/apis/ (#83551, @dims)

Release

  • Added the crictl Windows binaries as well as the Linux 32bit binary to the release archives (#83944, @saschagrunert)
  • Bumps the minimum version of Go required for building Kubernetes to 1.12.4. (#83596, @jktomer)
  • The deprecated mondo kubernetes-test tarball is no longer built. Users running Kubernetes e2e tests should use the kubernetes-test-portable and kubernetes-test-{OS}-{ARCH} tarballs instead. (#83093, @ixdy)

Scheduling

  • Only validate duplication of the RequestedToCapacityRatio custom priority and allow other custom predicates/priorities (#84646, @liu-cong)
  • Scheduler policy configs can no longer be declared multiple times (#83963, @damemi)
  • TaintNodesByCondition was graduated to GA, CheckNodeMemoryPressure, CheckNodePIDPressure, CheckNodeDiskPressure, CheckNodeCondition were accidentally removed since 1.12, the replacement is to use CheckNodeUnschedulablePred (#84152, @draveness)
  • [migration phase 1] PodFitsHostPorts as filter plugin (#83659, @wgliang)
  • [migration phase 1] PodFitsResources as framework plugin (#83650, @wgliang)
  • [migration phase 1] PodMatchNodeSelector/NodAffinity as filter plugin (#83660, @wgliang)
  • Add more tracing steps in generic_scheduler (#83539, @wgliang)
  • [migration phase 1] PodFitsHost as filter plugin (#83662, @wgliang)
  • Fixed a scheduler panic when using PodAffinity. (#82841, @Huang-Wei)
  • Take the context as the first argument of Schedule. (#82119, @wgliang)
  • Fixed an issue that the correct PluginConfig.Args is not passed to the corresponding PluginFactory in kube-scheduler when multiple PluginConfig items are defined. (#82483, @everpeace)
  • Profiling is enabled by default in the scheduler (#84835, @denkensk)
  • Scheduler now reports metrics on cache size including nodes, pods, and assumed pods (#83508, @damemi)
  • User can now use component config to configure NodeLabel plugin for the scheduler framework. (#84297, @liu-cong)
  • Optimize inter-pod affinity preferredDuringSchedulingIgnoredDuringExecution type, up to 4x in some cases. (#84264, @ahg-g)
  • Filter plugin for cloud provider storage predicate (#84148, @gongguan)
  • Refactor scheduler's framework permit API. (#83756, @hex108)
  • Add incoming pods metrics to scheduler queue. (#83577, @liu-cong)
  • Allow dynamically set glog logging level of kube-scheduler (#83910, @mrkm4ntr)
  • Add latency and request count metrics for scheduler framework. (#83569, @liu-cong)
  • Expose SharedInformerFactory in the framework handle (#83663, @draveness)
  • Add per-pod scheduling metrics across 1 or more schedule attempts. (#83674, @liu-cong)
  • Add podInitialBackoffDurationSeconds and podMaxBackoffDurationSeconds to the scheduler config API (#81263, @draveness)
  • Expose kubernetes client in the scheduling framework handle. (#82432, @draveness)
  • Remove MaxPriority in the scheduler API, please use MaxNodeScore or MaxExtenderPriority instead. (#83386, @draveness)
  • Consolidate ScoreWithNormalizePlugin into the ScorePlugin interface (#83042, @draveness)
  • New APIs to allow adding/removing pods from pre-calculated prefilter state in the scheduling framework (#82912, @ahg-g)
  • Added Clone method to the scheduling framework's PluginContext and ContextData. (#82951, @ahg-g)
  • Modified the scheduling framework's Filter API. (#82842, @ahg-g)
  • Critical pods can now be created in namespaces other than kube-system. To limit critical pods to the kube-system namespace, cluster admins should create an admission configuration file limiting critical pods by default, and a matching quota object in the kube-system namespace permitting critical pods in that namespace. See https://kubernetes.io/docs/concepts/policy/resource-quotas/&#35;limit-priority-class-consumption-by-default for details. (#76310, @ravisantoshgudimetla)
  • Scheduler ComponentConfig fields are now pointers (#83619, @damemi)
  • Scheduler Policy API has a new recommended apiVersion apiVersion: kubescheduler.config.k8s.io/v1 which is consistent with the scheduler API group kubescheduler.config.k8s.io. It holds the same API as the old apiVersion apiVersion: v1. (#83578, @Huang-Wei)
  • Rename PluginContext to CycleState in the scheduling framework (#83430, @draveness)
  • Some scheduler extender API fields are moved from pkg/scheduler/api to pkg/scheduler/apis/extender/v1. (#83262, @Huang-Wei)
  • Kube-scheduler: emits a warning when a malformed component config file is used with v1alpha1. (#84129, @obitech)
  • Kube-scheduler now falls back to emitting events using core/v1 Events when events.k8s.io/v1beta1 is disabled. (#83692, @yastij)
  • Expand scheduler priority functions and scheduling framework plugins' node score range to [0, 100]. Note: this change is internal and does not affect extender and RequestedToCapacityRatio custom priority, which are still expected to provide a [0, 10] range. (#83522, @draveness)

Storage

  • Bump CSI version to 1.2.0 (#84832, @gnufied)

  • CSI Migration: Fixes issue where all volumes with the same inline volume inner spec name were staged in the same path. Migrated inline volumes are now staged at a unique path per unique volume. (#84754, @davidz627)

  • CSI Migration: GCE PD access mode now reflects read only status of inline volumes - this allows multi-attach for read only many PDs (#84809, @davidz627)

  • CSI detach timeout increased from 10 seconds to 2 minutes (#84321, @cduchesne)

  • Ceph RBD volume plugin now does not use any keyring (/etc/ceph/ceph.client.lvs01cinder.keyring, /etc/ceph/ceph.keyring, /etc/ceph/keyring, /etc/ceph/keyring.bin) for authentication. Ceph user credentials must be provided in PersistentVolume objects and referred Secrets. (#75588, @smileusd)

  • Validate Gluster IP (#83104, @zouyee)

  • PersistentVolumeLabel admission plugin, responsible for labeling PersistentVolumes with topology labels, now does not overwrite existing labels on PVs that were dynamically provisioned. It trusts the dynamic provisioning that it provided the correct labels to the PersistentVolume, saving one potentially expensive cloud API call. PersistentVolumes created manually by users are labelled by the admission plugin in the same way as before. (#82830, @jsafrane)

  • Existing PVs are converted to use volume topology if migration is enabled. (#83394, @bertinatto)

  • local: support local filesystem volume with block resource reconstruction (#84218, @cofyc)

  • Fixed binding of block PersistentVolumes / PersistentVolumeClaims when BlockVolume feature is off. (#84049, @jsafrane)

  • Report non-confusing error for negative storage size in PVC spec. (#82759, @sttts)

  • Fixed "requested device X but found Y" attach error on AWS. (#85675, @jsafrane)

  • Reduced frequency of DescribeVolumes calls of AWS API when attaching/detaching a volume. (#84181, @jsafrane)

  • Fixed attachment of AWS volumes that have just been detached. (#83567, @jsafrane)

  • Fix possible fd leak and closing of dirs when using openstack (#82873, @odinuge)

  • local: support local volume block mode reconstruction (#84173, @cofyc)

  • Fixed cleanup of raw block devices after kubelet restart. (#83451, @jsafrane)

  • Add data cache flushing during unmount device for GCE-PD driver in Windows Server. (#83591, @jingxu97)

Windows

  • Adds Windows Server build information as a label on the node. (#84472, @gab-satchi)
  • Fixes kube-proxy bug accessing self nodeip:port on windows (#83027, @liggitt)
  • When using Containerd on Windows, the TerminationMessagePath file will now be mounted in the Windows Pod. (#83057, @bclau)
  • Fix kubelet metrics gathering on non-English Windows hosts (#84156, @wawa0210)

Dependencies

  • Update etcd client side to v3.4.3 (#83987, @wenjiaswe)
  • Kubernetes now requires go1.13.4+ to build (#82809, @liggitt)
  • Update to use go1.12.12 (#84064, @cblecker)
  • Update to go 1.12.10 (#83139, @cblecker)
  • Update default etcd server version to 3.4.3 (#84329, @jingyih)
  • Upgrade default etcd server version to 3.3.17 (#83804, @jpbetz)
  • Upgrade to etcd client 3.3.17 to fix bug where etcd client does not parse IPv6 addresses correctly when members are joining, and to fix bug where failover on multi-member etcd cluster fails certificate check on DNS mismatch (#83801, @jpbetz)

Detailed go Dependency Changes

Added

  • github.com/OpenPeeDeeP/depguard: v1.0.1
  • github.com/StackExchange/wmi: 5d04971
  • github.com/agnivade/levenshtein: v1.0.1
  • github.com/alecthomas/template: a0175ee
  • github.com/alecthomas/units: 2efee85
  • github.com/andreyvit/diff: c7f18ee
  • github.com/anmitsu/go-shlex: 648efa6
  • github.com/bazelbuild/rules_go: 6dae44d
  • github.com/bgentry/speakeasy: v0.1.0
  • github.com/bradfitz/go-smtpd: deb6d62
  • github.com/cockroachdb/datadriven: 80d97fb
  • github.com/creack/pty: v1.1.7
  • github.com/gliderlabs/ssh: v0.1.1
  • github.com/go-critic/go-critic: 1df3008
  • github.com/go-kit/kit: v0.8.0
  • github.com/go-lintpack/lintpack: v0.5.2
  • github.com/go-logfmt/logfmt: v0.3.0
  • github.com/go-ole/go-ole: v1.2.1
  • github.com/go-stack/stack: v1.8.0
  • github.com/go-toolsmith/astcast: v1.0.0
  • github.com/go-toolsmith/astcopy: v1.0.0
  • github.com/go-toolsmith/astequal: v1.0.0
  • github.com/go-toolsmith/astfmt: v1.0.0
  • github.com/go-toolsmith/astinfo: 9809ff7
  • github.com/go-toolsmith/astp: v1.0.0
  • github.com/go-toolsmith/pkgload: v1.0.0
  • github.com/go-toolsmith/strparse: v1.0.0
  • github.com/go-toolsmith/typep: v1.0.0
  • github.com/gobwas/glob: v0.2.3
  • github.com/golangci/check: cfe4005
  • github.com/golangci/dupl: 3e9179a
  • github.com/golangci/errcheck: ef45e06
  • github.com/golangci/go-misc: 927a3d8
  • github.com/golangci/go-tools: e32c541
  • github.com/golangci/goconst: 041c5f2
  • github.com/golangci/gocyclo: 2becd97
  • github.com/golangci/gofmt: 0b8337e
  • github.com/golangci/golangci-lint: v1.18.0
  • github.com/golangci/gosec: 66fb7fc
  • github.com/golangci/ineffassign: 42439a7
  • github.com/golangci/lint-1: ee948d0
  • github.com/golangci/maligned: b1d8939
  • github.com/golangci/misspell: 950f5d1
  • github.com/golangci/prealloc: 215b22d
  • github.com/golangci/revgrep: d9c87f5
  • github.com/golangci/unconvert: 28b1c44
  • github.com/google/go-github: v17.0.0+incompatible
  • github.com/google/go-querystring: v1.0.0
  • github.com/gostaticanalysis/analysisutil: v0.0.3
  • github.com/jellevandenhooff/dkim: f50fe3d
  • github.com/julienschmidt/httprouter: v1.2.0
  • github.com/klauspost/compress: v1.4.1
  • github.com/kr/logfmt: b84e30a
  • github.com/logrusorgru/aurora: a7b3b31
  • github.com/mattn/go-runewidth: v0.0.2
  • github.com/mattn/goveralls: v0.0.2
  • github.com/mitchellh/go-ps: 4fdf99a
  • github.com/mozilla/tls-observatory: 8791a20
  • github.com/mwitkow/go-conntrack: cc309e4
  • github.com/nbutton23/zxcvbn-go: eafdab6
  • github.com/olekukonko/tablewriter: a0225b3
  • github.com/quasilyte/go-consistent: c6f3937
  • github.com/rogpeppe/fastuuid: 6724a57
  • github.com/ryanuber/go-glob: 256dc44
  • github.com/sergi/go-diff: v1.0.0
  • github.com/shirou/gopsutil: c95755e
  • github.com/shirou/w32: bb4de01
  • github.com/shurcooL/go-goon: 37c2f52
  • github.com/shurcooL/go: 9e1955d
  • github.com/sourcegraph/go-diff: v0.5.1
  • github.com/tarm/serial: 98f6abe
  • github.com/tidwall/pretty: v1.0.0
  • github.com/timakin/bodyclose: 87058b9
  • github.com/ultraware/funlen: v0.0.2
  • github.com/urfave/cli: v1.20.0
  • github.com/valyala/bytebufferpool: v1.0.0
  • github.com/valyala/fasthttp: v1.2.0
  • github.com/valyala/quicktemplate: v1.1.1
  • github.com/valyala/tcplisten: ceec8f9
  • github.com/vektah/gqlparser: v1.1.2
  • go.etcd.io/etcd: 3cf2f69
  • go.mongodb.org/mongo-driver: v1.1.2
  • go4.org: 417644f
  • golang.org/x/build: 2835ba2
  • golang.org/x/perf: 6e6d33e
  • golang.org/x/xerrors: a985d34
  • gopkg.in/alecthomas/kingpin.v2: v2.2.6
  • gopkg.in/cheggaaa/pb.v1: v1.0.25
  • gopkg.in/resty.v1: v1.12.0
  • grpc.go4.org: 11d0a25
  • k8s.io/system-validators: v1.0.4
  • mvdan.cc/interfacer: c200402
  • mvdan.cc/lint: adc824a
  • mvdan.cc/unparam: fbb5962
  • sourcegraph.com/sqs/pbtypes: d3ebe8f

Changed

  • github.com/Azure/azure-sdk-for-go: v32.5.0+incompatible → v35.0.0+incompatible
  • github.com/Microsoft/go-winio: v0.4.11 → v0.4.14
  • github.com/bazelbuild/bazel-gazelle: c728ce9 → 70208cb
  • github.com/bazelbuild/buildtools: 80c7f0d → 69366ca
  • github.com/beorn7/perks: 3a771d9 → v1.0.0
  • github.com/container-storage-interface/spec: v1.1.0 → v1.2.0
  • github.com/coredns/corefile-migration: v1.0.2 → v1.0.4
  • github.com/coreos/etcd: v3.3.17+incompatible → v3.3.10+incompatible
  • github.com/coreos/go-systemd: 39ca1b0 → 95778df
  • github.com/docker/go-units: v0.3.3 → v0.4.0
  • github.com/docker/libnetwork: a9cd636 → f0e46a7
  • github.com/fatih/color: v1.6.0 → v1.7.0
  • github.com/ghodss/yaml: c7ce166 → v1.0.0
  • github.com/go-openapi/analysis: v0.19.2 → v0.19.5
  • github.com/go-openapi/jsonpointer: v0.19.2 → v0.19.3
  • github.com/go-openapi/jsonreference: v0.19.2 → v0.19.3
  • github.com/go-openapi/loads: v0.19.2 → v0.19.4
  • github.com/go-openapi/runtime: v0.19.0 → v0.19.4
  • github.com/go-openapi/spec: v0.19.2 → v0.19.3
  • github.com/go-openapi/strfmt: v0.19.0 → v0.19.3
  • github.com/go-openapi/swag: v0.19.2 → v0.19.5
  • github.com/go-openapi/validate: v0.19.2 → v0.19.5
  • github.com/godbus/dbus: v4.1.0+incompatible → 2ff6f7f
  • github.com/golang/protobuf: v1.3.1 → v1.3.2
  • github.com/google/btree: 4030bb1 → v1.0.0
  • github.com/google/cadvisor: v0.34.0 → v0.35.0
  • github.com/gregjones/httpcache: 787624d → 9cad4c3
  • github.com/grpc-ecosystem/go-grpc-middleware: cfaf568 → f849b54
  • github.com/grpc-ecosystem/grpc-gateway: v1.3.0 → v1.9.5
  • github.com/heketi/heketi: v9.0.0+incompatible → c2e2a4a
  • github.com/json-iterator/go: v1.1.7 → v1.1.8
  • github.com/mailru/easyjson: 94de47d → v0.7.0
  • github.com/mattn/go-isatty: v0.0.3 → v0.0.9
  • github.com/mindprince/gonvml: fee913c → 9ebdce4
  • github.com/mrunalp/fileutils: 4ee1cc9 → 7d4729f
  • github.com/munnerz/goautoneg: a547fc6 → a7dc8b6
  • github.com/onsi/ginkgo: v1.8.0 → v1.10.1
  • github.com/onsi/gomega: v1.5.0 → v1.7.0
  • github.com/opencontainers/runc: 6cc5158 → v1.0.0-rc9
  • github.com/opencontainers/selinux: v1.2.2 → 5215b18
  • github.com/pkg/errors: v0.8.0 → v0.8.1
  • github.com/prometheus/client_golang: v0.9.2 → v1.0.0
  • github.com/prometheus/client_model: 5c3871d → fd36f42
  • github.com/prometheus/common: 4724e92 → v0.4.1
  • github.com/prometheus/procfs: 1dc9a6c → v0.0.2
  • github.com/soheilhy/cmux: v0.1.3 → v0.1.4
  • github.com/spf13/pflag: v1.0.3 → v1.0.5
  • github.com/stretchr/testify: v1.3.0 → v1.4.0
  • github.com/syndtr/gocapability: e7cb7fa → d983527
  • github.com/vishvananda/netlink: b2de5d1 → v1.0.0
  • github.com/vmware/govmomi: v0.20.1 → v0.20.3
  • github.com/xiang90/probing: 07dd2e8 → 43a291a
  • go.uber.org/atomic: 8dc6146 → v1.3.2
  • go.uber.org/multierr: ddea229 → v1.1.0
  • go.uber.org/zap: 67bc79d → v1.10.0
  • golang.org/x/crypto: e84da03 → 60c769a
  • golang.org/x/lint: 8f45f77 → 959b441
  • golang.org/x/net: cdfb69a → 13f9640
  • golang.org/x/oauth2: 9f33145 → 0f29369
  • golang.org/x/sync: 42b3178 → cd5d95a
  • golang.org/x/sys: 3b52091 → fde4db3
  • golang.org/x/text: e6919f6 → v0.3.2
  • golang.org/x/time: f51c127 → 9d24e82
  • golang.org/x/tools: 6e04913 → 65e3620
  • google.golang.org/grpc: v1.23.0 → v1.23.1
  • gopkg.in/inf.v0: v0.9.0 → v0.9.1
  • k8s.io/klog: v0.4.0 → v1.0.0
  • k8s.io/kube-openapi: 743ec37 → 30be4d1
  • k8s.io/repo-infra: 00fe14e → v0.0.1-alpha.1
  • k8s.io/utils: 581e001 → e782cd3
  • sigs.k8s.io/structured-merge-diff: 6149e45 → b1b620d

Removed

  • github.com/cloudflare/cfssl: 56268a6
  • github.com/coreos/bbolt: v1.3.3
  • github.com/coreos/rkt: v1.30.0
  • github.com/globalsign/mgo: eeefdec
  • github.com/google/certificate-transparency-go: v1.0.21
  • github.com/heketi/rest: aa6a652
  • github.com/heketi/utils: 435bc5b
  • github.com/pborman/uuid: v1.2.0

v1.17.0-rc.2

Documentation

Downloads for v1.17.0-rc.2

filename sha512 hash
kubernetes.tar.gz c71521ab0ab1905776b4e05d99672b7ae6555693b95bc4b84c61134197afe4bf9c49297abdfcf87b34d5e8922550d4e45b7e06073881fa5033d39034f3cba402
kubernetes-src.tar.gz 68248a0610e6971db509fa3475032704ed2d37bb5937ee462fff0a7f0b84ee9753ae49fbc66f00eebc6cc5f455b6c41327c50078708c1570c9bf3d1186f5ff6f

Client Binaries

filename sha512 hash
kubernetes-client-darwin-386.tar.gz 1bd00995cc4a58050d42bd4b430cd353f808eade67556946d70e3e8a365d9e05a49c44d611ff6fe97f89f01a2dfa7f297ea66f0edba39333f9a4bcd06991375b
kubernetes-client-darwin-amd64.tar.gz 704727bc0d1ca207ff75f901ffb7b8afd29cdc532455e76bfcf8c0223c605c104f3a588173eef3a3e8b8f976fed34b870d398383454ad201f10ca430d72365ac
kubernetes-client-linux-386.tar.gz 706a08ecd314afbc63a7fdedcc47f17d4ab8bc36a2a7239d1f86e4321a6bc274b740893508558e6ca6492dc690b1a6042fc3a6bd3cddeb7bbb84ba851609c974
kubernetes-client-linux-amd64.tar.gz 36e7041453f735ea19141eebcce48fdc18cd3cae76fa7ac97bc7b46077e9208cad9974479d93450932d338ea162d4153ad0ec6f56f3f2cb8e8d98a132f14f833
kubernetes-client-linux-arm.tar.gz e6f1bfa5170238fc676e3717ee212e96076e8ec3ceca6b9a4bd4233822185ed8d2aee826d4061bbb1638b0996488e400443d88667948d6b2e5290c3647036dca
kubernetes-client-linux-arm64.tar.gz 29bf7b4df9786c9be9995b15f05ebb18bd1dfad9cebf61207c1cac050000cabb41b816c4cd6022710c01fc712624359988aac307bcec12a51e2dae3163ac9406
kubernetes-client-linux-ppc64le.tar.gz d1774fbeafd01447d11e05734055ad133dc90108f5c1bc9adaf84b8334f997e24324f59ee664c7e723b1dfb05e8ec4a59f956148718fdb277be40f9a7886c28e
kubernetes-client-linux-s390x.tar.gz ca93b0539bda64f3e168b1ca1178eaf13f81de297475fc750893678a0cc6c626c7dea69253079c16b3875ec68f8162f8c01469f72e5481c1bb21d3f57e5745a1
kubernetes-client-windows-386.tar.gz cda5e2526779991d3169d0089e69a2b9e7aa2a127aeb7eec339f4ed1b2b74afc74ef8154964bffb4219eed4e0956b4eb4356ed8cea6364085a163737010a8286
kubernetes-client-windows-amd64.tar.gz c95b17ecd976cf33f182c2d26d49102dad2d7d78fe36d389b18730d59a8866cd28d1b3b28a20126c28b86cdb02ec3e58b8a86397cd778a7642160f11fd789e27

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz aac109354621dc061e01aa6c72aabe43bb1c784a986be86f2e53c8f8243a2470d31955b06111648ddf0a9be686c7cda97c1106d46772c5db100061a7d3cb2521
kubernetes-server-linux-arm.tar.gz d9755aa8c2b0c2d2fdd0756422176c7b34a85553edde03f075363f79bada3f349facb19123cab7ac4ed0ff3159d256b1c968037aa25118ab2fb3f67a118fce35
kubernetes-server-linux-arm64.tar.gz e0bf1247872b0361237e7c5f0837b496b2c4ed05e38d5878c11a81d86282ac83ba382e3cf606a14a3a6af73085e137a82d193790e2805c88ea35bdf07c163e2a
kubernetes-server-linux-ppc64le.tar.gz 44d54154e37b87841123d9f12bce979e5faec88fa9654c0f46904f1bf477aff28790b59af241706b2f729cf1e0e56b146512c2c66bb28909c898e9df2f6ea920
kubernetes-server-linux-s390x.tar.gz 9f42dd736b9575eabb4ef37dae8f18e2f433d00d94e7a994734a9b362127de060ee67e02fc6578dd599b0120b58564aa4fdc5bf583e4e7cc825d7875b3ca099a

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 6f94d2f4a9e2b37797c482a9d42ee9c93d6100fb8b21a983944cb611d57e4e0e6c69a8aa6e8200927f81d68f614913c5453400d298ef884d687f9629d0a213ba
kubernetes-node-linux-arm.tar.gz 7078f7372d733b2933d24ddca38401b6d044f90a9c82e12d61c968f545a24a03d738e7402501771cfbe403a47b96a5f8d2e662823e8fa138b2e30804ceb688bf
kubernetes-node-linux-arm64.tar.gz 9d9365699ecfecac6e7413bdf6d77b917a0e4ed5747810db8c04a3a6d5d2ae416067ae0164fb909e6849179a5117b9df26f39d2bdeca73e706f17ddb84ac2f78
kubernetes-node-linux-ppc64le.tar.gz 6103b55f433a864360231cc509bd692b3ebdc6be34e3fe43fdc2fbd1a2bff750cc2800a68792fe53b87803197cba95a90317b404c8ef3cd2ac3be3b0c54c0c34
kubernetes-node-linux-s390x.tar.gz d83008a4cc86c837afd89b3ef7eece8deac67ba29a9a29076333481e630b59acc044917ad54fd1658932569fc2f11f350267ebe9dd2089a865163dfecea55798
kubernetes-node-windows-amd64.tar.gz 50b03637ecaacf3e6acd56d1a1eac7a95bb05697a179cd83494726ae05480839161d62be2572040607ad9f1fe21ffb8d0c68a91f3f3aa2d99d6ecc8cde30204f

Changelog since v1.17.0-rc.1

Other notable changes

v1.17.0-rc.1

Documentation

Downloads for v1.17.0-rc.1

filename sha512 hash
kubernetes.tar.gz f349b451362bf489066a5a0ad29e0eeb4c3c9bedd05c46309dbdac85abab6ae0fcf7b21f36cf25094bae76d388ef937beca4bdf1d2aaf4afffd7b620b856ed8d
kubernetes-src.tar.gz 7bda9be86cf317827b66d553eb876ec24a649e60d558f9e6e66db842fdf21eefd8354e7d816d4a08b42d5b8db1172c98efd732a41d601c31cfca83d18e0b7548

Client Binaries

filename sha512 hash
kubernetes-client-darwin-386.tar.gz 2ada2da6da63ae97dad4a6b5b64326501eed3a19d6f52fdf36b8224a1341142d72b25968cd978414ce5ae432c6cab41372b1b4ef1603b0256055522c580c6a65
kubernetes-client-darwin-amd64.tar.gz ac06923e4c056d5ab97688e1f42ff408eeab0c0e8f3b010630d45f3530696cfcb1352c49b9cc64c723f0e24663b2f5690865e5243158c3eb8887a47872d40082
kubernetes-client-linux-386.tar.gz bb1f4384b6e3aa4cdaf6f629adeb0f81df138f17fc1c5a39c1584c31e228340761d78bf762fa83e69de687c98f2055ecf91a0ac39d82ec2d76ca09111d3bfd56
kubernetes-client-linux-amd64.tar.gz f66119eb66f87f19c993e380813c0a8051e562fb62c1e8a2f49237774fe5d9e132cbaaaf265be812d5fd1bbf8ad1ff5a6dc7cb9f8915d241109765cd9b10ef34
kubernetes-client-linux-arm.tar.gz 5a077f979ea775ba45d741b1137ab8a579164601bd8033704e03646ba1c99322c08ced72fdb12f073b6e92df159474f23e3f44be40a17aa45999940062150418
kubernetes-client-linux-arm64.tar.gz 1b595b0aa568b8de3a4a56d9226e618c3648fb167c5ad62c833578ced95293cf77f1a066012a8f82e38c60cbb38b016665f8a5d151497d9c77a5edaceb541ceb
kubernetes-client-linux-ppc64le.tar.gz 955b1ecf8b04944cc04a06c3023574b7ccfad655df658320402bca15647b8fca65c9c5f4f9482989da4f5740b6f973e312287dda871abf0b17a56fcfbc281b30
kubernetes-client-linux-s390x.tar.gz 273d5ca8b5fa042b68c7f01f8f6b293c308ee2ce5675419350f01f7763824b61a6ebc7a9470b1196c4c87e1481bbc91931d6d41b3eb50bd99fd0ecc06a65b189
kubernetes-client-windows-386.tar.gz 94e20c417c626166cff39a74b05d7eb443a00be9ef7d7f7bb014d170a41ff9b52999bc21bba19b97d27ad5c1a978e761e125a30295338e5db4dbac16d1661b66
kubernetes-client-windows-amd64.tar.gz 59deabc78139ad4d6dced570f5292eaa67da4d6fca88f90d7e1484b77a71ef64826632613ebedb79215dbaba6dfe4b3eda6cf8bbfa3fd0024b9ab290e3f8cd1f

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz b73983175bb95accb505ab953635e49d5ce3ef0a58e4de6575e431a6c0c81819bb8fc75949c5e3d35a395a96aaadad2ef6a777cc86bfa1b70afd02269cae58c0
kubernetes-server-linux-arm.tar.gz 7f2b75c7fe9f97dcab6ef00fa72bda3493225c58aa963e6843184f24c32631d33a05288ec525ef378296702d51c715800bc4394b9906285dc105e0dd984cf95b
kubernetes-server-linux-arm64.tar.gz 113e978c8400acc8048b0a1979cf5cf95cfec76de7a9b2a5e1c204de8969ff7e4662fecf232d03f5889f47e4633197ce5013769b2508350dc0002da9cb004957
kubernetes-server-linux-ppc64le.tar.gz 5dfb184689e9d534788f86dc29bf69e779c5a9927adc50338fee0f7a71603aeccd7822f6ca6ba017e73e595eb4b95c15b26fc2af2783a3b7fe5ac5095555e1be
kubernetes-server-linux-s390x.tar.gz ae7a6399672a7333ba85567bf9b6b1f9af7ad9616acbd0bd52237cb5d5c968f3f39617ffad9606c8486cfa253ea2dfc1ff47f55b96f1065998a03fa8a4a4c735

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 17279e9220a2423aba35056bc631d0b2af1df45297dd40e36949a0ff809d3b7c8cc410808638c266cdd02631b403461a92076ae8e8203398da7cbb1720e0625b
kubernetes-node-linux-arm.tar.gz 4611354f214b2d7d5138adaffd764574dfc6a68d27b7f53563c486a8ba25cefceeb5cf04d75cc1f47f525d450609f90c085ba36d35f166731fc0f51ea350a411
kubernetes-node-linux-arm64.tar.gz 361e44be2d5a98fb94fd8a39ab9a57198bc7613b2004b239920438853db0a33e360bd76c7e2c19e64e74e900c9cb7e912ab90a3d68ddbc560f0cae75b803818d
kubernetes-node-linux-ppc64le.tar.gz 69f9da64ae19bb4cfc616a2abd587c9711e9667da4743d776510e92aa23da72da85bd9aecd2d334066697d5e241abe4050142b8fb1d56b1db7c2037f944cccb6
kubernetes-node-linux-s390x.tar.gz 0fb52cfc24b58887be71d98dd1c826be520eef69448f2f207e849b85533c277084fa4b6e04445c0c1cf499e8fcc63f2088696554465caa954b5643aa9f555c40
kubernetes-node-windows-amd64.tar.gz 7d3ce000317a05737101ff1fd0fd0423be2649e5ae387b9d89e57242a3cf8b202a3152dd6b864601eba07ac319f5787fd9fae86594452a4652e29dd585314c2a

Changelog since v1.17.0-beta.2

Other notable changes

  • kubeadm: fix a panic in case the KubeProxyConfiguration feature gates were not initialized. (#85524, @Arvinderpal)
  • kubeadm: fix stray "node-cidr-mask-size" flag in the kube-controller-manager manifest when IPv6DualStack is enabled (#85494, @tedyu)
  • CRDs can have fields named type with value array and nested array with items fields without validation to fall over this. (#85223, @sttts)
  • Resolves error from v1.17.0-beta.2 with --authorizer-mode webhook complaining about an invalid version (#85441, @liggitt)
  • Promote CSIMigrationAWS to Beta (off by default since it requires installation of the AWS EBS CSI Driver) (#85237, @leakingtapan)
    • The in-tree AWS EBS plugin "kubernetes.io/aws-ebs" is now deprecated and will be removed in 1.21. Users should enable CSIMigration + CSIMigrationAWS features and install the AWS EBS CSI Driver (https://github.com/kubernetes-sigs/aws-ebs-csi-driver) to avoid disruption to existing Pod and PVC objects at that time.
    • Users should start using the AWS EBS CSI CSI Driver directly for any new volumes.

v1.17.0-beta.2

Documentation

Downloads for v1.17.0-beta.2

filename sha512 hash
kubernetes.tar.gz c4e937e784b26b5b18cac0bc8d4c91e1ae576107f14bb475e2d38687fbb5790f2c57898590a2f24d3c4ab4c6060a628e1acb2f15932b70183e5753f751237f60
kubernetes-src.tar.gz 79351b61539c7dc608f4c2a184e788f74503cd801304204de1d52e9ea7b50450503d46d48605b71240395173bcbf1a4727bad3a3dc800766ce4f1f103ca9f2ae

Client Binaries

filename sha512 hash
kubernetes-client-darwin-386.tar.gz 3940ed06c0848b6ddf9e37e66535085d58a5cf7b66a015eab718eb7b4927e9ebce9e0634040bd7a748610b1d881ad9f6e925650d959eaa37e304baa9bb21b6a9
kubernetes-client-darwin-amd64.tar.gz 2d6497ad8f5ca592717fcc704f581020922e317e60c2f7def6b6899398666c6c1c81b0b006ed02c923f54b8f0525dba85aadc5ce62926e9feefe18640c7f2fde
kubernetes-client-linux-386.tar.gz 220713aea7709facd2317015467ce1922abf39cdf486d44f4a3fae497aa119f5af00bbfaa46fb022e1b53de285d2366eba1847a98804af4891eec50306be23fb
kubernetes-client-linux-amd64.tar.gz afe7fbecb04bba24f6c2d794a7c9b83cdc48032137776e660537541e3b2da6a04a1f0b8dc2ac0826a7d3c3c6fb5f609086d6ffae411f3069737448136d78ea65
kubernetes-client-linux-arm.tar.gz 97addcebe381cfd6ccca94ca4f039ec6e300bda701fc005b1d292a055a2ed8515a80991d0013c3d388e742bb6fcf12f733a100eb1a7cd7e02e122c54bc715f4e
kubernetes-client-linux-arm64.tar.gz ca3880ce4c6ca1aa8500d67d0c0eeb85f0323306308d2abe26caa9f97b20432f25e00d25c52b460dbf0f62a65907b3201c51cedca30146dc373ea30331531fc2
kubernetes-client-linux-ppc64le.tar.gz 84c40110c8ae3bfc02edd3f6e937032b41dd67b9c48e738ff8590d9b26d249d4febcb6a5665261a3c835ce0df255fc1aeeeb0df7abd1a158d9cb5eafedbd66f2
kubernetes-client-linux-s390x.tar.gz 31521ef3a8426676e73011dbe5b00a7cb9479aa9d1147e824c1c4287634f1c0742c7990166e804ed7897c0629fd1eebedc6b8fc41788e1b145e26dd72bd8025d
kubernetes-client-windows-386.tar.gz 8601fb516a4557b7579dc3fb3e83e1be2f8e8a6a19aaca5232a5be25d9d23056a37fb5b30742454d8db6598539130c4b9bf6b9eb1b9ec6bdd73ca7c34953c23a
kubernetes-client-windows-amd64.tar.gz 00b4e163629f415c9c9caca4e5a9b0753ecddc29fde7d63b379d9d13af8992f78c2b1d4810a94264edc766bc41c2334cd81fe9e53d95a7bbfcf3795813e0327d

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz a5d21dfe1c05ca6fb1357975b75ff9549642f37e8754a884567c0a00048208ba9060d169b378f0be349197b3c55c41179c70a12aa0249fd149271198ebf1c9ba
kubernetes-server-linux-arm.tar.gz d33a18da3aa4305183f7e3cd3f43f100ed1484c59811c0e8172c5838b94cd0ea11dbc3733d49624e4787e6d12fb5e01b278fc7c328362fadece8cf6b9e91a9bb
kubernetes-server-linux-arm64.tar.gz 2bba562b5e5f28d1b840490cbcc837f421e30a715daca08234b1302f5b7a528b605594bffac7fa104ca2a023b569a3c38156aa3da3576db28f1bdccd37b274f5
kubernetes-server-linux-ppc64le.tar.gz a80a80233832aca887b90e688a5ab537468071d8d3237a28cf3e9d7cf4ef1f340ef243f173e67b74fc08f7a723660237dd1721872ca734bb7bcea87d3ecf0a34
kubernetes-server-linux-s390x.tar.gz ba23bc2a9c94dd19a8a0a739e15e72dcb30ec103978686e1b4a845175ae8cad66e34b266afcb5ba0adb5969bbc6e71bf4d5ef5664b703ddc2628907e211f3b86

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 76298a3e8a6184c7ca7026a83c016172e22f5be18e68d3ec01a3d6fc1b3ca2525232ef6f15b423c651c36694a9cad4fd0af08a93a187a7f42443aa0caa82baf7
kubernetes-node-linux-arm.tar.gz d21b60164cb3ee15a493567e1852fb8844f8a5fa5d6d4e71f8078a039d9fea4af00f992026bbc135fadcf0597da28bd0d8813b231ee05e5c31b17ecd224294b9
kubernetes-node-linux-arm64.tar.gz 9707c6e9c3835f8a5a80be9600f651c992ae717a8a6efc20cc8e67d562ddc62ec81b70deb74ba75a67eb1741cf2afd57312f1ccf6dc97ead8d7139651b20c09b
kubernetes-node-linux-ppc64le.tar.gz 4c8dea040321fbf8e444ba0252567cb3aed3173db4c82b7572b9f8053b1275b9bc45627510a9abe19c8be3d921a1b018b5b869f415762b2569325425ab9aa819
kubernetes-node-linux-s390x.tar.gz 7a720070b28ab5b83ec054a3137d434c39f7b8f1a0c751f5d06b1f2bbe00777ff444f2855c135ac113f0c5193680b27d80369761ff79f7ef22a9dc997037efe1
kubernetes-node-windows-amd64.tar.gz 85b3a0bcb5a319f443cc0f3a9aff0b6fb038d42f515698d51cba27a4a1a6b3d701085c6cfb525cf0f826af3fbdb26abc2ee00fcb8e5022ecd63b8f7697aace88

Changelog since v1.17.0-beta.1

Action Required

  • Renamed FeatureGate RequestManagement to APIPriorityAndFairness. This feature gate is an alpha and has not yet been associated with any actual functionality. (#85260, @MikeSpreitzer)
    • Action required: change references to feature gate RequestManagement into references to APIPriorityAndFairness
  • ACTION REQUIRED: kubeadm: add a new "kubelet-finalize" phase as part of the "init" workflow and an experimental sub-phase to enable automatic kubelet client certificate rotation on primary control-plane nodes. (#84118, @neolit123)
    • Prior to 1.17 and for existing nodes created by "kubeadm init" where kubelet client certificate rotation is desired, you must modify "/etc/kubernetes/kubelet.conf" to point to the PEM symlink for rotation:
    • "client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem" and "client-key: /var/lib/kubelet/pki/kubelet-client-current.pem", replacing the embedded client certificate and key.
  • action required: kubeadm deprecates the use of the hyperkube image (#85094, @rosti)

Other notable changes

  • Following metrics have been turned off: (#83837, @RainbowMango)
      • apiserver_request_count
      • apiserver_request_latencies
      • apiserver_request_latencies_summary
      • apiserver_dropped_requests
      • etcd_request_latencies_summary
      • apiserver_storage_transformation_latencies_microseconds
      • apiserver_storage_data_key_generation_latencies_microseconds
      • apiserver_storage_transformation_failures_total
  • OpenAPI v3 format in CustomResourceDefinition schemas are now documented. (#85381, @sttts)
  • The official kube-proxy image (used by kubeadm, among other things) is now (#82966, @danwinship)
    • compatible with systems running iptables 1.8 in "nft" mode, and will autodetect
    • which mode it should use.
  • Kubenet: added HostPort IPv6 support (#80854, @aojea)
    • HostPortManager: operates only with one IP family, failing if receives portmapping entries with different IP families
    • HostPortSyncer: operates only with one IP family, skipping portmap entries with different IP families
  • Implement the documented API semantics of list-type and map-type atomic to reject non-atomic sub-types. (#84722, @sttts)
  • kubeadm: Fix a bug where kubeadm cannot parse kubelet's version if the latter dumps logs on the standard error. (#85351, @rosti)
  • EndpointSlices are not enabled by default. Use the EndpointSlice feature gate to enable this feature. (#85365, @robscott)
  • Feature gates CSIMigration to Beta (on by default) and CSIMigrationGCE to Beta (off by default since it requires installation of the GCE PD CSI Driver) (#85231, @davidz627)
    • The in-tree GCE PD plugin "kubernetes.io/gce-pd" is now deprecated and will be removed in 1.21. Users should enable CSIMigration + CSIMigrationGCE features and install the GCE PD CSI Driver (https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver) to avoid disruption to existing Pod and PVC objects at that time.
    • Users should start using the GCE PD CSI CSI Driver directly for any new volumes.
  • kube-controller-manager: Fixes bug setting headless service labels on endpoints (#85361, @liggitt)
  • When upgrading to 1.17 with a cluster with EndpointSlices enabled, the endpointslice.kubernetes.io/managed-by label needs to be set on each EndpointSlice. (#85359, @robscott)
  • Remove redundant API validation when using Service Topology with externalTrafficPolicy=Local (#85346, @andrewsykim)
  • Following metrics have been turned off: (#83838, @RainbowMango)
      • scheduler_scheduling_latency_seconds
      • scheduler_e2e_scheduling_latency_microseconds
      • scheduler_scheduling_algorithm_latency_microseconds
      • scheduler_scheduling_algorithm_predicate_evaluation
      • scheduler_scheduling_algorithm_priority_evaluation
      • scheduler_scheduling_algorithm_preemption_evaluation
      • scheduler_scheduling_binding_latency_microseconds
  • CSI Migration: Fixes issue where all volumes with the same inline volume inner spec name were staged in the same path. Migrated inline volumes are now staged at a unique path per unique volume. (#84754, @davidz627)
  • kube-controller-manager (#79993, @aramase)
    • --node-cidr-mask-size-ipv4 int32 Default: 24. Mask size for IPv4 node-cidr in dual-stack cluster.
    • --node-cidr-mask-size-ipv6 int32 Default: 64. Mask size for IPv6 node-cidr in dual-stack cluster.
    • These 2 flags can be used only for dual-stack clusters. For non dual-stack clusters, continue to use
    • --node-cidr-mask-size flag to configure the mask size.
    • The default node cidr mask size for IPv6 was 24 which is now changed to 64.
  • The following information is available through environment variables: (#83123, @aramase)
    • status.podIPs - the pod's IP addresses
  • update github.com/vishvananda/netlink to v1.0.0 (#83576, @andrewsykim)
  • kubectl: --resource-version now works properly in label/annotate/set selector commands when racing with other clients to update the target object (#85285, @liggitt)
  • --runtime-config now supports an api/beta=false value which disables all built-in REST API versions matching v[0-9]+beta[0-9]+. (#84304, @liggitt)
    • --feature-gates now supports an AllBeta=false value which disables all beta feature gates.
  • kube-proxy now supports DualStack feature with EndpointSlices and IPVS. (#85246, @robscott)
  • Add table convertor to componentstatus. (#85174, @zhouya0)
  • kubeadm: added retry to all the calls to the etcd API so kubeadm will be more resilient to network glitches (#85201, @fabriziopandini)
  • azure: update disk lock logic per vm during attach/detach to allow concurrent updates for different nodes. (#85115, @aramase)
  • Scale custom resource unconditionally if resourceVersion is not provided (#80572, @knight42)
  • Bump CSI version to 1.2.0 (#84832, @gnufied)
  • Adds Windows Server build information as a label on the node. (#84472, @gab-satchi)
  • Deprecated metric kubeproxy_sync_proxy_rules_latency_microseconds has been turned off. (#83839, @RainbowMango)
  • Existing PVs are converted to use volume topology if migration is enabled. (#83394, @bertinatto)
  • Finalizer Protection for Service LoadBalancers is now in GA (enabled by default). This feature ensures the Service resource is not fully deleted until the correlating load balancer resources are deleted. (#85023, @MrHohn)
  • EndpointSlices are now beta and enabled by default for better Network Endpoint performance at scale. (#84390, @robscott)
  • When using Containerd on Windows, the TerminationMessagePath file will now be mounted in the Windows Pod. (#83057, @bclau)
  • apiservers based on k8s.io/apiserver with delegated authn based on cluster authentication will automatically update to new authentication information when the authoritative configmap is updated. (#85004, @deads2k)
  • fix vmss dirty cache issue in disk attach/detach on vmss node (#85158, @andyzhangx)
  • Fixes a bug in kubeadm that caused init and join to hang indefinitely in specific conditions. (#85156, @chuckha)
  • kube-apiserver: Authentication configuration for mutating and validating admission webhooks referenced from an --admission-control-config-file can now be specified with apiVersion: apiserver.config.k8s.io/v1, kind: WebhookAdmissionConfiguration. (#85138, @liggitt)
  • Kubeadm now includes CoreDNS version 1.6.5 (#85109, @rajansandeep)
      • kubernetes plugin adds metrics to measure kubernetes control plane latency.
      • the health plugin now includes the lameduck option by default, which waits for a duration before shutting down.
  • Kubeadm now includes CoreDNS version 1.6.5 (#85108, @rajansandeep)
      • kubernetes plugin adds metrics to measure kubernetes control plane latency.
      • the health plugin now includes the lameduck option by default, which waits for a duration before shutting down.
  • kube-apiserver: The ResourceQuota admission plugin configuration referenced from --admission-control-config-file admission config has been promoted to apiVersion: apiserver.config.k8s.io/v1, kind: ResourceQuotaConfiguration with no schema changes. (#85099, @liggitt)
  • kube-apiserver: The AdmissionConfiguration type accepted by --admission-control-config-file has been promoted to apiserver.config.k8s.io/v1 with no schema changes. (#85098, @liggitt)
  • New flag --show-hidden-metrics-for-version in kube-apiserver can be used to show all hidden metrics that deprecated in the previous minor release. (#84292, @RainbowMango)
  • The ResourceQuotaScopeSelectors feature has graduated to GA. The ResourceQuotaScopeSelectors feature gate is now unconditionally enabled and will be removed in 1.18. (#82690, @draveness)
  • Fixed bug when using kubeadm alpha certs commands with clusters using external etcd (#85091, @fabriziopandini)
  • Fix a bug that a node Lease object may have been created without OwnerReference. (#84998, @wojtek-t)
  • Splitting IP address type into IPv4 and IPv6 for EndpointSlices (#84971, @robscott)
  • Pod process namespace sharing is now Generally Available. The PodShareProcessNamespace feature gate is now deprecated and will be removed in Kubernetes 1.19. (#84356, @verb)
  • Fix incorrect network policy description suggesting that pods are isolated when a network policy has no rules of a given type (#84194, @jackkleeman)
  • add RequiresExactMatch for label.Selector (#85048, @shaloulcy)
  • Deprecated metric rest_client_request_latency_seconds has been turned off. (#83836, @RainbowMango)
  • Removed dependency on kubectl from several storage E2E tests (#84042, @okartau)
  • kubeadm no longer defaults or validates the component configs of the kubelet or kube-proxy (#79223, @rosti)
  • Add plugin_execution_duration_seconds metric for scheduler framework plugins. (#84522, @liu-cong)
  • Moving WindowsRunAsUserName feature to beta (#84882, @marosset)
  • Node-specific volume limits has graduated to GA. (#83568, @bertinatto)
  • kubelet and aggregated API servers now use v1 TokenReview and SubjectAccessReview endpoints to check authentication/authorization. (#84768, @liggitt) * kube-apiserver can now specify --authentication-token-webhook-version=v1 or --authorization-webhook-version=v1 to use v1 TokenReview and SubjectAccessReview API objects when communicating with authentication and authorization webhooks.
  • BREAKING CHANGE: Remove plugin watching of deprecated directory {kubelet_root_dir}/plugins and CSI V0 support in accordance with deprecation announcement in https://v1-13.docs.kubernetes.io/docs/setup/release/notes/ (#84533, @davidz627)
  • Adds a new label to indicate what is managing an EndpointSlice. (#83965, @robscott)
  • Fix a racing issue in client-go UpdateTransportConfig. (#80284, @danielqsj)
  • Enables VolumeSnapshotDataSource feature gate and promotes volume snapshot APIs to beta. (#80058, @xing-yang)
  • Added appProtocol field to EndpointSlice Port (#83815, @howardjohn)
  • kubeadm alpha certs command now skip missing files (#85092, @fabriziopandini)
  • A new flag "progress-report-url" has been added to the test context which allows progress information about the test run to be sent to a webhook. In addition, this information is printed to stdout to aid in users watching the logs. (#84524, @johnSchnake)
  • kubeadm: remove the deprecated "--cri-socket" flag for "kubeadm upgrade apply". The flag has been deprecated since v1.14. (#85044, @neolit123)
  • Clients can request protobuf and json and correctly negotiate with the server for JSON for CRD objects, allowing all client libraries to request protobuf if it is available. If an error occurs negotiating a watch with the server, the error is immediately return by the client Watch() method instead of being sent as an Error event on the watch stream. (#84692, @smarterclayton)
  • Following metrics from kubelet are now marked as with the ALPHA stability level: (#84987, @RainbowMango)
    • node_cpu_usage_seconds_total
    • node_memory_working_set_bytes
    • container_cpu_usage_seconds_total
    • container_memory_working_set_bytes
    • scrape_error
  • Following metrics from kubelet are now marked as with the ALPHA stability level: (#84907, @RainbowMango)
    • kubelet_container_log_filesystem_used_bytes
    • kubelet_volume_stats_capacity_bytes
    • kubelet_volume_stats_available_bytes
    • kubelet_volume_stats_used_bytes
    • kubelet_volume_stats_inodes
    • kubelet_volume_stats_inodes_free
    • kubelet_volume_stats_inodes_used
    • plugin_manager_total_plugins
    • volume_manager_total_volumes
  • kubeadm: enable the usage of the secure kube-scheduler and kube-controller-manager ports for health checks. For kube-scheduler was 10251, becomes 10259. For kube-controller-manager was 10252, becomes 10257. (#85043, @neolit123)
  • kubeadm: prevent potential hanging of commands such as "kubeadm reset" if the apiserver endpoint is not reachable. (#84648, @neolit123)
  • Mirror pods now include an ownerReference for the node that created them. (#84485, @tallclair)
  • kubeadm: fix skipped etcd upgrade on secondary control-plane nodes when the command "kubeadm upgrade node" is used. (#85024, @neolit123)
  • fix race condition when attach/delete azure disk in same time (#84917, @andyzhangx)
  • If given an IPv6 bind-address, kube-apiserver will now advertise an IPv6 endpoint for the kubernetes.default service. (#84727, @danwinship)
  • kubeadm: the command "kubeadm token create" now has a "--certificate-key" flag that can be used for the formation of join commands for control-planes with automatic copy of certificates (#84591, @TheLastProject)
  • Deprecate the instance type beta label ("beta.kubernetes.io/instance-type") in favor of it's GA equivalent: "node.kubernetes.io/instance-type" (#82049, @andrewsykim)
  • kube-apiserver: Fixed a regression accepting patch requests > 1MB (#84963, @liggitt)
  • Promote NodeLease feature to GA. (#84351, @wojtek-t)
    • The feature make Lease object changes an additional healthiness signal from Node. Together with that, we reduce frequency of NodeStatus updates to 5m by default in case of no changes to status itself
  • Following metrics from kube-controller-manager are now marked as with the ALPHA stability level: (#84896, @RainbowMango)
    • storage_count_attachable_volumes_in_use
    • attachdetach_controller_total_volumes
    • pv_collector_bound_pv_count
    • pv_collector_unbound_pv_count
    • pv_collector_bound_pvc_count
    • pv_collector_unbound_pvc_count
  • Deprecate the beta labels for zones ("failure-domain.beta.kubernetes.io/zone") and (#81431, @andrewsykim)
    • regions ("failure-domain.beta.kubernetes.io/region") in favor of their GA equivalents:
    • "topology.kubernetes.io/zone" and "topology.kubernetes.io/region".
    • The beta labels "failure-domain.beta.kubernetes.io/zone" and "failure-domain.beta.kubernetes.io/region" will be removed in v1.21
  • kube-apiserver: fixed a bug that could cause a goroutine leak if the apiserver encountered an encoding error serving a watch to a websocket watcher (#84693, @tedyu)
  • EndpointSlice hostname is now set in the same conditions Endpoints hostname is. (#84207, @robscott)
  • Simple script based hyperkube image that bundles all the necessary binaries. This is a equivalent replacement for the image based on the go based hyperkube command + image. (#84662, @dims)
  • configmaps/extension-apiserver-authentication in kube-system is continuously updated by kube-apiservers, instead of just at apiserver start (#82705, @deads2k)
  • kubeadm: fix an issue with the kube-proxy container env. variables (#84888, @neolit123)
  • Updated EndpointSlices to use PublishNotReadyAddresses from Services. (#84573, @robscott)
  • The example API server has renamed its wardle.k8s.io API group to wardle.example.com (#81670, @liggitt)
  • A new kubelet command line option, --reserved-cpus, is introduced to explicitly define the the CPU list that will be reserved for system. For example, if --reserved-cpus=0,1,2,3 is specified, then cpu 0,1,2,3 will be reserved for the system. On a system with 24 CPUs, the user may specify isolcpus=4-23 for the kernel option and use CPU 4-23 for the user containers. (#83592, @jianzzha)
  • Utilize diagnostics tool to dump GKE windows test logs (#83517, @YangLu1031)
  • Improving the performance of Endpoint and EndpointSlice controllers by caching Service Selectors (#84280, @gongguan)
  • When the go-client reflector relists, the ResourceVersion list option is set to the reflector's latest synced resource version to ensure the reflector does not "go back in time" and reprocess events older than it has already processed. If the the server responds with an HTTP 410 (Gone) status code response, the relist falls back to using resourceVersion="". (#83520, @jpbetz)
  • Kubernetes now requires go1.13.4+ to build (#82809, @liggitt)
  • Ensure health probes are created for local traffic policy UDP services on Azure (#84802, @feiskyer)
  • CRDs defaulting is promoted to GA. Note: the feature gate CustomResourceDefaulting will be removed in 1.18. (#84713, @sttts)
  • Profiling is enabled by default in the scheduler (#84835, @denkensk)
  • CSI Migration: GCE PD access mode now reflects read only status of inline volumes - this allows multi-attach for read only many PDs (#84809, @davidz627)
  • All resources within the rbac.authorization.k8s.io/v1alpha1 and rbac.authorization.k8s.io/v1beta1 API groups are deprecated in favor of rbac.authorization.k8s.io/v1, and will no longer be served in v1.20. (#84758, @liggitt)
  • Scheduler ComponentConfig fields are now pointers (#83619, @damemi)
  • Adding initial EndpointSlice metrics. (#83257, @robscott)

v1.17.0-beta.1

Documentation

Downloads for v1.17.0-beta.1

filename sha512 hash
kubernetes.tar.gz 6d6c61bb4d3372d56b7a429b5b8b5adbfb0aaddd65283d169bb719b8aca7c270db34f4699c4efee364565414770f9870c77a74a958725a8258f4bca271582e4c
kubernetes-src.tar.gz 9878c454c5b482621a7ddeab2ab3290fdafd0cfb3d580b261081ba3943b19b13e54aaa3a80ba68d7cbfa46864e51baedc686ab2a5271da6948493cc7ad730e2c

Client Binaries

filename sha512 hash
kubernetes-client-darwin-386.tar.gz 67c9d4d97db40ee94a5e021642eeea006dfd66f0c50ccd0d833c52d2bc4156fb044bb481b77e235db330e34ef580d42d8f1b366420abbab0b62c1cfe59f168a3
kubernetes-client-darwin-amd64.tar.gz 9a1494c082af52186620d1cd1b02f8a8f4af7e676e2ec217f41cf2915bd6fb1717e2c65e42c84ca842a542526f23edaa5ea378932b37f628611d00b08e9ff102
kubernetes-client-linux-386.tar.gz 9de6254a6a267ea283b6118d9da079f072e73ba377e81a361943f6d42baa5dd1b668b20f6909b697cbb5163930e9e497a08b16aa1d3e13feaf5be37047bcf83e
kubernetes-client-linux-amd64.tar.gz a93d028c3adda047864b36f314752fbe4745bb6ad8f37574cc124eb1453bad07e3790dac4cc230e3ee2d3f6e9fb8c75d16860454acb3a6049400bb46489f7c51
kubernetes-client-linux-arm.tar.gz 5ca7feea1c4a33cf92f0ee79a92daee4876b43c626346dcc701bb7d6b956c0050f2ed6be1f2ba31756bf3b651da354bbad511cc3ce6c6349c12bcde41c8aec87
kubernetes-client-linux-arm64.tar.gz d6a63efa140a1c2cde43252e9f917a02752e90628b51ca28ec7118245cd00da03b83f1bd920d0f1da789b8a0f3c73f41fbc9710c7bddcc24e6ac401966180cc5
kubernetes-client-linux-ppc64le.tar.gz cdb805bc7bae052a0585b88c5e7980bd8bf9f32a840728455c18f4f01e03cda823bede2145772c4338e95c1a9b258bba7b8154714457fdb72114ac482eca122e
kubernetes-client-linux-s390x.tar.gz a9ff545cad6a42dbcdf9f91214c15e2ebee2df20579b7f62ec07397eca792f20ee550841759d9d38c0affdb3071ad4a0a741c8641955eb222a5535dd8fb2d3e4
kubernetes-client-windows-386.tar.gz eb3e4dbbb1dc6829bfa320853b695e61f4daafdff4aaa1f4ffb2e4be4b3f2e0c78f385a8a370811cb379f85d5e48a9b55608747592c771d4cbffd446a586cc6d
kubernetes-client-windows-amd64.tar.gz ad0087ef7a0da961d3f22eab2ddab302be2190df5a2150046f7162dfc5072aa1866449a1aafc1c3db65246c392ec47bda20f2b4e7f750e895106fa9cbe1c80f8

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz 8f4878dfbe7e6abc30516bd801bb5c07873b4a80d8bc560e5b5593e0c1d64be2fa662a5f10dd93c947cfb1cfb7336db995a3f2b5c5cc3b259c929f058f27e222
kubernetes-server-linux-arm.tar.gz 3e5745bffbfb3551b4d4962b7ab6524c9b71f55860a91992dd0495266c56b740061f6b0711882e931ead456b14e4bfc9f08c4115a81553c1cfa2aa1cbd769d52
kubernetes-server-linux-arm64.tar.gz ed00c196e6e229229b6523e7c3a201e00805304ad72c54bf7d0fc456d1791404bacf5120317f9a833b0bfddf70f4318d8ac274e3d94b80de0567dfea136b0b13
kubernetes-server-linux-ppc64le.tar.gz 57b5cc144fc4f3bfa6217e0d5494e4a4367f0c0d3504721d4343ff009f00fef1212150d0f1925fe0710eb335c526720e8a5c6fd54d27739b75cc91a06f27df94
kubernetes-server-linux-s390x.tar.gz 6f6d2b61a11e30199997582487f7e4f967771e0367d7f471043e0b9b373d463c5d7370ac3a8e5bbf4761e98e0ab19564f74aa7bf2c8443c9dd53397836d4db9b

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 0c1b9dbb630a3bc47a835f9bfd9259d464abae8a30b0824a73b20892b013ed60d7e4989f48172b122bcc87e08bdd1af9ca9e790ae768b17e1dff190ae8f11b69
kubernetes-node-linux-arm.tar.gz 191087e26632dcc80991530b79bcda49bd4d0a131689ef48164a0bdb30e0a52ca69aa9a1ad42165707669287b8fd09afd407e556803d15b8c66739359a2b13b5
kubernetes-node-linux-arm64.tar.gz f0f58aa8f9ad0ac0a1ee29d318ebb4d14f0bef4cba1fee9081ee2bfb6b41245108bd849470529668a93dcf8b41e53a319bd80ee0bd46ef02b844a995ffcc68e8
kubernetes-node-linux-ppc64le.tar.gz bf574c4a46731ebc273910176ab67b2455b021972de3aeeb2a2b04af2a1079243728f151b9f06298b84832425ce600e54d8c13eec58598b284b44b21beaf73eb
kubernetes-node-linux-s390x.tar.gz 5a5d2ece704178a630dc228b51229b8598eb45bf3eaeae75b1475f249922c6a10b93220fd0f3f29d279ee0ceb34e8537a5b197b9454c4171adcc81facc80c3b6
kubernetes-node-windows-amd64.tar.gz 22fc9a7eb0e8244d51fd11f6d90f44e973983cb21692724c919493a235d2f9b1f22788fbfe8abaec9c52a98385767d2f0dd0bfebbc39c9e23c1047ebdaeb87cb

Changelog since v1.17.0-alpha.3

Action Required

  • Graduate ScheduleDaemonSetPods to GA. (feature gate will be removed in 1.18) action required. (#82795, @draveness)

Other notable changes

  • kube-scheduler: emits a warning when a malformed component config file is used with v1alpha1. (#84129, @obitech)
  • add azure disk encryption(SSE+CMK) support (#84605, @andyzhangx)
  • The certificate signer no longer accepts ca.key passwords via the CFSSL_CA_PK_PASSWORD environment variable. This capability was not prompted by user request, never advertised, and recommended against in the security audit. (#84677, @mikedanese)
  • Reduce default NodeStatusReportFrequency to 5 minutes. With this change, periodic node status updates will be send every 5m if node status doesn't change (otherwise they are still send with 10s). (#84007, @wojtek-t)
    • Bump NodeProblemDetector version to v0.8.0 to reduce forced NodeStatus updates frequency to 5 minutes.
  • CSI Topology feature is GA. The CSINodeInfo feature gate is deprecated and will be removed in a future release. The storage.k8s.io/v1beta1 CSINode object is deprecated and will be removed in a future release. (#83474, @msau42)
  • Only validate duplication of the RequestedToCapacityRatio custom priority and allow other custom predicates/priorities (#84646, @liu-cong)
  • Added kubelet serving certificate metric server_rotation_seconds which is a histogram reporting the age of a just rotated serving certificate in seconds. (#84534, @sambdavidson)
  • During namespace deletion some controllers create event and log spam because they do not recognize namespace deletion as a terminal state. (#84123, @smarterclayton)
  • Removed Alpha feature MountContainers (#84365, @codenrhoden)
  • People can see the right log and note. (#84637, @zhipengzuo)
  • Ensure the KUBE-MARK-DROP chain in kube-proxy mode=iptables. The chain is ensured for both ipv4 and ipv6 in dual-stack operation. (#84422, @aojea)
  • deprecate cleanup-ipvs flag (#83832, @gongguan)
  • Scheduler Policy API has a new recommended apiVersion "apiVersion: kubescheduler.config.k8s.io/v1" which is consistent with the scheduler API group "kubescheduler.config.k8s.io". It holds the same API as the old apiVersion "apiVersion: v1". (#83578, @Huang-Wei)
  • Fixed a bug in the single-numa-policy of the TopologyManager. Previously, best-effort pods would result in a terminated state with a TopologyAffinity error. Now they will run as expected. (#83777, @lmdaly)
  • local: support local filesystem volume with block resource reconstruction (#84218, @cofyc)
  • Fix the bug that EndpointSlice for masters wasn't created after enabling EndpointSlice feature on a pre-existing cluster. (#84421, @tnqn)
  • kubelet: a configuration file specified via --config is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. (#83204, @obitech)
  • kubeadm now propagates proxy environment variables to kube-proxy (#84559, @yastij)
  • Reload apiserver SNI certificates from disk every minute (#84303, @jackkleeman)
  • sourcesReady provides the readiness of kubelet configuration sources such as apiserver update readiness. (#81344, @zouyee)
  • Update Azure SDK versions to v35.0.0 (#84543, @andyzhangx)
  • Fixed EndpointSlice port name validation to match Endpoint port name validation (allowing port names longer than 15 characters) (#84481, @robscott)
  • Scheduler now reports metrics on cache size including nodes, pods, and assumed pods (#83508, @damemi)
  • kube-proxy: emits a warning when a malformed component config file is used with v1alpha1. (#84143, @phenixblue)
  • Update default etcd server version to 3.4.3 (#84329, @jingyih)
  • Scheduler policy configs can no longer be declared multiple times (#83963, @damemi)
  • This PR sets the --cluster-dns flag value to kube-dns service IP whether or not NodeLocal DNSCache is enabled. NodeLocal DNSCache will listen on both the link-local as well as the service IP. (#84383, @prameshj)
  • Remove prometheus cluster monitoring addon from kube-up (#83442, @serathius)
  • update the latest validated version of Docker to 19.03 (#84476, @neolit123)
  • kubeadm: always mount the kube-controller-manager hostPath volume that is given by the --flex-volume-plugin-dir flag. (#84468, @neolit123)
  • Introduce x-kubernetes-map-type annotation as a CRD API extension. Enables this particular validation for server-side apply. (#84113, @enxebre)
  • kube-scheduler now fallbacks to emitting events using core/v1 Events when events.k8s.io/v1beta1 is disabled. (#83692, @yastij)
  • Migrate controller-manager and scheduler to EndpointsLeases leader election. (#84084, @wojtek-t)
  • User can now use component config to configure NodeLabel plugin for the scheduler framework. (#84297, @liu-cong)
  • local: support local volume block mode reconstruction (#84173, @cofyc)
  • Fixed kubectl endpointslice output for get requests (#82603, @robscott)
  • set config.BindAddress to IPv4 address "127.0.0.1" if not specified (#83822, @zouyee)
  • CSI detach timeout increased from 10 seconds to 2 minutes (#84321, @cduchesne)
  • Update etcd client side to v3.4.3 (#83987, @wenjiaswe)
    • Deprecated prometheus request meta-metrics have been removed (http_request_duration_microseconds, http_request_duration_microseconds_sum, http_request_duration_microseconds_count, http_request_size_bytes, http_request_size_bytes_sum, http_request_size_bytes_count, http_requests_total, http_response_size_bytes, http_response_size_bytes_sum, http_response_size_bytes_count) due to removal from the prometheus client library. Prometheus http request meta-metrics are now generated from promhttp.InstrumentMetricHandler instead.
  • The built-in system:csi-external-provisioner and system:csi-external-attacher cluster roles are removed as of 1.17 release (#84282, @tedyu)
  • Pod labels can no longer be updated through the pod/status updates by nodes. (#84260, @tallclair)
  • Reload apiserver serving certificate from disk every minute (#84200, @jackkleeman)
  • Adds FQDN addressType support for EndpointSlice. (#84091, @robscott)
  • Add permit_wait_duration_seconds metric for scheduler. (#84011, @liu-cong)
  • Optimize inter-pod affinity preferredDuringSchedulingIgnoredDuringExecution type, up to 4x in some cases. (#84264, @ahg-g)
  • When a namespace is being deleted and spec.finalizers are still being processed, stop returning a 409 conflict error and instead return the object as we would during metadata.finalizer processing. (#84122, @smarterclayton)
  • client-ca bundles for the all generic-apiserver based servers will dynamically reload from disk on content changes (#83579, @deads2k)
  • Reduced frequency of DescribeVolumes calls of AWS API when attaching/detaching a volume. (#84181, @jsafrane)
  • Add a metric to track number of scheduler binding and prioritizing goroutines (#83535, @wgliang)
  • Fix kubelet metrics gathering on non-English Windows hosts (#84156, @wawa0210)
  • A new kubelet_preemptions metric is reported from Kubelets to track the number of preemptions occuring over time, and which resource is triggering those preemptions. (#84120, @smarterclayton)

v1.17.0-alpha.3

Documentation

Downloads for v1.17.0-alpha.3

filename sha512 hash
kubernetes.tar.gz dfdb758b21a3dbd820063cb2ba4b4a19e5e1e03fdb95856bf9c99c2c436bbc2c259cd9ac233f0388b5c3690f2c78680362130e045442f4da5b8b94c3013bdc72
kubernetes-src.tar.gz 1718547ef5baf7ab6514bafff05451fc9d2f0db0b74f094b4d9004e949ef86ed246abf538fabe221e1adbe5aabc39b831c5d332d1aca8d65d58050092b8bcc8c

Client Binaries

filename sha512 hash
kubernetes-client-darwin-386.tar.gz 207b281b7da796faa34beaf0c8f7e70f9685b132c2838a12e0c8f2084627e2c98890379cc84eff851349d74ec0a273c1f8967085e1c6471acfa1d5fcf251b1cb
kubernetes-client-darwin-amd64.tar.gz b0c19de40aa4210c0f06e1864779d60a69b75a443042f448746a0cd8cae680a7f4fab2dc7f3c61a31bde39ef9f490224be9559d6b15225cb7502b281c9968e51
kubernetes-client-linux-386.tar.gz e8149e6373b48ab97b844b5450be12ec4bb86c869cdf71f98b78b88a9a9ef535df443241bf385fa4588dbe44abd0771b08a2af64dcdd6b891a4d3001cee9ac95
kubernetes-client-linux-amd64.tar.gz d4176dfd049ffa1e59b7c4efd4d4189463153fa6cf53d5fc43c953983b74cbd75af9b8a0f7c13f86c5c3a3bb75ec453a676a931a38acaf32eb3ab98001d8f168
kubernetes-client-linux-arm.tar.gz d6de12989c091e78ad95d7e01274936a28ca74219196c27775d00665c9fb98fa1e485652c395114d3ed09534932390035e6d5c7c14d5753614929ecaf90baa2e
kubernetes-client-linux-arm64.tar.gz 2f62aeaa39d7b7ab0840bd6c845d73e6135357edbbf93046c1fbd52d02a8c19377787ec016af2db74f236c82b71c1f9f704b650ee689876138f0da828a61951b
kubernetes-client-linux-ppc64le.tar.gz 895fd028e409cca1667a08ba6d1b32517b23b796530eecef7c1a4783287b45ff826e692d7a4ca103979f5b3a2e8d6550c1df5b3144aa686fe7fed7122b2ef051
kubernetes-client-linux-s390x.tar.gz 50fe03594da3c90932e83d0befad9053ef1fb72f4af1a5c139455c8acc6c10adec522efe80195013f88a3b40b0371ff7de85544c3c3d770fea37cff727ff5147
kubernetes-client-windows-386.tar.gz 3e32d47078da5d3d31c4b854c01e081a437fab2c01c7e7be291262b029046acaf96efa6529383b3410b53bc2973ec82c6fe7b4eb4193d15eda4abb73210cecf6
kubernetes-client-windows-amd64.tar.gz 42ec1f3273ea070cedcf65bbb76ebf05a24aca5ed55af4b17889fb1be3df99b4e4c023099994cbec572968c297a4671ce4a966c9dacc3c8385a380741d067f2b

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz 0b1fe1eca603579f70c13509dff32dcb8383ee340b8fc6b8edf23cdaf1f86cf9c8d710380a9350de8b516c5c742eb10e28c97cadf7afa57532bd663c88cad96f
kubernetes-server-linux-arm.tar.gz ac1a804dd8980281afe12a116ffaf4ed9fc1066c3a531f3a6ced14f021c69d60ff7120bfbfe159cd93898b937c4c3baddce4429dd7933654f0a6f6bcbeed8fab
kubernetes-server-linux-arm64.tar.gz e34daf4c37ab5c2f52a116ed4ca0f7b52c0d0b5863d027550241b03f9019dcb6dd7d16df7c6ec7a43a86737b16015c19a2a75a19173c2dd9ee574b5f08098881
kubernetes-server-linux-ppc64le.tar.gz 9c285800beecc53cf5293604270cbcc5ba6245ddf4dfe0c0ec9a1359ed3771d7e1939c2496b2c24bf4cc541e8b29408ec66c24804b321612331c553b55c2c3c8
kubernetes-server-linux-s390x.tar.gz ea883497e80fdb2182342a2bae0382c7d38b0273cae6f8a5ee05d149669134d9b50613184e1fc289d7ce999c172f455f76b08e40cc014a33fc2c9c6491eee9c1

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 450197a746a52c3129f97e50b4b69bf2d1e94290be72f1ae143e36bb0fd76b2175cc917ae47389feb0163ae108a752929ac4fa8f90b8dee21f0be5e198c847a5
kubernetes-node-linux-arm.tar.gz c10c6ea591a15dc873afb01a7aa1916188e411ba57201f436d6d96cd2bad8cbb4e3bccc743759e3ce6d93e0f13c026ae5d646684611fe11134ba05321522f78c
kubernetes-node-linux-arm64.tar.gz 47764435a9367571f7de12d54f0aff7d615fc50383230c6fba08475dc33be60d2912dd2ee5c3f083a450866281c1df90663b1737f2d8293a73b48720aeda6a8b
kubernetes-node-linux-ppc64le.tar.gz 48d2694450f4b94e8ff76e95ef102670d4a4c933010afadf7a019db73d966462a07ff222f9b48510f5dd3ac8f9076e31646d490ec0c1425d4be7b8475cd11cd6
kubernetes-node-linux-s390x.tar.gz 20e712415af7304ecb55e9c2c2f29dda3af4a78f5833499c1f51a492c929a4590717d60fce537fb81c70784a6ca1503f7e731e1779cbc59673f69a03f7533bc0
kubernetes-node-windows-amd64.tar.gz 2e88bf26e1293dd733cf1bfe1f0f2dcfb5c482687cf52690488e943a0922a5c9565dd9ce000af76597e0bf1ac8f42ce12044c0ec5e3564db5c2f6a409e9efdb6

Changelog since v1.17.0-alpha.2

Action Required

  • Graduate TaintNodesByCondition to GA in 1.17. (feature gate will be removed in 1.18) action required (#82703, @draveness)

Other notable changes

  • TaintNodesByCondition was graduated to GA, CheckNodeMemoryPressure, CheckNodePIDPressure, CheckNodeDiskPressure, CheckNodeCondition were accidentally removed since 1.12, the replacement is to use CheckNodeUnschedulablePred (#84152, @draveness)
  • filter plugin for cloud provider storage predicate (#84148, @gongguan)
  • Fixed binding of block PersistentVolumes / PersistentVolumeClaims when BlockVolume feature is off. (#84049, @jsafrane)
  • Updated kube-proxy ipvs README with correct grep argument to list loaded ipvs modules (#83677, @pete911)
  • Add data cache flushing during unmount device for GCE-PD driver in Windows Server. (#83591, @jingxu97)
  • Adds a metric apiserver_request_error_total to kube-apiserver. This metric tallies the number of request_errors encountered by verb, group, version, resource, subresource, scope, component, and code. (#83427, @logicalhan)
  • Refactor scheduler's framework permit API. (#83756, @hex108)
  • The kubectl's api-resource command now has a --sort-by flag to sort resources by name or kind. (#81971, @laddng)
  • Update to use go1.12.12 (#84064, @cblecker)
  • Update to Ingress-GCE v1.6.1 (#84018, @rramkumar1)
  • Update Cluster Autoscaler version to 1.16.2 (CA release docs: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.16.2) (#84038, @losipiuk)
  • When scaling down a ReplicaSet, delete doubled up replicas first, where a "doubled up replica" is defined as one that is on the same node as an active replica belonging to a related ReplicaSet. ReplicaSets are considered "related" if they have a common controller (typically a Deployment). (#80004, @Miciah)
  • Promote WatchBookmark feature to GA. (#83195, @wojtek-t)
    • With WatchBookmark feature, clients are able to request watch events with BOOKMARK type. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session.
  • kubeadm no longer removes /etc/cni/net.d as it does not install it. Users should remove files from it manually or rely on the component that created them (#83950, @yastij)
  • kubeadm: enhance certs check-expiration to show the expiration info of related CAs (#83932, @SataQiu)
  • Add incoming pods metrics to scheduler queue. (#83577, @liu-cong)
  • An end-user may choose to request logs without confirming the identity of the backing kubelet. This feature can be disabled by setting the AllowInsecureBackendProxy feature-gate to false. (#83419, @deads2k)
  • Switched intstr.Type to sized integer to follow API guidelines and improve compatibility with proto libraries (#83956, @liggitt)
  • Fix handling tombstones in pod-disruption-budged controller. (#83951, @zouyee)
  • client-go: improved allocation behavior of the delaying workqueue when handling objects with far-future ready times. (#83945, @barkbay)
  • Added the crictl Windows binaries as well as the Linux 32bit binary to the release archives (#83944, @saschagrunert)
  • Fixed an issue with informers missing an Added event if a recently deleted object was immediately recreated at the same time the informer dropped a watch and relisted. (#83911, @matte21)
  • Allow dynamically set glog logging level of kube-scheduler (#83910, @mrkm4ntr)
  • clean duplicate GetPodServiceMemberships function (#83902, @gongguan)
  • Add information from Lease object corresponding to a given Node to kubectl describe node output (#83899, @wojtek-t)
  • Gives the right error message when using kubectl delete a wrong resource. (#83825, @zhouya0)
  • The userspace mode of kube-proxy no longer confusingly logs messages about deleting endpoints that it is actually adding. (#83644, @danwinship)
  • Add latency and request count metrics for scheduler framework. (#83569, @liu-cong)
  • ETCD version monitor metrics are now marked as with the ALPHA stability level. (#83283, @RainbowMango)
  • Significant kube-proxy performance improvements when using Endpoint Slices at scale. (#83206, @robscott)
  • Upgrade default etcd server version to 3.3.17 (#83804, @jpbetz)

v1.17.0-alpha.2

Documentation

Downloads for v1.17.0-alpha.2

filename sha512 hash
kubernetes.tar.gz 37583337b992d9a5ebe5a4677e08c13617b8b9db9ee8f049773b624351c00acacf02daca2f87a357aaa75edcc3a4db2c64e6a7da502a6153d06e228ff6be6006
kubernetes-src.tar.gz a44fee5be20c7fb64c58d0a69377074db05ec6889892c93ce970406cb393a1fde60a75612e74802cb2e0085b6357183c1f30e4b322dacf6f30597ab5fd5948f9

Client Binaries

filename sha512 hash
kubernetes-client-darwin-386.tar.gz 4aa92894eeaedb022e5409e08784ce1bd34ba268032ef93ad4c438b6ed9f1a210222f5f4a4fc68198d71e167c78bb7695459e4c99059898e1e0cf7c1ae70080c
kubernetes-client-darwin-amd64.tar.gz 1815a3bdd1c13782026fced8720201dea2e518dc56a43e2b53f89341108f03ec0b5ea6efadd8460ab1715b65ae52f9bdd49066f716573e0d76ff3036e193b8d3
kubernetes-client-linux-386.tar.gz 9a470907d6203e69c996f8db3cc257af23f9b35236ee2d5a87d22cd6056eef4f07671cd5711ec4999c1edd93385c4f7e5d6d0b8096404e88414a1ed83b58de4f
kubernetes-client-linux-amd64.tar.gz 011d44cf35c841d331a5a0d88b8a5deb7781fa678702ac6402050d096e72396dc76ccaa67a371273bc428612536357c19306d250bd47db4ac5147ff8cc5e1296
kubernetes-client-linux-arm.tar.gz 1f45d9a9852d2b0a0420b0a26b3add9031d7d691c55660d60580614e6ab6e2d732017832ed3f737f8a43db088e91b64edf12298675be6d128775dce3e4d0ddbe
kubernetes-client-linux-arm64.tar.gz e355f69caed044e5e27efe7ae42027e799a87ec647810fbadf644d147de2f6bd478e338ebb211044a9e6483d32f3534cc40d7b4d735d16d3b6c55e7975515f20
kubernetes-client-linux-ppc64le.tar.gz 355e0d8c5f241bc2303c38447c241ff8f5151af51aeacf15fa2b96e2721ecc011b5aec84c3f93a26aad86aa29179d16054e34d45bff2824c1abbf1deb571f0f5
kubernetes-client-linux-s390x.tar.gz 7cdfc6cde7922290b46f291a168519f4c923fee97968399940164a8a7d8592701b262b30fa299c13f025c70f46f5d32c17a9699f0bf3e5bd55ab4811f01f59ed
kubernetes-client-windows-386.tar.gz 7170da100b2d1d8700990c4175c7d048347b8dcc71e0ceb6c01728f5e6266dd0d5766e5206820d9e54d243ffa73abd5dd72715d6984598655f6160d43cb45a15
kubernetes-client-windows-amd64.tar.gz 74484b5c841e1c57c9baf88b84a9cbf3b9865527a8723815cbe8e7384805c80d971126c0b54d52e446d55b04e209984461ec8a8eff4c58aaa50397db0111cca5

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz 3fb3c5da6e45b32e8d89d4914f0b04cf95242cb0e4ea70b06a665c2975d8b6bbff6206e1f8769f49836b9dc12fb0946cc1986e475945413aff053661941f622b
kubernetes-server-linux-arm.tar.gz ff71c9a3f81f2e43d541b9b043e5f43fd30972c2b0ae5d9f3992f76effdcab2d027835844109ee3b501e365994f97aa5b6528a9d23db8ec3f05af6cb6d0e01d0
kubernetes-server-linux-arm64.tar.gz 26b9fce5ed930ad3eea5eeab3bec3b009f65837139f7da3644aacdcccda654fe542b03e1c4280950ca561f624ef24da01acff23e3f3b72d1001d794c8d6aa230
kubernetes-server-linux-ppc64le.tar.gz ad980f5efe83da1f2a202035eb1cff44ea72692fc3fc5f7d23fd8fc3b80a6797dbb263cc240d8fd2cde80a786b48352127f52c0a1db02e9d09a44440c1704406
kubernetes-server-linux-s390x.tar.gz 8e1ab7abd4c13c3d4211e5dd1be63ecd482691fd2cb7b2d3492bb2a02003ec33abe0a7b26e4e93f9586c5fc6fddbfbb559c4c28dcdc65564aeadceb2bc543a7d

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz c0928e414e439ba63321ce770a04ea332a4cc93ec87dd9d222fe3f5a593995111a6c0a60a413018d59367df6b4d0ab6f64904551f29f5c94ea406c68cc43b3b3
kubernetes-node-linux-arm.tar.gz 990a253ba49203348a587ca4d4acf7c25ff47a97b39519dfc7d5bdc2f3ea4713930e17dc6b9ff02a2a6ae2e84011d05d4471dfbfe1ab0627c102f9aa2205114d
kubernetes-node-linux-arm64.tar.gz 79381ad17eefc679fb549126eba23ffa65e625d0e1fec459dd54823897947b17a0e7ef6f446dc9e54f16b3e4995e4a084146dcf895e994813233953a3795e3a3
kubernetes-node-linux-ppc64le.tar.gz 7cfea9b9fa27dcc2024260e19d5e74db2175b491093c8906721d99c94b46af1c2b3ad91fe0fb799de639191fcb0e8ceab1b67bb260d615825002a3239c7b3ed0
kubernetes-node-linux-s390x.tar.gz 590bc2afd835a4a236a4a2ab2cde416aae9efdec14c34355a54b671d89308f3729f5af076139cc9c78e323666565ba1fa441149b681fc6addcab133205a3c41f
kubernetes-node-windows-amd64.tar.gz 4c15c7c30de0f9d921b534433332b14eb685ad8a3a416315def1cc1064b802227ea4b556bc53a68d75be898b49acadee8317a2355635a69d1c4d305d890e5009

Changelog since v1.17.0-alpha.1

Action Required

  • Expand scheduler priority functions and scheduling framework plugins' node score range to [0, 100]. action required. Note: this change is internal and does not affect extender and RequestedToCapacityRatio custom priority, which are still expected to provide a [0, 10] range. (#83522, @draveness)
  • action required: kubeadm: when adding extra apiserver authorization-modes, the defaults "Node,RBAC" are no longer prepended in the resulting static Pod manifests and a full override is allowed. (#82616, @ghouscht)
  • ACTION REQUIRED: kubeadm: properly enable kubelet client certificate rotation on primary control-plane nodes, created using "kubeadm init". A side effect of this change is that for external CA users, kubeadm now requires "bootstrap-kubelet.conf" instead of "kubelet.conf" during "kubeadm init" and its phases. (#83339, @neolit123)
  • Action Required: kubeadm.k8s.io/v1beta1 has been deprecated, you should update your config to use newer non-deprecated API versions. (#83276, @Klaven)

Other notable changes

  • [migration phase 1] PodFitsHostPorts as filter plugin (#83659, @wgliang)
  • [migration phase 1] PodFitsResources as framework plugin (#83650, @wgliang)
  • Fixed attachment of AWS volumes that have just been detached. (#83567, @jsafrane)
  • [migration phase 1] PodMatchNodeSelector/NodAffinity as filter plugin (#83660, @wgliang)
  • Upgrade to etcd client 3.3.17 to fix bug where etcd client does not parse IPv6 addresses correctly when members are joining, and to fix bug where failover on multi-member etcd cluster fails certificate check on DNS mismatch (#83801, @jpbetz)
  • Fixed panic when accessing CustomResources of a CRD with x-kubernetes-int-or-string. (#83787, @sttts)
  • Change pod_preemption_victims metric from Gauge to Histogram. (#83603, @Tabrizian)
  • Expose SharedInformerFactory in the framework handle (#83663, @draveness)
  • Add more tracing steps in generic_scheduler (#83539, @wgliang)
  • [migration phase 1] PodFitsHost as filter plugin (#83662, @wgliang)
  • The topology manager aligns resources for pods of all QoS classes with respect to NUMA locality, not just Guaranteed QoS pods. (#83492, @ConnorDoyle)
  • Fix unsafe JSON construction in a number of locations in the codebase (#81158, @zouyee)
  • Fixed a bug in the single-numa-node policy of the TopologyManager. Previously, pods that only requested CPU resources and did not request any third-party devices would fail to launch with a TopologyAffinity error. Now they will launch successfully. (#83697, @klueska)
  • Add per-pod scheduling metrics across 1 or more schedule attempts. (#83674, @liu-cong)
  • Fix validation message to mention bytes, not characters. (#80880, @DirectXMan12)
  • external facing APIs in pluginregistration and deviceplugin packages are now available under k8s.io/kubelet/pkg/apis/ (#83551, @dims)
  • Fix error where metrics related to dynamic kubelet config isn't registered (#83184, @odinuge)
  • The VolumeSubpathEnvExpansion feature is graduating to GA. The VolumeSubpathEnvExpansion feature gate is unconditionally enabled, and will be removed in v1.19. (#82578, @kevtaylor)
  • Openstack: Do not delete managed LB in case of security group reconciliation errors (#82264, @multi-io)
  • The mutating and validating admission webhook plugins now read configuration from the admissionregistration.k8s.io/v1 API. (#80883, @liggitt)
  • kubeadm: implemented structured output of 'kubeadm token list' in JSON, YAML, Go template and JsonPath formats (#78764, @bart0sh)
  • kube-proxy: a configuration file specified via --config is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. (#82927, @obitech)
  • Add "podInitialBackoffDurationSeconds" and "podMaxBackoffDurationSeconds" to the scheduler config API (#81263, @draveness)
  • Authentication token cache size is increased (from 4k to 32k) to support clusters with many nodes or many namespaces with active service accounts. (#83643, @lavalamp)
  • Bumps the minimum version of Go required for building Kubernetes to 1.12.4. (#83596, @jktomer)
  • kube-proxy iptables probabilities are now more granular and will result in better distribution beyond 319 endpoints. (#83599, @robscott)
  • Fixed the bug that deleted services were processed by EndpointSliceController repeatedly even their cleanup were successful. (#82996, @tnqn)
  • If a bad flag is supplied to a kubectl command, only a tip to run --help is printed, instead of the usage menu. Usage menu is printed upon running kubectl command --help. (#82423, @sallyom)
  • If container fails because ContainerCannotRun, do not utilize the FallbackToLogsOnError TerminationMessagePolicy, as it masks more useful logs. (#81280, @yqwang-ms)
  • Fixed cleanup of raw block devices after kubelet restart. (#83451, @jsafrane)
  • Commands like kubectl apply now return errors if schema-invalid annotations are specified, rather than silently dropping the entire annotations section. (#83552, @liggitt)
  • Expose kubernetes client in the scheduling framework handle. (#82432, @draveness)
  • kubeadm: fix wrong default value for the "upgrade node --certificate-renewal" flag. (#83528, @neolit123)
  • IP validates if a string is a valid IP address (#83104, @zouyee)
  • The --certificate-authority flag now correctly overrides existing skip TLS or CA data settings in the kubeconfig file (#83547, @liggitt)
  • hyperkube will now be available in a new github repository and will not be included in the kubernetes release from 1.17 onwards (#83454, @dims)
  • more complete and accurate logging of stack backtraces in E2E failures (#82176, @pohly)
  • Kubeadm: add support for 127.0.0.1 as advertise address. kubeadm will automatically replace this value with matching global unicast IP address on the loopback interface. (#83475, @fabriziopandini)
  • Rename PluginContext to CycleState in the scheduling framework (#83430, @draveness)
  • kube-scheduler: a configuration file specified via --config is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. (#83030, @obitech)
  • Significant kube-proxy performance improvements for non UDP ports. (#83208, @robscott)
  • Fixes a flaw (CVE-2019-11253) in json/yaml decoding where large or malformed documents could consume excessive server resources. Request bodies for normal API requests (create/delete/update/patch operations of regular resources) are now limited to 3MB. (#83261, @liggitt)

v1.17.0-alpha.1

Documentation

Downloads for v1.17.0-alpha.1

filename sha512 hash
kubernetes.tar.gz 40985964b5f4b1e1eb448a8ca61ae5fe05b76cf4e97a4a6b0df0f7933071239ed8c3a6753d8ed8ba0c963694c0f94cce2b5976ddcc0386018cdc66337d80d006
kubernetes-src.tar.gz 475dfeb8544804dcc206f2284205fb1ee0bcb73169419be5e548ff91ffe6a35cea7e94039af562baee15933bef3afaa7ff10185e40926c7baa60d5936bcc9c1b

Client Binaries

filename sha512 hash
kubernetes-client-darwin-386.tar.gz 3f894661ed9b6ed3e75b6882e6c3e4858325f3b7c5c83cb8f7f632a8c8f30dd96a7dd277e4676a8a2ab598fe68da6473f414b494c63bfb4ed386a20dad7ae11a
kubernetes-client-darwin-amd64.tar.gz f3070d79b0835fdc0791bbc31a334d8b46bf1bbb02f389c871b31063417598d17dd464532df420f2fa0dbbbb9f8cc0730a7ca4e19af09f873e0777d1e296f20c
kubernetes-client-linux-386.tar.gz 64e8961fa32a18a780e40b753772c6794c90a6dd5834388fd67564bb36f5301ea82377893f51e7c7c7247f91ca813e59f5f293522a166341339c2e5d34ac3f28
kubernetes-client-linux-amd64.tar.gz d7ba0f5f4c879d8dcd4404a7c18768190f82985425ab394ddc832ee71c407d0ac517181a24fd5ca2ebfd948c6fa63d095a43c30cf195c9b9637e1a762a2d8d2f
kubernetes-client-linux-arm.tar.gz 36fc47ee9530ee8a89d64d4be6b78b09831d0838a01b63d2a824a9e7dd0c2127ef1b49f539d16ba1248fbf40a7eb507b968b18c59080e7b80a7a573138218e36
kubernetes-client-linux-arm64.tar.gz a0a8fba0f4424f0a1cb7bad21244f47f98ba717165eaa49558c2612e1949a1b34027e23ccbd44959b391b6d9f82046c5bc07eb7d773603b678bbc0e5bf54502c
kubernetes-client-linux-ppc64le.tar.gz eaae9ed0cc8c17f27cff31d92c95c11343b9f383de27e335c83bfdf236e6da6ab55a9d89b3e0b087be159d6b64de21827ca19c861ecfb6471b394ea3720bcb61
kubernetes-client-linux-s390x.tar.gz 994cf2dc42d20d36956a51b98dde31a00eae3bd853f7be4fbc32f48fec7b323a47ea5d841f31d2ca41036d27fbfaa3be4f2286654711245accf01c3be81f540c
kubernetes-client-windows-386.tar.gz 68ebe4abea5a174eb189caea567e24e87cca57e7fbc9f8ec344aafbaf48c892d52d179fef67f9825be0eb93f5577f7573873b946e688de78c442c798a5b426bc
kubernetes-client-windows-amd64.tar.gz f29cd3caf5b40622366eae87e8abb47bea507f275257279587b507a00a858de87bcfa56894ae8cd6ba754688fd5cdf093ce6c4e0d0fd1e21ca487a3a8a9fd9f9

Server Binaries

filename sha512 hash
kubernetes-server-linux-amd64.tar.gz 93e560e8572c6a593583d20a35185b93d04c950e6b1980a7b40ca5798958d184724ddebd1fa9377cfe87be4d11169bdba2a9f7fa192690f9edae04779aaf93a4
kubernetes-server-linux-arm.tar.gz fe2af93336280e1251f97afecbdfb7416fd9dd7d08b3e5539abeea8ccaf7114cac399e832fa52359d2bc63ec9f8703ae3bca340db85f9b764816f4c36e4eefee
kubernetes-server-linux-arm64.tar.gz efc32c8477efda554d8e82d6e52728f58e706d3d53d1072099b3297c310632e8adece6030427154287d5525e74158c0b44a33421b3dd0ffb57372d63768e82ec
kubernetes-server-linux-ppc64le.tar.gz bda4fce6f5be7d0102ff265e0ba10e4dab776caeba1cebdf57db9891a34b4974fa57ac014aa6eca2dcfc1b64e9f67c8168e18026ae30c72ba61205d180f6e8ff
kubernetes-server-linux-s390x.tar.gz 655c7157176f4f972c80877d73b0e390aaff455a5dcd046b469eb0f07d18ea1aaef447f90127be699e74518072ea1605652798fa431eb6ac7ee4e4fd85676362

Node Binaries

filename sha512 hash
kubernetes-node-linux-amd64.tar.gz 1ec25c0350973ed06f657f2b613eb07308a9a4f0af7e72ebc5730c3c8d39ce3139a567acc7a224bebbe4e3496633b6053082b7172e2ce76b228c4b697f03f3d1
kubernetes-node-linux-arm.tar.gz c65ac3db834596bcb9e81ffa5b94493244385073a232e9f7853759bce2b96a8199f79081d2f00a1b5502d53dc1e82a89afa97ffdb83994f67ebc261de9fb62b9
kubernetes-node-linux-arm64.tar.gz 0de8af66269db1ef7513f92811ec52a780abb3c9c49c0a4de9337eb987119bb583d03327c55353b4375d233e1a07a382cc91bdbf9477cf66e3f9e7fb0090499e
kubernetes-node-linux-ppc64le.tar.gz adb43c68cd5d1d52f254a14d80bb66667bfc8b367176ff2ed242184cf0b5accd3206bcbd42dec3f132bf1a230193812ae3e7a0c48f68634cb5f67538385e142a
kubernetes-node-linux-s390x.tar.gz 1c834cfc06b9ba4a6da3bca2d504b734c935436546bc9304c7933e256dba849d665d34e82f48180f3975a907d37fec5ffb929923352ff63e1d3ff84143eea65b
kubernetes-node-windows-amd64.tar.gz 6fc54fd17ebb65a6bd3d4efe93a713cc2aaea54599ddd3d73d01e93d6484087271b3ca65ed7a5861090356224140776a9606c10873b6b106bc9a6634c25b1677

Changelog since v1.16.0

Action Required

  • The deprecated feature gates GCERegionalPersistentDisk, EnableAggregatedDiscoveryTimeout and PersistentLocalVolumes are now unconditionally enabled and can no longer be specified in component invocations. (#82472, @draveness)
  • ACTION REQUIRED: (#81668, @darshanime)
    • Deprecate the default service IP CIDR. The previous default was 10.0.0.0/24 which will be removed in 6 months/2 releases. Cluster admins must specify their own desired value, by using --service-cluster-ip-range on kube-apiserver.
  • Remove deprecated "include-uninitialized" flag. action required (#80337, @draveness)

Other notable changes

  • Bump version of event-exporter to 0.3.1, to switch it to protobuf. (#83396, @loburm)
  • kubeadm: use the --service-cluster-ip-range flag to init or use the ServiceSubnet field in the kubeadm config to pass a comma separated list of Service CIDRs. (#82473, @Arvinderpal)
  • Remove MaxPriority in the scheduler API, please use MaxNodeScore or MaxExtenderPriority instead. (#83386, @draveness)
  • Fixes a goroutine leak in kube-apiserver when a request times out. (#83333, @lavalamp)
  • Some scheduler extender API fields are moved from pkg/scheduler/api to pkg/scheduler/apis/extender/v1. (#83262, @Huang-Wei)
  • Fix aggressive VM calls for Azure VMSS (#83102, @feiskyer)
  • Update Azure load balancer to prevent orphaned public IP addresses (#82890, @chewong)
  • Use online nodes instead of possible nodes when discovering available NUMA nodes (#83196, @zouyee)
  • Fix typos in certificates.k8s.io/v1beta1 KeyUsage constant names: UsageContentCommittment becomes UsageContentCommitment and UsageNetscapSGC becomes UsageNetscapeSGC. (#82511, @abursavich)
  • Fixes the bug in informer-gen that it produces incorrect code if a type has nonNamespaced tag set. (#80458, @tatsuhiro-t)
  • Update to go 1.12.10 (#83139, @cblecker)
  • Update crictl to v1.16.1. (#82856, @Random-Liu)
  • Reduces the number of calls made to the Azure API when requesting the instance view of a virtual machine scale set node. (#82496, @hasheddan)
  • Consolidate ScoreWithNormalizePlugin into the ScorePlugin interface (#83042, @draveness)
  • On AWS nodes with multiple network interfaces, kubelet should now more reliably report the same primary node IP. (#80747, @danwinship)
  • Fixes kube-proxy bug accessing self nodeip:port on windows (#83027, @liggitt)
  • Resolves bottleneck in internal API server communication that can cause increased goroutines and degrade API Server performance (#80465, @answer1991)
  • The deprecated mondo kubernetes-test tarball is no longer built. Users running Kubernetes e2e tests should use the kubernetes-test-portable and kubernetes-test-{OS}-{ARCH} tarballs instead. (#83093, @ixdy)
  • Improved performance of kube-proxy with EndpointSlice enabled with more efficient sorting. (#83035, @robscott)
  • New APIs to allow adding/removing pods from pre-calculated prefilter state in the scheduling framework (#82912, @ahg-g)
  • Conformance tests may now include disruptive tests. If you are running tests against a live cluster, consider skipping those tests tagged as Disruptive to avoid non-test workloads being impacted. Be aware, skipping any conformance tests (even disruptive ones) will make the results ineligible for consideration for the CNCF Certified Kubernetes program. (#82664, @johnSchnake)
  • Resolves regression generating informers for packages whose names contain . characters (#82410, @nikhita)
  • Added metrics 'authentication_latency_seconds' that can be used to understand the latency of authentication. (#82409, @RainbowMango)
  • kube-dns add-on: (#82347, @pjbgf)
      • All containers are now being executed under more restrictive privileges.
      • Most of the containers now run as non-root user and has the root filesystem set as read-only.
      • The remaining container running as root only has the minimum Linux capabilities it requires to run.
      • Privilege escalation has been disabled for all containers.
  • k8s dockerconfigjson secrets are now compatible with docker config desktop authentication credentials files (#82148, @bbourbie)
  • Use ipv4 in wincat port forward. (#83036, @liyanhui1228)
  • Added Clone method to the scheduling framework's PluginContext and ContextData. (#82951, @ahg-g)
  • Bump metrics-server to v0.3.5 (#83015, @olagacek)
  • dashboard: disable the dashboard Deployment on non-Linux nodes. This step is required to support Windows worker nodes. (#82975, @wawa0210)
  • Fix possible fd leak and closing of dirs when using openstack (#82873, @odinuge)
  • PersistentVolumeLabel admission plugin, responsible for labeling PersistentVolumes with topology labels, now does not overwrite existing labels on PVs that were dynamically provisioned. It trusts the dynamic provisioning that it provided the correct labels to the PersistentVolume, saving one potentially expensive cloud API call. PersistentVolumes created manually by users are labelled by the admission plugin in the same way as before. (#82830, @jsafrane)
  • Fixes a panic in kube-controller-manager cleaning up bootstrap tokens (#82887, @tedyu)
  • Fixed a scheduler panic when using PodAffinity. (#82841, @Huang-Wei)
  • Modified the scheduling framework's Filter API. (#82842, @ahg-g)
  • Fix panic in kubelet when running IPv4/IPv6 dual-stack mode with a CNI plugin (#82508, @aanm)
  • Kubernetes no longer monitors firewalld. On systems using firewalld for firewall (#81517, @danwinship)
    • maintenance, kube-proxy will take slightly longer to recover from disruptive
    • firewalld operations that delete kube-proxy's iptables rules.
  • Added cloud operation count metrics to azure cloud controller manager. (#82574, @kkmsft)
  • Report non-confusing error for negative storage size in PVC spec. (#82759, @sttts)
  • When registering with a 1.17+ API server, MutatingWebhookConfiguration and ValidatingWebhookConfiguration objects can now request that only v1 AdmissionReview requests be sent to them. Previously, webhooks were required to support receiving v1beta1 AdmissionReview requests as well for compatibility with API servers <= 1.15. (#82707, @liggitt) * When registering with a 1.17+ API server, a CustomResourceDefinition conversion webhook can now request that only v1 ConversionReview requests be sent to them. Previously, conversion webhooks were required to support receiving v1beta1 ConversionReview requests as well for compatibility with API servers <= 1.15.
  • Resolves issue with /readyz and /livez not including etcd and kms health checks (#82713, @logicalhan)
  • fix: azure disk detach failure if node not exists (#82640, @andyzhangx)
  • Single static pod files and pod files from http endpoints cannot be larger than 10 MB. HTTP probe payloads are now truncated to 10KB. (#82669, @rphillips)
  • Restores compatibility with <=1.15.x custom resources by not publishing OpenAPI for non-structural custom resource definitions (#82653, @liggitt)
  • Take the context as the first argument of Schedule. (#82119, @wgliang)
  • Fixes regression in logging spurious stack traces when proxied connections are closed by the backend (#82588, @liggitt)
  • Correct a reference to a not/no longer used kustomize subcommand in the documentation (#82535, @demobox)
  • Limit the body length of exec readiness/liveness probes. remote CRIs and Docker shim read a max of 16MB output of which the exec probe itself inspects 10kb. (#82514, @dims)
  • fixed an issue that the correct PluginConfig.Args is not passed to the corresponding PluginFactory in kube-scheduler when multiple PluginConfig items are defined. (#82483, @everpeace)
  • Adding TerminationGracePeriodSeconds to the test framework API (#82170, @vivekbagade)
  • /test/e2e/framework: Adds a flag "non-blocking-taints" which allows tests to run in environments with tainted nodes. String value should be a comma-separated list. (#81043, @johnSchnake)