Private clusters #2326

CecileRobertMichon · 2018-02-22T23:44:23Z

What this PR does / why we need it: The goal of this PR is to allow building a cluster without public IPs by:

building a cluster without a publically facing load balancer
generating kubeconfig artifacts that point to the API server via the internal load balancer IP address
The option can be enabled with the flag enablePrivateCluster in KubernetesConfig in the apimodel. By default enablePrivateCluster is false.
A user would then be able to move the appropriate kubeconfig artifact onto a host with kubectl installed in the same k8s VNET (or onto a peer VNET that routes to the k8s VNET) and access the cluster from that host.

What is left TODO:

add docs
add example
remove publicIP resource (right now it is still being created but the DNS name doesn't resolve to an actual IP) and master FQDN output

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #221

Special notes for your reviewer:

What do people think about calling the flag enablePrivateCluster and placing it inside the KubernetesConfig? Another option would be to place it in the MasterProfile but I thought this would add some complexity since it only applies to k8s clusters. Thoughts?

Release note:

Move change to a separate PR because it is unrelated

dennis-benzinger-hybris · 2018-02-23T10:22:20Z

Another option would be to place it in the MasterProfile but I thought this would add some complexity since it only applies to k8s clusters. Thoughts?

MasterProfile seems to be more about VM level settings and not about the the K8s cluster as such. So I would prefer KubernetesConfig.

jackfrancis · 2018-02-26T21:58:04Z

parts/k8s/kubernetesagentresourcesvmas.t

@@ -68,10 +68,12 @@
        "name": "loop"
      },
      {{if not IsHostedMaster}}
+      {{if not IsPrivateCluster}}


Is there no way to do an and to the above required if conditions?

jackfrancis

lgtm

yfried · 2018-02-27T17:02:46Z

Hi,
How do I create the jumpbox? the doc just tells me to create a VM.
Do I need the jumpbox if I have a VPN connection to the cluster?

CecileRobertMichon · 2018-02-27T18:33:45Z

@yfried if you have a vnet peered with another or a VPN connection to the cluster then you don't need a jumpbox and you can use the other vnet or your VPN to access the cluster.

If you don't have a peer vnet nor a VPN you will need to create a jumpbox. After deploying your cluster with no public IPs, you can use the Azure Portal to create a VM in the same resource group and vnet as your cluster. You should then be able to ssh into the VM and access your private cluster (after following the instructions in the docs to install kubectl and copy over the kubeconfig).

The next step to this process is to automate the jumpbox creation as part of deployment but the first step is to get feedback on the initial implementation and make sure we're meeting the functionality + security requirements.

SilverGaucho · 2018-03-02T14:06:55Z

Hi

@CecileRobertMichon the flag enablePrivateCluster is not a recognized Flag using --api-model by the acs-engine

CecileRobertMichon · 2018-03-02T22:08:40Z

@SilverGaucho are you building from the master branch? This PR has not yet been included in an acs-engine release

sylus · 2018-03-07T15:54:51Z

I was curious if I have already deployed a cluster is it possible to change my apimodel and then do another az group deployment or will I need a fresh cluster?

Thanks for the amazing work!

CecileRobertMichon · 2018-03-07T18:03:09Z

@sylus the simple answer is no. Right now there is no "official" solution for modifying running clusters with acs-engine. Note that if you can, it is better hygiene to build a fresh cluster whenever possible. That being said, there is a way to manually modify a running cluster to delete all public IP resources and make it private. I can walk you through it if you're interested.

sylus · 2018-03-08T13:54:24Z

Ah thanks you so much for your response, I am using Ark so in theory it should be easy to bring up another cluster. We had a lot of security policies re AD and a custom vnet so have mostly just being doing az group deployment updates and have been lucky so far that nothing has seemed to have any issues.

Thanks for the heads up that this won't be possible and what the best practice is, I greatly appreciate it.

I definitely would be interested if you get the chance and will happily submit a P.R. based on that if helps. Thanks again! :)

yfried · 2018-03-13T15:59:58Z

Hi, when will this be released? I see it's not in 1.13.

sylus · 2018-03-13T16:53:27Z

@CecileRobertMichon if get a chance would mind pinging me on k8s slack channel so can run through private ip? Thanks for any help you can provide :) Username: willh

CecileRobertMichon · 2018-03-13T17:50:09Z

@yfried this will be released very shortly in v0.14.0 along with PR #2401. @sylus I will ping you when I get the chance, might not be today though, we're trying to get k8s 1.9.4 merged and v0.14.0 out...

jackfrancis · 2018-03-14T18:57:12Z

This is in https://github.com/Azure/acs-engine/releases/tag/v0.14.0

* add isprivatecluster func * wip remove load balancer for PC * add enablePrivateCluster flag * no public IPs for private cluster * working private cluster for 3 masters * remove duplicate iptables cmd * remove useless function * fmt * revert dnsprefix docs change * undo etcd change Move change to a separate PR because it is unrelated * remove masterPublicIpAddress * fix typo * handle DCOS + swarm nil case * add docs and example * replace host by jumpbox in the docs * add instructions to create jumpbox * indents

This reverts commit 1c3d6a7.

Revert private clusters

This reverts commit 1c3d6a7.

* update ubuntu image for German cloud (Azure#2036) * Fix issue with apiserver when using AADProfile (Azure#2047) (Azure#2055) * Fix issue with apiserver when using AADProfile (Azure#2047) * fixing failed test * missed another test * clear containers (Azure#1945) * clear-containers: add runtime to api and pass through parameters Signed-off-by: Jess Frazelle <[email protected]> * clear-containers: add scripts Signed-off-by: Jess Frazelle <[email protected]> * clear-containers: add example Signed-off-by: Jess Frazelle <[email protected]> * clear-containers: fix variables Signed-off-by: Jess Frazelle <[email protected]> * clear-containers: add docs Signed-off-by: Jess Frazelle <[email protected]> * clear-containers: update install script Signed-off-by: Jess Frazelle <[email protected]> * clear-containers: fix script Signed-off-by: Jess Frazelle <[email protected]> * clear-containers: update example Signed-off-by: Jess Frazelle <[email protected]> * clear-containers: update features docs Signed-off-by: Jess Frazelle <[email protected]> * clear-containers: make test linters happy Signed-off-by: Jess Frazelle <[email protected]> * setKubeletOpts to work better with kubeconfig Signed-off-by: Jess Frazelle <[email protected]> * whitespace cruft * more whitespace fun * Add --feature-gates handling for kubelet and api server (Azure#2032) * Add --feature-gates handling For kubeletConfig, preserve the existing behaviour of adding Accelerators=true for agent config (for kubernetes 1.6.0 and later) * simplified default implementation and removed KUBELET_FEATURE_GATES * unnecessary default assignment, and simple validation * removed copyMap func * enforce min version, "Accelerators=true" is only for agents * Pass current version in to addDefaultFeatureGates Generalise addDefaultFeatureGates by passing in current orchestrator version, and deferring the minimum required version to callers * Add tests for --feature-gates behaviour Test for only adding Accelerators=true for 1.6.0+ Test for correct application of KubeletConfig from top-level vs Master/AgentProfile * Avoid ref sharing for kubeletConfig Fix failing test by avoiding reference sharing of kubeletConfig properites on master and agent profiles * Remove outdated comments * k8s/script: allow parallelizing custom script without clear-containers (Azure#2067) Signed-off-by: Jess Frazelle <[email protected]> * Improving IP address assignment for master nodes with Azure CNI. (Azure#1966) * Azure cni static ip change (#1) * VSTS#1828538 Modified master IPs from dynamic to static and made agent nic dependent on master nic * Modified firstconsecutive static IP for default vnet * updated documentation specific to Azure CNI * Fixed styling test errors * Handled no master scenario * updated one of the examples cluster config to use firstConsecutiveStaticIp from the start of subnet * removed azureCNI check so that agent nic will always depend on master nic * Specified dependency of agent nic on master nic in windows agents template * moving firstConsecutiveStaticIP away from the edge of usable address space * Update documentation to help customers how to specify firstConsecutiveStaticIP and ipAddressCount for master nodes. * Add support for Kubernetes v1.8.7 (Azure#2068) * added support for Kubernetes v1.8.7 // TODO: build/publish Windows image * fix unit test * fixed dirty windows 1.8.6 build artifact * Upgrade Azure CNI to 1.0.1 (Azure#2064) * Azure CNI version bump * s/conf/conflist * Adopt CIS Kubernetes Benchmark, Part 2: Controller Manager. (Azure#2066) * Jenkins soak tests (Azure#2028) * added soak test * added soak test name * add location to name * delete rg if re-provisioning cluster * fix typo * added wait flag * fix no wait arg * add err log * remove ssh key * remove unused vars * Revert "remove ssh key" This reverts commit 9041c1a. * terminology * delete files if deployment failed * remove soak test spec * Pass in soak cluster name * only generate ssh key once * add default for KeyGenerated * fix typo * Do not generate SSH for soak test * fix typo in comment * add dashboard test debug log * Revert "add dashboard test debug log" This reverts commit 4937282. * Add Enable Pod Security Option (Azure#2048) * Add PodSecurityPolicy * use helpers.IsTrueBoolPointer, delete EnablePodSecurityPolicy function and update defaultAPIServerConfig * latest dashboard for v1.8 and v1.9 clusters (Azure#2070) * latest addon-resizer for v1.8 and v1.9 clusters (Azure#2071) * update kube-dns for v1.8 and v1.9 k8s clusters (Azure#2073) * latest kube-dns for v1.8 and v1.9 clusters * k8s-dns-dnsmasq-nanny-amd64:1.14.5 * latest heapster for v1.8 and v1.9 clusters (Azure#2072) * update pause image for v1.8 and v1.9 k8s clusters (Azure#2074) * latest pause image for v1.8 and v1.9 clusters * omitted v1.9.1 * changed wording of error (Azure#2089) * update Ubuntu image (Azure#2079) * Update docs for --feature-gates (Azure#2081) * re-enable read-only port on kubelet (Azure#2091) fixes heapster connection issues * revert addon-resizer version update (Azure#2090) * revert heapster version and re-enable kubelet read-only-port * revert addon-resizer to 1.7 * isolated bug fix to addon-resizer version * Add support for k8s 1.9.2 (Azure#2092) * add support for k8s 1.9.2 * updated windows zip * revert to 1.7 for addon resizer * Extend windows os drive size when customized OSDiskSizeGB is used (Azure#2097) * Adopt CIS Kubernetes Benchmark, Part 3: Kubelet (Azure#2098) * Restore KubernetesConfig sans struct embedding (Azure#2108) * restore properties to KubernetesConfig * lint * comment * rebase errata * Kind should be only EncryptionConfig for encyption-config.yaml (Azure#2104) * Kind should be only Config * removed the wrong Kind entry! * remove redundant apiVersion * running CSE provisioning script in foreground (Azure#2113) * Add autoscale test to E2E (Azure#2096) * initial attempt at autoscale test * working autoscale test * only add add’l options if passed in * Adding 3 replicas for load tester deployment * wait longer and linux only * skip autoscale test for v1.9 clusters Azure#2114 * Add member update after restarting etcd (Azure#2118) * Remove SecurityContextDeny setting in API server admission control. (Azure#2125) * Update custom vnet doc (Azure#2128) * add Azure Active Directory Admin Group Object ID flag (Azure#2111) * we don’t want to see stderr when checking for provision.complete (Azure#2126) * only create cert files on master (Azure#2120) * only create cert files on master * master node provision script cleanup * Enable iptables forward for kubernetes (Azure#2139) * --authorization-mode=Node only if secure kubelet (Azure#2138) * --authorization-mode=Node only if secure kubelet * EnableSecureKubelet unit test errata, using defaults generally * Validate k8s versions for PSP (Azure#2145) * Update window binary build documentation (Azure#2147) * Update window binary build documentation * move background section * Upgrade docker-engine to 1.13.* for all Kubernetes clusters >= v1.7 (Azure#2144) * update 1.7, 1.8, 1.9 latest to docker 1.13.* * docker-engine update to 1.13.* for >= 1.7 clusters * remove add 3 hours from timestamp, add’l \n (Azure#2149) * remove add 3 hours from timestamp, add’l \n * more \n * language * Add regression tests for 3 masters & 5 masters (Azure#2154) * added multi master configs * fix fmt * Mount in /var/lib/cni from the host. (Azure#2165) * Kubernetes E2E: test addons if present (Azure#2156) * conditional addon tests * uses generated model to introspect cluster features * I heart output * deployment flows need expanded cluster definition * reverting to ClusterDefinition for node counts * standard stdout implementation for all commands * typo * disable broken chmod command * stdout tweaks * retrieving deployment error details during upgrade (Azure#1995) * analyze and return deployment status during upgrade * added unittests for DeployTemplateSync * adding francecentral to azureconst generative script (Azure#2164) * add francecentral (Azure#2167) * validation error if custom VNET + Windows (Azure#2168) * Allow 1 core master node VM sizes (Azure#2173) * Allow DS1 Master VM sizes * 1 core for K8s masters * revert constants * update azure consts (Azure#2179) * update prometheus-grafana addon (Azure#2183) * Update clusterdefinition.md (Azure#2171) Fix apiserver options table markdown * Adding ServiceNodeExclusion as a default flag for Controller Manager (Azure#2180) * remove francecentral (Azure#2193) * improve networkpolicy documentation (Azure#2170) * Protect etcd tls from race conditions (Azure#2160) * chown etcd for keys in custom script * remove certs complete * add longer retry for etcd * remove second retry cmd * fix retrycmd_if_failure * add retries * passwd -u “etcd” * fix redirect output * pulling down provision logs during e2e runs (Azure#2190) * pulling down provision logs during e2e runs * setting no deployment retries by default * remove /opt/azure/containers/setup-etcd.sh logs * logs = finding bugs! * deleting deployments in resource groups (Azure#2195) * added kubernetes version validation for managed clusters (Azure#2194) * fix-api-server-bind-address-flag - Fixes a flag typo in the api serve… (Azure#2192) * fix-api-server-bind-address-flag - Fixes a flag typo in the api server yaml * fix-api-server-bind-address-flag - update docs with --bind-address fix * Update doc entry for bind-address flag * Network validation checks during provision (Azure#2196) * Add DNS + HTTPS checks, capture DNS packets * ARM doesn’t like ‘{‘ * standardizing retrycmd_if_failure usage patterns * Adding DNS pre-check for aptdocker.azureedge.net * tracking time for each retried provision event * standardizing to 3 masters api model for e2e tests * retain e2e resources for debugging * getting metrics logs from all cluster hosts * improved master/agent host retrieval * lint * lint * Adding “agent” substring to e2e api model pools * invalid agent pool name * revert agent forwarding ssh config * restore cleanup * add agent dns validation * 5 seconds between etcddisk mount retries * Fix DC/OS release version (Azure#2197) * update ApiServerConfig customization/override example (Azure#2201) * idiomatic windows e2e definition (Azure#2206) * don’t abort errors during log gathering (Azure#2207) * Make sure --cluster-dns uses DNSServiceIp set in KubernetesConfig and not always default value (Azure#2078) * Updated broken links (Azure#2208) * E2E nginx outbound access test: simplify port test (Azure#2204) * replace curl with nc * what does output look like on mismatch? * We are testing for outbound internet access not web content matching * testing curl for err * real tests, and not installing curl a bunch of times * don’t cleanup k8s e2e clusters (Azure#2210) * Add additional v3 vm sizes permitted for dcos master and agent (Azure#2184) * Cloud init improvements (Azure#2203) * chown etcd for keys in custom script * remove certs complete * add longer retry for etcd * remove second retry cmd * fix retrycmd_if_failure * add retries * passwd -u “etcd” * fix redirect output * remove extra lines * ignore warnings for etcd user changes * Parametrize retry cmd * removed unused data dir * use etcd args * Revert "use etcd args" This reverts commit ccbff6d. * parametrize sleep * changed the retries to 120 for network stuff * Remove Agent NICs dependency on Master NICs during upgrade. (Azure#2213) * replaced apierror with armerror (Azure#2205) * replaced apierror with armerror * addressed comments * addressed comments * reverted change in pkg/api/types.go * Kubernetes provision script: check for kubectl and docker files (Azure#2211) * unnecessary add’l systemctl enable * generalize ensureFilepath * fail provision if etcd check fails * rationalize azureconst (Azure#2215) * e2e ssh cleanup (Azure#2216) * return nil error on successful deployment (Azure#2218) * explicitly check aptdocker.azureedge.net (Azure#2220) * Add more etcd setup visibility (Azure#2214) * add etcd setup log to artifacts * remove hiding useradd output * show output of user add * add check for etcd user * add default audit policy (Azure#2189) * add default audit policy * apiserver audit log rotation is user-configurable * add nc checks to agent (Azure#2221) * Enabling Azure CNI for Windows (Azure#2174) * enabling azure cni * delete overwrite * address comments * address comments * fix kubeStartStr * fix kubeStartStr * remove misc files * squash commits for kubeStartStr * passed final test * rebase cleanup * setting Azure CNI for vlabs only * default back to kubenet * more set -x (Azure#2224) * more set -x * send ps to background * timestamps * adding certs dependency in cloud-init * rationalize etcd certs dep * extra ensure_etcd_ready * fixed version checking for managed clusters (Azure#2226) * Enabled preprovisioning on windows dcos agents (Azure#2228) * retry get aptdocker gpg key many times (Azure#2229) * Keyvault etcd certs (Azure#2155) * Use single values for etcdpeer key params * fixed param logic and added logic to vars * remove unused code * only add master certs/keys to params and vars if master is not hosted * move apiserver cert * add master profile != nil check * undo move api server key * Enable cloud controller manager support for 1.9 * Remove debug binary * adding debug to gitignore * minor doc fixes * Fix azure cni service ip (Azure#2237) * enabling azure cni * fix Azure CNI service IP connectivity * fix --auto-suffix when dnsPrefix is defined in apimodel json file (Azure#2239) * E2E: don’t collect logs if soak test (Azure#2240) * don’t collect logs if soak test * this! * Kubernetes 1.9.3 support (Azure#2242) * Add version 1.9.3 * update win zip and re-fmt * Kubernetes 1.8.8 support (Azure#2243) * add k8s 1.8.8 * updated win zip * rebase errata * more rebase errata * Update kuberneteswindowssetup.ps1 for azure cni to remove redundant code (Azure#2244) * enabling azure cni * remove redundant line * Windows RS3 hot fix for k8s (Azure#2230) * wait for certs to start etcd stuff in cloud init (Azure#2245) * set addon enabled value if nil (Azure#2254) * update generateproxycertscript.sh to use secure etcd endpoint/certs (Azure#2252) * enforce apt-get update warnings/errors retries (Azure#2241) * enforce apt-get update warnings/errors retries * Add single quotes around sp secret (Azure#2255) * --use-service-account-credentials=false if no rbac (Azure#2253) * new ubuntu image (Azure#2259) * Kubernetes Tiller Addon: configuration to set max-history (Azure#2217) * Add max-history configuration to tiller addon. * Test for max-history configuration for tiller addon. * freshen go-dev image (Azure#2261) * freshen go-dev image * lint * - keeping original DeploymentOperationsListResult in DeploymentError (Azure#2266) * - keeping original DeploymentOperationsListResult in DeploymentError - add DeploymentValidationError to distinguish validation errors * addressed comments * untangle —authorization-mode from enableSecureKubelet (Azure#2267) * untangle —authorization-mode from “secure kubelet” * fix typo * fix monitoring extension and add support for prometheus v2 (Azure#2257) This commit includes the following changes: - fixes the broken monitoring (prometheus/grafana) extension - makes this more resilient in the future, as the chart versions are now static (future to-do item would be to have extensionParameters override these versions) - gives the user and contributor more flexibility by allowing them to pass in a custom url for the prometheus chart values config (this is primarily important for developing and testing away from the Azure/acs-engine repo) * enable AggregatedAPI's by default for k8s 1.9.0+ (Azure#2264) * E2E test - 50 nodes (Azure#2260) * E2E: cleanup legacy kubernetes (Azure#2275) * add e2e hybrid definition also remove tiller explicit config from windows api model * removing windows + hybrid from legacy e2e * removing tests from legacy e2e that are elsewhere * add rescheduler, remove more from legacy e2e * add debug for service URL content mismatch * kubelet —cluster-domain is user-overridable (Azure#2276) * api/vlabs: fix typos in tests (Azure#2280) Signed-off-by: Jess Frazelle <[email protected]> * add prerequisit to have permissions to create service principals in the subscription (Azure#2281) acs-engine hangs with "WARN[0008] apimodel: ServicePrincipalProfile was empty, assigning role to application..." if user does not have enough permissions to create and assign service principals ans azure applications * E2E: service LB validations and pod Ready/NotReady (Azure#2279) * debug output if service URL validate error * debugging num retries * rearranging deck chairs * service validate should guarantee service IP * this actually works * improve pod Ready/NotReady check * general hpa foo (Azure#2291) * more time and avoid nil panic (Azure#2289) * New etcd versions and update default to v3.2.16 (Azure#2292) * new etcd versions and set default to 3.3.1 * using 3.2.16 as default * More e2e tests (Azure#2277) * add features on and features off * fix off model * add seperate tests for each feature disabled * move features off dir * rbac bool * added clear containers * added addons enabled test * fix typo in apimodel * remove aci-connector * move addons to default * Don't display "Error: <nil>" on successful deployment (Azure#2300) * E2E Addons (Azure#2294) * add features on and features off * fix off model * add seperate tests for each feature disabled * move features off dir * rbac bool * added clear containers * added addons enabled test * fix typo in apimodel * remove aci-connector * wip add mem/cpu limits/requests checks * add resources to container spec * fix resources type * add checks to tiller * remove extra err var * add check for dashboard and aci connector * update default definition * fmt * fix typo * Refactor resources validation * fix error string * fix linter * remove pointer * fix ineffassign * small fixes * ensure docker installs before ensure docker runs (Azure#2305) * Save apimodel after upgrade (Azure#2306) * cmd/deploy: Handle error due to missing permissions during deploy (Azure#2297) * Handle error due to missing permissions during deploy * CreateRoleAssignmentSimple can already return an error. Use this if a status 403 (not enough permissions) occurs. This is opposed to status 404 that seems to be issued to signal work in progress during service principal generation (by arm). * autoFillApimodel: remove the duplicated retry logic of CreateRoleAssignmentSimple. this allows to properly fail if CreateRoleAssignmentSimple returns an error * style fix: gofmt -s * Clarify that only Calico supports K8s network policies (Azure#2270) * using --cluster-domain for kube-dns domain (Azure#2303) * set kubelet defaults for --cgroups-per-qos & --enforce-node-allocatable (Azure#2310) * set kubelet defaults for --cgroups-per-qos & --enforce-node-allocatable * update docs * Updates NVIDIA drivers installation (Azure#2219) * updated NVIDIA drivers installation * linting engine.go * update GPU doc * Revert static IP allocation logic in Azure CNI, PR 1966. (Azure#2315) * add restarts to nvidia drivers download in cloud-init (Azure#2316) * add restarts to nvidia drivers download and only create cloud-init string if necessary * add tests * add v1.8 gpu-enabled api model for e2e testing * trying Standard_NC6 * e2e * lint * updated comment * bad match string, less freq checks, - unused func * more general success determination, typo * more typo * Support multiple AcsEngineClientIDs (Azure#2293) * Support multiple AcsEngineClientIDs * Fix acsEngineClientID assignment * Fix formatting azureclient.go * Fix2 formatting azureclient.go * docs cruft (Azure#2321) we are not actually setting —read-only-port=0 for kubelet * Use FirstConsecutiveStaticIP in original API model instead of resetting it to default during upgrade. * fix quotation in etcd daemon args (Azure#2325) * remove Windows + custom VNET validation error (Azure#2322) * Add isUpgrade flag. * Apply same logic to other routes setting FirstConsecutiveStaticIP. * Reboot etcd fix (Azure#2329) * fix quotation in etcd daemon args * Revert "fix quotation in etcd daemon args" This reverts commit 606bab4. * fix reboot by adding systemctl enable service * Remove agent NICs if upgrade master nodes. * Private clusters (Azure#2326) * add isprivatecluster func * wip remove load balancer for PC * add enablePrivateCluster flag * no public IPs for private cluster * working private cluster for 3 masters * remove duplicate iptables cmd * remove useless function * fmt * revert dnsprefix docs change * undo etcd change Move change to a separate PR because it is unrelated * remove masterPublicIpAddress * fix typo * handle DCOS + swarm nil case * add docs and example * replace host by jumpbox in the docs * add instructions to create jumpbox * indents * missing import (Azure#2348) * Allow "v" prefix in orchestrator version and release (Azure#2344) * fix quotation in etcd daemon args * Revert "fix quotation in etcd daemon args" This reverts commit 606bab4. * trim v in orch ver / rel * add unit tests * Improve the instructions for AAD. (Azure#2330) * Improve the instructions for AAD. * broken link and syntax * typo (Azure#2342) * Fix master resources merge conflict (Azure#2353) * apply azure CNI static IP revert * Improve info to get issuerurl (Azure#2356) * circleci: compile as separate step (Azure#2350) * Improve code blocks (Azure#2335) * Update Azure Gov ACSEngineClientID (Azure#2352) * add ClustrRole & ClusterRoleBinding for azure file (Azure#2238) add as cluster-service for azure-cloud-provider * mount /sbin/apparmor_parser if PodSecurityPolicy is enabled (Azure#2320) * mount /sbin/apparmor_parser if PSP * this is the correct kubelet service file * this is the correct sed command * /sbin/apparmor_parser already exists * Allow a default k8s version for loading agentpool-only clusters (Azure#2357) The defaultKubernetesVersion argument will be used if Properties.KubernetesVersion was empty. * Private clusters iteration 2: change the server for the cluster kubeconfig (Azure#2354) * modify 2nd kubeconfig for private clusters * typo * fix customscript kubeconfig * revert change in custom data kubeconfig * update docs for private clusters (Azure#2363) * Remove 1.6.x upgrade tests. (Azure#2364) * Add k8s 1.7.13 support (Azure#2369) * add version 1.7.13 * update win zip * Resolve merge conflict in building 1.7.13 (Azure#2370) * Resolve merge conflict in building 1.7.13 * Add comment * Set custom UbutuImageConfig for gov (Azure#2375) * Fix guid validation (Azure#2373) * metrics server addon (Azure#2339) * metrics server addon * use addonmanager mode EnsureExists * fix labels on metrics APIService * enable hpa autoscale test for 1.9 clusters * Reuse GetCloudTargetEnv in FormatAzureProdFQDN (Azure#2376) * reuse GetCloudTargetEnv in FormatAzureProdFQDN * Fix FQDNFormat lint error * minor fix in build script (Azure#2379) * rationalized vendor/ (Azure#2390) * remove unnecessary hyperkube reference (Azure#2391) * Remove deprecated '--require-kubeconfig' for k8s (Azure#2365) * Remove deprecated --require-kubeconfig * adding —require-kubeconfig back to 1.7j * less than is what we want here * Pass --location to containerService (Azure#2381) * Notes on day-to-day operations on an acs-engine cluster (Azure#2351) * Notes from my experiences.. .. over the last couple of days * Rename day-two-operations.md to kubernetes-day2-operations.md * add etcd certs to KV docs (Azure#2396) * upgrade tiller to 2.8.1 (Azure#2397) * remove k8s 1.5 related code/artifacts (Azure#2394) * blocking cse on cluster nodes ready (Azure#2225) * blocking cse on cluster nodes ready * deal with agent-only clusters * use kubectl var and ignore stderr * increase node active check timeout to 30 mins * test single master node Windows clusters (Azure#2402) * a miss * fix 2 more missed error * remove unnecessary * remove more unnecessary

CecileRobertMichon added 10 commits February 20, 2018 17:44

add isprivatecluster func

f511d93

wip remove load balancer for PC

c028a02

add enablePrivateCluster flag

c061eb7

no public IPs for private cluster

63f4dcf

working private cluster for 3 masters

edafd5a

remove duplicate iptables cmd

5f6cc91

remove useless function

65a1ea7

fmt

e9dfb37

revert dnsprefix docs change

e6240a7

undo etcd change

a446755

Move change to a separate PR because it is unrelated

ghost assigned CecileRobertMichon Feb 22, 2018

ghost added the in progress label Feb 22, 2018

CecileRobertMichon mentioned this pull request Feb 22, 2018

[WIP - Do not Merge] Feat - Clusters without public IPs #674

Closed

CecileRobertMichon changed the title ~~[WIP] Feat Private clusters~~ [WIP] Private clusters Feb 23, 2018

CecileRobertMichon added 2 commits February 22, 2018 17:33

remove masterPublicIpAddress

c381420

fix typo

1864fdb

CecileRobertMichon added 2 commits February 23, 2018 16:44

handle DCOS + swarm nil case

acde053

add docs and example

d9b3389

jackfrancis reviewed Feb 26, 2018

View reviewed changes

CecileRobertMichon added 3 commits February 26, 2018 14:02

replace host by jumpbox in the docs

550e65e

add instructions to create jumpbox

d91f3c2

indents

948e080

CecileRobertMichon changed the title ~~[WIP] Private clusters~~ Private clusters Feb 26, 2018

CecileRobertMichon added awaiting review and removed in progress labels Feb 27, 2018

jackfrancis approved these changes Feb 27, 2018

View reviewed changes

jackfrancis merged commit 1c3d6a7 into Azure:master Feb 27, 2018

ghost removed the awaiting review label Feb 27, 2018

CecileRobertMichon mentioned this pull request Feb 27, 2018

Fix master resources merge conflict #2353

Merged

CecileRobertMichon deleted the private-clusters branch March 15, 2018 19:02

khaldoune pushed a commit to khaldoune/acs-engine that referenced this pull request Mar 25, 2018

Revert "Private clusters (Azure#2326)"

20afbd2

This reverts commit 1c3d6a7.

khaldoune pushed a commit to khaldoune/acs-engine that referenced this pull request Mar 25, 2018

Revert "Private clusters (Azure#2326)"

cc79a4d

Revert private clusters

khaldoune pushed a commit to khaldoune/acs-engine that referenced this pull request Mar 26, 2018

Revert "Private clusters (Azure#2326)"

d8b4488

This reverts commit 1c3d6a7.

khaldoune pushed a commit to khaldoune/acs-engine that referenced this pull request Mar 26, 2018

Revert "Private clusters (Azure#2326)"

6cf424f

This reverts commit 1c3d6a7.

khaldoune pushed a commit to khaldoune/acs-engine that referenced this pull request Mar 26, 2018

Revert "Private clusters (Azure#2326)"

eaea922

This reverts commit 1c3d6a7.

benjamingorman mentioned this pull request Nov 5, 2018

Support private cluster option for kubernetes clusters hashicorp/terraform-provider-azurerm#2241

Closed

martell mentioned this pull request Mar 9, 2019

Prevent Creation of Public IP for Master VM when connected to Custom VNET #1895

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Private clusters #2326

Private clusters #2326

CecileRobertMichon commented Feb 22, 2018 •

edited

Loading

dennis-benzinger-hybris commented Feb 23, 2018

jackfrancis Feb 26, 2018

jackfrancis left a comment

yfried commented Feb 27, 2018

CecileRobertMichon commented Feb 27, 2018

SilverGaucho commented Mar 2, 2018

CecileRobertMichon commented Mar 2, 2018

sylus commented Mar 7, 2018

CecileRobertMichon commented Mar 7, 2018

sylus commented Mar 8, 2018

yfried commented Mar 13, 2018

sylus commented Mar 13, 2018

CecileRobertMichon commented Mar 13, 2018

jackfrancis commented Mar 14, 2018

Private clusters #2326

Private clusters #2326

Conversation

CecileRobertMichon commented Feb 22, 2018 • edited Loading

dennis-benzinger-hybris commented Feb 23, 2018

jackfrancis Feb 26, 2018

Choose a reason for hiding this comment

jackfrancis left a comment

Choose a reason for hiding this comment

yfried commented Feb 27, 2018

CecileRobertMichon commented Feb 27, 2018

SilverGaucho commented Mar 2, 2018

CecileRobertMichon commented Mar 2, 2018

sylus commented Mar 7, 2018

CecileRobertMichon commented Mar 7, 2018

sylus commented Mar 8, 2018

yfried commented Mar 13, 2018

sylus commented Mar 13, 2018

CecileRobertMichon commented Mar 13, 2018

jackfrancis commented Mar 14, 2018

CecileRobertMichon commented Feb 22, 2018 •

edited

Loading