Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Able to quick create a HA cluster by kube-up.sh centos provider #39462

Merged
merged 1 commit into from
Jan 26, 2017

Conversation

Shawyeok
Copy link
Contributor

@Shawyeok Shawyeok commented Jan 5, 2017

Make kube-up.sh centos provider support quick create a HA cluster, as I said above #39430, it's more flexible than kops or kubeadm for some people in a limited network region.

I'm new to k8s dev, so if this pull request need to change, please let me know.

Added support for creating HA clusters for centos using kube-up.sh.

@k8s-ci-robot
Copy link
Contributor

Hi @Shawyeok. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with @k8s-bot ok to test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot
Copy link
Contributor

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://github.com/kubernetes/kubernetes/wiki/CLA-FAQ to sign the CLA.

Once you've signed, please reply here (e.g. "I signed it!") and we'll verify. Thanks.


If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Jan 5, 2017
@k8s-reviewable
Copy link

This change is Reviewable

@k8s-github-robot k8s-github-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Jan 5, 2017
@Shawyeok
Copy link
Contributor Author

Shawyeok commented Jan 5, 2017

I signed it!

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jan 5, 2017
@Shawyeok
Copy link
Contributor Author

Shawyeok commented Jan 5, 2017

@xiaoping378 Would you mind take a look about this PR?

@foxish
Copy link
Contributor

foxish commented Jan 5, 2017

/cc @kubernetes/sig-cluster-lifecycle-misc

@roberthbailey
Copy link
Contributor

It looks like your PR only affect the centos provider. If that was the intent, please update the PR description and the release note to make it clear to reviewers and users. Thanks!

@Shawyeok Shawyeok changed the title make kube-up.sh support quick create a HA cluster Able to quick create a HA cluster by kube-up.sh centos provider Jan 5, 2017
@Shawyeok
Copy link
Contributor Author

Shawyeok commented Jan 5, 2017

@roberthbailey OK, I've changed that. Maybe I should add support for ubuntu provider and others later.

@roberthbailey
Copy link
Contributor

@Shawyeok - doing it in multiple PRs sounds like the best path forward.

@Shawyeok
Copy link
Contributor Author

Shawyeok commented Jan 5, 2017

@roberthbailey Yeah, I think so. If a PR is too large, it's hard to review or find bug in it.

@Shawyeok
Copy link
Contributor Author

Shawyeok commented Jan 6, 2017

@roberthbailey @zmerlynn I added two more commits to fix a problem. What should I do to make this PR move forward?

@Shawyeok
Copy link
Contributor Author

Shawyeok commented Jan 9, 2017

/lgtm

@k8s-ci-robot
Copy link
Contributor

@Shawyeok: you can't LGTM your own PR.

In response to this comment:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@Shawyeok
Copy link
Contributor Author

Shawyeok commented Jan 9, 2017

@mikedanese @jbeda @eparis Would you mind take a look about this PR?

@roberthbailey
Copy link
Contributor

Please squash your commits.

@Shawyeok
Copy link
Contributor Author

Shawyeok commented Jan 9, 2017

@roberthbailey OK, I got squashed my commits.

@jszczepkowski
Copy link
Contributor

@Shawyeok
How did you check this PR? Is it working?

@Shawyeok
Copy link
Contributor Author

Shawyeok commented Jan 9, 2017

@jszczepkowski I'm check the PR for quick create a HA cluster using centos provider. e.g. code below will crate a HA k8s cluster (replicated api servers, etcd cluster, leader elect)

#!/bin/bash

# file: cluster/env.sh
# define your master machines in comma separate: <user0>@<ip0> <user1>@<ip1> <user2>@<ip2>
export MASTERS="[email protected] [email protected] [email protected]"
# define your node machines in comma separate: <user0>@<ip0> <user1>@<ip1> <user2>@<ip2>
export NODES="[email protected] [email protected]"

export KUBERNETES_PROVIDER=centos

export CERT_GROUP=ec2-user

# ssh-agent
eval `ssh-agent -s` && ssh-add ~/.ssh/<my-private-key>.pem
$ cd cluster
$ ./kube-up.sh

As I said above #39430 , kubeadm and kops doesn't work for a limited network region, but kube-xxx.sh is more flexible, we just need to prepare machines(don't care about gce or aws or azure, even in a local network.)

@justinsb
Copy link
Member

justinsb commented Jan 9, 2017

We would like to make kops work (of course), and I'm sure team-kubeadm would love to make kubeadm work also. For kops, it's Route53, right? If you have time it would be great to know if the following are supported in AWS cn-north:

  • Route53 Public Zones (I think no)
  • Route53 Private Zones (?)
  • Internet-Facing ELBs
  • Internal ELBs

@Shawyeok
Copy link
Contributor Author

Shawyeok commented Jan 9, 2017

@justinsb Route53 doesn't serve on aws cn-north region, and ELB works well. But as I knew, route53 is requirement of kops. I'm looking forward to kops and kubeadm works well in china region and don't need too many hack. Thanks very much for your services.


# By default, etcd cluster will use etcd-discovery
# https://github.com/coreos/etcd/blob/master/Documentation/op-guide/clustering.md#etcd-discovery
export NUM_ETCD_NODES=${NUM_MASTERS}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variables should be in the following format: "${VALUE}"
So, here you should have: NUM_ETCD_NODES="${NUM_MASTERS}"
Please, update also other places.

@zmerlynn
Copy link
Member

@Shawyeok: Run hack/verify-flags-underscore.py -e > hack/verify-flags/exceptions.txt and commit the result.

@k8s-github-robot k8s-github-robot removed lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jan 25, 2017
@zmerlynn
Copy link
Member

/approve
/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 25, 2017
@k8s-github-robot k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 25, 2017
Fix: cannot get default master advertise address correctly

Set default value of NUM_MASTERS and NUM_NODES by MASTERS and NODES themself

Code cleanup and documented

Using runtime reconfiguration for etcd cluster instead of etcd discovery

Add exceptions for verify-flags
@k8s-github-robot k8s-github-robot removed lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jan 25, 2017
@Shawyeok
Copy link
Contributor Author

@zmerlynn I've missed run hack/verify-flags-underscore.py -e > hack/verify-flags/exceptions.txt again after changed cluster/centos/config-default.sh.

Change of cluster/centos/config-default.sh:

-etcd_servers="${prefix}http://$master_ip:2379"
+etcd_servers="${prefix}http://${master_ip}:2379"

Sorry for delay.

@Shawyeok
Copy link
Contributor Author

@zmerlynn Now all checks were passed, need lgtm and approved again.

@zmerlynn
Copy link
Member

/approve
/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 25, 2017
@k8s-github-robot k8s-github-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jan 25, 2017
@k8s-github-robot
Copy link

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

@k8s-github-robot
Copy link

Automatic merge from submit-queue

@k8s-github-robot k8s-github-robot merged commit 75c6990 into kubernetes:master Jan 26, 2017
@strugglingyouth
Copy link

@Shawyeok

I see the official said 1.5 version integrated HA, but it seems only in gce use, we use the centos, so I see your PR, So I tested it,download the k8s 1.5.2 source, I will cluster/centos directory file replacement for you to modify, and then use kube-up to start the test, but failed, the following is my steps:

And first,config the cluster/centos/config-default.sh file.Then,Manually download kubernetes-client-linux-amd64.tar.gz kubernetes-server-linux-amd64.tar.gz easy-rsa.tar.gz packages and placed in the specified directory.

export KUBERNETES_PROVIDER=centos

bash -x ./kube-up.sh

>  .........
>
>  + local host=centos@xxxx
>  + shift
>  + ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oLogLevel=ERROR -C -t centos@xxx 'mkdir -p ~/kube_temp;                    sudo mkdir -p /opt/kubernetes/bin;                    sudo mkdir -p /opt/kubernetes/cfg'
>  + kube-scp centos@xxx './../cluster/../cluster/centos/ca-cert ./../cluster/../cluster/centos/binaries/master ./../cluster/../cluster/centos/master ./../cluster/../cluster/centos/config-default.sh ./../cluster/../cluster/centos/util.sh' '~/kube_temp'
>  + local host=centos@xxx
>  + src=($2)
>  + local src
>  + local 'dst=~/kube_temp'
>  + scp -r -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oLogLevel=ERROR -C ./../cluster/../cluster/centos/ca-cert ./../cluster/../cluster/centos/binaries/master ./../cluster/../cluster/centos/master ./../cluster/../cluster/centos/config-default.sh ./../cluster/../cluster/centos/util.sh 'centos@xxxxx:~/kube_temp'
>
>  ./../cluster/../cluster/centos/ca-cert: No such file or directory
>
>  scheduler.sh                                                              100% 1601     1.6KB/s   00:00
>  controller-manager.sh                                                     100% 2126     2.1KB/s   00:00
>  apiserver.sh                                                              100% 4439     4.3KB/s   00:00
>  etcd.sh                                                                   100% 2219     2.2KB/s   00:00
>  apiserver.sh                                                              100% 4424     4.3KB/s   00:00
>  controller-manager.sh                                                     100% 2237     2.2KB/s   00:00
>  etcd.sh                                                                   100% 2299     2.3KB/s   00:00
>  flannel.sh                                                                100% 1758     1.7KB/s   00:00
>  post-etcd.sh                                                              100%  853     0.8KB/s   00:00
>  scheduler.sh                                                              100% 1708     1.7KB/s   00:00
>  config-default.sh                                                         100% 4753     4.6KB/s   00:00
>  util.sh                                                                   100%   11KB  10.5KB/s   00:00
>  + kill 10478
>

And then it's over.

I would like to ask you this installation method is correct?

@Shawyeok
Copy link
Contributor Author

The problem is cluster/centos/ca-cert: No such file or directory, the certificates are required for kube-apiserver.

@strugglingyouth
Copy link

@Shawyeok
Now there is no configuration is completed, has been stuck in the service start process.

+ scp -r -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oLogLevel=ERROR -C ./cluster/../cluster/../cluster/centos/binaries/node ./cluster/../cluster/../cluster/centos/node ./cluster/../cluster/../cluster/centos/config-default.sh ./cluster/../cluster/../cluster/centos/util.sh 'centos@xxxx:~/kube_temp'
flannel.sh                                                                         100% 1881     1.8KB/s   00:00
proxy.sh                                                                           100% 1717     1.7KB/s   00:00
kubelet.sh                                                                         100% 2455     2.4KB/s   00:00
docker.sh                                                                          100% 1405     1.4KB/s   00:00
remove-docker0.sh                                                                  100%  850     0.8KB/s   00:00
mk-docker-opts.sh                                                                  100% 2590     2.5KB/s   00:00
mk-docker-opts.sh                                                                  100% 2590     2.5KB/s   00:00
remove-docker0.sh                                                                  100%  850     0.8KB/s   00:00
docker.sh                                                                          100% 1402     1.4KB/s   00:00
flannel.sh                                                                         100% 1881     1.8KB/s   00:00
kubelet.sh                                                                         100% 2737     2.7KB/s   00:00
proxy.sh                                                                           100% 1719     1.7KB/s   00:00
config-default.sh                                                                  100% 4753     4.6KB/s   00:00
util.sh                                                                            100%   11KB  10.5KB/s   00:00
+ kube-ssh centos@xxxx '     rm -rf /opt/kubernetes/bin;     sudo cp -r ~/kube_temp/node/bin /opt/kubernetes;     sudo chmod -R +x /opt/kubernetes/bin;     sudo ln -s /opt/kubernetes/bin/* /usr/local/bin/;     sudo bash ~/kube_temp/node/scripts/flannel.sh http://xxxx:2379 172.16.0.0/16;     sudo bash ~/kube_temp/node/scripts/docker.sh "";     sudo bash ~/kube_temp/node/scripts/kubelet.sh xxxx xxx 192.168.3.100 cluster.local;     sudo bash ~/kube_temp/node/scripts/proxy.sh xxxx'
+ local host=centos@xxxx
+ shift
+ ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oLogLevel=ERROR -C -t centos@xxxx '     rm -rf /opt/kubernetes/bin;     sudo cp -r ~/kube_temp/node/bin /opt/kubernetes;     sudo chmod -R +x /opt/kubernetes/bin;     sudo ln -s /opt/kubernetes/bin/* /usr/local/bin/;     sudo bash ~/kube_temp/node/scripts/flannel.sh http://xxxx:2379 172.16.0.0/16;     sudo bash ~/kube_temp/node/scripts/docker.sh "";     sudo bash ~/kube_temp/node/scripts/kubelet.sh xxxx xxxx 192.168.3.100 cluster.local;     sudo bash ~/kube_temp/node/scripts/proxy.sh xxxx'

And stop in here.I found that there is a problem with the docker start.Can you leave a contact, like qq, or a tutorial for my reference.

@Shawyeok
Copy link
Contributor Author

Would you mind provide some logs of docker service?

journalctl -xf -u docker

@Shawyeok
Copy link
Contributor Author

Shawyeok commented Feb 22, 2017

There is a docker dependencies problems in centos:

error while loading shared libraries: libltdl.so.7: cannot open shared object file: No such file or directory

It cause by a runtime lib which docker depend is missing. You can resolve it as below:

yum provides libltdl.so.7
yum install -y libtool-ltdl

@strugglingyouth
Copy link

@Shawyeok
Although it is not successful, but I would like to know what you are the principle of how to do the handover between master and elections, the trouble to talk briefly, thank you very much.

@Shawyeok
Copy link
Contributor Author

Actually, I don't understand your question. If you want to know about the principle of master-elected, you should take a look about raft.

k8s-github-robot pushed a commit that referenced this pull request Mar 28, 2017
Automatic merge from submit-queue

Centos provider: generate SSL certificates for etcd cluster.

**What this PR does / why we need it**:
Support secure etcd cluster for centos provider, generate SSL certificates for etcd in default. Running it w/o SSL is exposing cluster data to everyone and is not recommended. [#39462](#39462 (comment))

/cc @jszczepkowski @zmerlynn 

**Release note**:
```release-note
Support secure etcd cluster for centos provider.
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.