{"id":46274,"date":"2017-02-22T16:59:10","date_gmt":"2017-02-22T11:29:10","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=46274"},"modified":"2017-02-22T16:59:10","modified_gmt":"2017-02-22T11:29:10","slug":"how-to-setup-kubernetes-master-ha-on-centos","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/how-to-setup-kubernetes-master-ha-on-centos\/","title":{"rendered":"How to Setup Kubernetes Master HA on CentOS?"},"content":{"rendered":"

\"blogha2\"<\/p>\n

This blog describes how to set up a high-availability (HA) Kubernetes cluster<\/a>.\u00a0This is an advanced topic and setting up a truly reliable, highly available distributed system requires few steps to be performed. We will go into each of these steps in detail, but a summary will help the user as a guide.<\/p>\n

Here’s what the system should look like when it’s finished:<\/p>\n

\"blogha\"<\/p>\n

Prerequisites:<\/strong> <\/span>You should have an existing Kubernetes Cluster on Bare Metal on CentOS. It means Master1 is already configured for the cluster but we need to make some changes on existing master to create ETCD cluster. Refer to this blog to learn how to setup a Kubernetes cluster on CentOS<\/a>.<\/p>\n

Assumptions:<\/strong> We are taking following assumptions for the blog:<\/p>\n

[js]Master1: 172.16.0.1<\/p>\n

Master2:172.16.0.2<\/p>\n

Master3: 172.16.0.3[\/js]<\/p>\n

Steps to configure Kubernetes Master HA<\/strong><\/p>\n

1. Install Kubernetes on Master2 and Master3:\u00a0<\/b>Create a new repo for Kubernetes using the following command:<\/p>\n

[js]vi \/etc\/yum.repos.d\/virt7-docker-common-release.repo<\/p>\n

[virt7-docker-common-release]<\/p>\n

name=virt7-docker-common-release
\nbaseurl=http:\/\/cbs.centos.org\/repos\/virt7-docker-common-release\/x86_64\/os\/
\ngpgcheck=0[\/js]<\/p>\n

Now Install Kubernetes:<\/p>\n

[js]yum -y install –enablerepo=virt7-docker-common-release kubernetes etcd flannel[\/js]<\/p>\n

The above command also installs Docker and cadvisor.<\/p>\n

2. Create ETCD Cluster using following etcd configuration:\u00a0<\/b>ETCD configuration file is located at \/etc\/etcd\/etcd.conf<\/p>\n

For Master1<\/b><\/p>\n

[js]# [member]<\/p>\n

ETCD_NAME=infra0<\/p>\n

ETCD_DATA_DIR="\/var\/lib\/etcd\/default.etcd"<\/p>\n

ETCD_LISTEN_PEER_URLS="http:\/\/172.16.0.1:2380"<\/p>\n

ETCD_LISTEN_CLIENT_URLS="http:\/\/172.16.0.1:2379.http:\/\/127.0.0.1:2379"<\/p>\n

#<\/p>\n

#[cluster]<\/p>\n

ETCD_INITIAL_CLUSTER="infra0=http:\/\/172.16.0.1:2380,infra1=http:\/\/172.16.0.2:2380,infra2=http:\/\/172.16.0.3:2380"<\/p>\n

ETCD_INITIAL_CLUSTER_STATE="new"<\/p>\n

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"<\/p>\n

ETCD_ADVERTISE_CLIENT_URLS="http:\/\/172.16.0.1:2379"<\/p>\n

[\/js]<\/p>\n

For Master2<\/b><\/p>\n

[js]# [member]<\/p>\n

ETCD_NAME=infra1<\/p>\n

ETCD_DATA_DIR="\/var\/lib\/etcd\/default.etcd"<\/p>\n

ETCD_LISTEN_PEER_URLS="http:\/\/172.16.0.2:2380"<\/p>\n

ETCD_LISTEN_CLIENT_URLS="http:\/\/172.16.0.2:2379.http:\/\/127.0.0.1:2379"<\/p>\n

#<\/p>\n

#[cluster]<\/p>\n

ETCD_INITIAL_CLUSTER="infra0=http:\/\/172.16.0.1:2380,infra1=http:\/\/172.16.0.2:2380,infra2=http:\/\/172.16.0.3:2380"<\/p>\n

ETCD_INITIAL_CLUSTER_STATE="new"<\/p>\n

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"<\/p>\n

ETCD_ADVERTISE_CLIENT_URLS="http:\/\/172.16.0.2:2379"<\/p>\n

[\/js]<\/p>\n

For Master3<\/b><\/p>\n

[js]# [member]<\/p>\n

ETCD_NAME=infra2<\/p>\n

ETCD_DATA_DIR="\/var\/lib\/etcd\/default.etcd"<\/p>\n

ETCD_LISTEN_PEER_URLS="http:\/\/172.16.0.3:2380"<\/p>\n

ETCD_LISTEN_CLIENT_URLS="http:\/\/172.16.0.3:2379,http:\/\/127.0.0.1:2379"<\/p>\n

#<\/p>\n

#[cluster]<\/p>\n

ETCD_INITIAL_CLUSTER="infra0=http:\/\/172.16.0.1:2380,infra1=http:\/\/172.16.0.2:2380,infra2=http:\/\/172.16.0.3:2380"<\/p>\n

ETCD_INITIAL_CLUSTER_STATE="new"<\/p>\n

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"<\/p>\n

ETCD_ADVERTISE_CLIENT_URLS="http:\/\/172.16.0.3:2379"<\/p>\n

Restart etcd in all the master nodes using following command:<\/p>\n

systemctl restart etcd<\/p>\n

Run the following command in any one of the etcd node to check if etcd cluster is formed properly:<\/p>\n

etcdctl cluster-health<\/p>\n

[\/js]<\/p>\n

3. Configure other Kubernetes master components:\u00a0<\/b>Let\u2019s start with Kubernetes common config<\/p>\n

On Master<\/b><\/p>\n

[js]vi \/etc\/kubernetes\/config<\/p>\n

# Comma separated list of nodes running etcd cluster
\nKUBE_ETCD_SERVERS="–etcd-servers=http:\/\/Master_Private_IP:2379"
\n# Logging will be stored in system journal
\nKUBE_LOGTOSTDERR="–logtostderr=true"
\n# Journal message level, 0 is debug
\nKUBE_LOG_LEVEL="–v=0"
\n# Should this cluster be allowed to run privileged docker containers
\nKUBE_ALLOW_PRIV="–allow-privileged=false"
\n# Api-server endpoint used in scheduler and controller-manager
\nKUBE_MASTER="–master=http:\/\/Master_Private_IP:8080"[\/js]<\/p>\n

On Minion<\/b><\/p>\n

[js]vi \/etc\/kubernetes\/config<\/p>\n

# Comma separated list of nodes running etcd cluster
\nKUBE_ETCD_SERVERS="–etcd-servers=http:\/\/k8_Master:2379"
\n# Logging will be stored in system journal
\nKUBE_LOGTOSTDERR="–logtostderr=true"
\n# Journal message level, 0 is debug
\nKUBE_LOG_LEVEL="–v=0"
\n# Should this cluster be allowed to run privileged docker containers
\nKUBE_ALLOW_PRIV="–allow-privileged=false"
\n# Api-server endpoint used in scheduler and controller-manager
\nKUBE_MASTER="–master=http:\/\/k8_Master:8080"<\/p>\n

[\/js]<\/p>\n

On Master: <\/b>API Server Configuration needs to be updated on master<\/span>\u00a0<\/strong><\/strong><\/p>\n

Copy all the certificates from existing master to the other two master(with same file permission). All the certificates are stored at <\/span>\/srv\/kubernetes\/<\/b>\u00a0<\/strong><\/strong><\/p>\n

Now, configure API server as follows:<\/span><\/p>\n

[js]vi \/etc\/kubernetes\/apiserver<\/p>\n

# Bind kube api server to this IP
\nKUBE_API_ADDRESS="–address=0.0.0.0"
\n# Port that kube api server listens to.
\nKUBE_API_PORT="–port=8080"
\n# Port kubelet listen on
\nKUBELET_PORT="–kubelet-port=10250"
\n# Address range to use for services(Work unit of Kubernetes)
\nKUBE_SERVICE_ADDRESSES="–service-cluster-ip-range=10.254.0.0\/16"
\n# Add your own!
\nKUBE_API_ARGS=" –client-ca-file=\/srv\/kubernetes\/ca.crt –tls-cert-file=\/srv\/kubernetes\/server.cert –tls-private-key-file=\/srv\/kubernetes\/server.key"<\/p>\n

[\/js]<\/p>\n

\u00a0<\/strong><\/strong>On Master:\u00a0<\/b>Configure Kubernetes Controller Manager<\/p>\n

[js]vi \/etc\/kubernetes\/controller-manager<\/p>\n

KUBE_CONTROLLER_MANAGER_ARGS="–root-ca-file=\/srv\/kubernetes\/ca.crt –service-account-private-key-file=\/srv\/kubernetes\/server.key"<\/p>\n

[\/js]<\/p>\n

On Master and Minion:\u00a0<\/b>Configure Flanneld<\/p>\n

[js]vi \/etc\/sysconfig\/flanneld<\/p>\n

# etcd url location. Point this to the server where etcd runs
\nFLANNEL_ETCD="http:\/\/k8_Master:2379"
\n# etcd config key. This is the configuration key that flannel queries
\n# For address range assignment
\nFLANNEL_ETCD_PREFIX="\/kube-centos\/network"
\n# Any additional options that you want to pass
\nFLANNEL_OPTIONS=""<\/p>\n

[\/js]<\/p>\n

Only one master must be active at any particular time so that the cluster remains in the consistent state. For this, we need to configure Kubernetes Controller Manager and Scheduler. Start these two services with –leader-elect<\/strong> option.<\/p>\n

Update their configuration file as follows:<\/span><\/p>\n

[js]vi \/etc\/kubernetes\/controller-manager<\/p>\n

KUBE_CONTROLLER_MANAGER_ARGS="–root-ca-file=\/srv\/kubernetes\/ca.crt –service-account-private-key-file=\/srv\/kubernetes\/server.key –leader-elect"<\/p>\n

vi \/etc\/kubernetes\/scheduler<\/p>\n

KUBE_SCHEDULER_ARGS="–leader-elect"<\/p>\n

[\/js]<\/p>\n

\u00a04.\u00a0<\/strong><\/strong>Create a Load Balancer(Internal) as follows:<\/b><\/p>\n

[js]
\n _____ Master1 Port 8080
\n |
\nLoad Balancer Port 8080 — —- Master2 Port 8080
\n |
\n —– Master3 Port 8080<\/p>\n

_____ Master1 Port 2379
\n |
\nLoad Balancer Port 2379 — —- Master2 Port 2379
\n |
\n —– Master3 Port 2379
\n[\/js]<\/p>\n

5.\u00a0<\/strong><\/strong>Now\u00a0<\/strong><\/strong>Replace Master IP in \/etc\/hosts<\/strong> of all <\/span>Minion<\/b> by IP address of Load Balancer and restart all kubernetes service in Minions.<\/span><\/p>\n

At this point, we are done with the master components. If we have an existing cluster, this is as simple as reconfiguring our kubelets to talk to the load-balanced endpoint, and restarting the kubelets on each node. If we are turning up a fresh cluster, we will need to install the kubelet and kube-proxy on each worker node, and set the –apiserver<\/strong> flag to our replicated endpoint.<\/p>\n","protected":false},"excerpt":{"rendered":"

This blog describes how to set up a high-availability (HA) Kubernetes cluster.\u00a0This is an advanced topic and setting up a truly reliable, highly available distributed system requires few steps to be performed. We will go into each of these steps in detail, but a summary will help the user as a guide. Here’s what the […]<\/p>\n","protected":false},"author":968,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":27},"categories":[4308,2348,1],"tags":[4374,1892,1883,3965,3984,4437,3979,4375],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/46274"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/968"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=46274"}],"version-history":[{"count":0,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/46274\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=46274"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=46274"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=46274"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}