
This post is Part 2 of a series on creating a custom Kubernetes cluter from scratch. As such, you will need to have finished the previous Part 1 to be able to move on.
Last time, we set up etcd, flannel, and docker to be our distributed configuration, network layer, and container tooling. Today, we'll be setting up the kubernetes' binaries: kube-apiserver
, kube-controller-manager
, kube-scheduler
, kube-proxy
, and kubelet
. With which we'll run the kubernetes dashboard inside using kubernetes itself (so meta!)
As with the previous post in this series, it's recommended that you have an OK understanding of:
- Linux, bash, general command line experience (
cd
,cp
,tar
,mv
,mkdir
, etc) - Networking, specifically CIDR notation, network interfaces, and basic understanding of firewalls and maybe port mapping too
- Docker
- Ruby, very basic familiarity with the programming language, specifically basic syntax and condition flow
Quick recap of the versions we'll be working with:
- Vagrant 1.8.5
- Ubuntu 14.04
- etcd 3.0.1
- flanneld 0.5.5
- Docker 1.11.2
- kubernetes 1.3.0
Note: I'm sure there will eventually be newer versions so feel free to use updated binaries of each, just be aware of breaking changes.
Okay, lets get started
From last time, we should have a three machine vagrant setup that has etcd and flannel running with docker using flannel's network as its bridge interface. You can follow along using the same Vagrantfile
or clone the repository:
git clone https://github.com/mihok/kubernetes-scratch --branch part.1
Note: I always like to start from a blank slate so if you already have those machines provisioned from last time you'll want to vagrant destroy
before we provision again
Okay, our Vagrantfile
should look something like this:
# -*- mode: ruby -*-
# vi: set ft=ruby :
$instances = 3
$instance_name_prefix = "app"
$app_cpus = 1
$app_memory = 1024
Vagrant.configure(2) do |config|
config.vm.box = "ubuntu/trusty64"
(1..$instances).each do |i|
config.vm.define vm_name = "%s-%02d" % [$instance_name_prefix, i] do |config|
config.vm.hostname = vm_name
# Set the VM memory and cpu allocations
config.vm.provider :virtualbox do |vb|
vb.memory = $app_memory
vb.cpus = $app_cpus
end
# Create a private network, which allows host-only access to the machine
# using a specific IP.
ip = "44.0.0.#{i+100}"
config.vm.network :private_network, ip: ip
# Section (A) -- etcd
# First machine gets a new state, and is the first in our cluster list
state = "new"
cluster = "app-01=http:\\/\\/44.0.0.101:2380"
if i > 1
# All other machines get an existing state since they're set up sequentially
state = "existing"
# Add each additional machine to our cluster list
(2..i).each do |j|
cluster = "#{cluster},app-0#{j}=http:\\/\\/44.0.0.#{j+100}:2380"
end
end
# The actual vagrant provision call
config.vm.provision "shell", path: "etcd.sh", name: "etcd", env: {"IP" => ip, "CLUSTER_STATE" => state, "CLUSTER" => cluster}
# Section (B) -- flannel
# Provision flannel binaries and services
config.vm.provision "shell", path: "flanneld.sh", name: "flannel"
if i == 1
# Create our flannel configuration in etcd
config.vm.provision "shell", name: "flannel-config", inline: "etcdctl mkdir /network; etcdctl mk /network/config </vagrant/flanneld.json"
end
# Start flannel
config.vm.provision "shell", name: "flannel", inline: "start flanneld"
# Add the next node if we aren't the last node
if $instances > 1 && i < $instances
config.vm.provision "shell", name: "etcd-add", inline: "etcdctl member add app-0#{i+1} http://44.0.0.#{i+101}:2380"
end
# Section (C) -- docker
config.vm.provision "docker"
config.vm.provision "shell", name: "docker", path: "docker.sh"
# Section (D) -- kubernetes (NEW)
end
end
end
Note the addition of section (D)
, which will be our focus for today.
Lets go over the different binaries that come with kubernetes. Each one has a different function, but only the kublet
and kube-proxy
are required on each machine. Leave kube-scheduler
, kube-controller-manager
, and kube-apiserver
for our master node.
kubelet
The kubelet
is our bread and butter. If you want a machine to be orchestrated by kubernetes, you will need to run the kubelet
. From kubernetes:
The kubelet is the primary “node agent” that runs on each node. The kubelet works in terms of a PodSpec. A PodSpec is a YAML or JSON object that describes a pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms (primarily through the apiserver) and ensures that the containers described in those PodSpecs are running and healthy.
http://kubernetes.io/docs/admin/kubelet/
kube-proxy
Next, is the kube-proxy
, this little guy's sole purpose is to facilitate networking between the Pods/Services and each node:
The Kubernetes network proxy runs on each node. This reflects services as defined in the Kubernetes API on each node and can do simple TCP,UDP stream forwarding or round robin TCP,UDP forwarding across a set of backends.
http://kubernetes.io/docs/admin/kube-proxy/
kube-scheduler
The kube-scheduler
is the mechanism that orchestrates what Pods go where:
The Kubernetes scheduler is a policy-rich, topology-aware, workload-specific function that significantly impacts availability, performance, and capacity. The scheduler needs to take into account individual and collective resource requirements, quality of service requirements, hardware/software/policy constraints, affinity and anti-affinity specifications, data locality, inter-workload interference, deadlines, and so on.
http://kubernetes.io/docs/admin/kube-scheduler/
kube-controller-manager
The kube-controller-manager
is the ever relenting guard of our infrastructure. Making sure we're in a state that is always desired, specifically:
The Kubernetes controller manager is a daemon that embeds the core control loops shipped with Kubernetes... In Kubernetes, a controller is a control loop that watches the shared state of the cluster through the apiserver and makes changes attempting to move the current state towards the desired state
http://kubernetes.io/docs/admin/kube-controller-manager/
kube-apiserver
Last, but not least we have our kube-apiserver
. The apisever is exactly what you think it is, an API daemon that we'll use to interact with our kubernetes infrastructure:
The API Server services REST operations and provides the frontend to the cluster’s shared state through which all other components interact.
http://kubernetes.io/docs/admin/kube-apiserver/
Each of these components makes up the kubernetes infrastructure (along with etcd, flannel, and docker too.) Similar to what we did in Part 1, we'll install kubelet
and kube-proxy
on each vagrant machine and leave the other binaries for the last machine that gets created.
In section (D)
we'll create a shell provision for kubernetes as a whole, and then one for each binary:
config.vm.provision "shell", name: "kubernetes", path: "kubernetes.sh"
config.vm.provision "shell", name: "kubernetes", path: "kubelet.sh"
config.vm.provision "shell", name: "kubernetes", path: "kube-proxy.sh"
# Only provision these if we're the last node
if i == $instances
config.vm.provision "shell", name: "kubernetes", path: "kube-apiserver.sh"
config.vm.provision "shell", name: "kubernetes", path: "kube-controller-manager.sh"
config.vm.provision "shell", name: "kubernetes", path: "kube-scheduler.sh"
end
The first shell script will download the binaries, and the following scripts will make them accessible from anywhere. They will also install our Upstart definitions as well. I've made quite a big gist for these files here, but they'll also are available in the repo which is tagged as part.2
.
In addition, I've also included the dashboard.yaml
file in the gist too, we'll use this in the next section once the Vagrant machines are up and running.
OK, vagrant up
and it should provision everything~
Kubernetes Dashboard
Ah the fun part :)
Once you've watched our cluster get built, confirm that:
kubelet
andkube-proxy
are running on each nodeetcd
,flannel
, anddocker
should be running too.kube-scheduler
,kube-controller-manager
, andkube-apiserver
are running on app-03 (our master node.)
You can confirm this by vagrant ssh
into each machine and running:
ps -e -o pid,cmd | grep --color -E 'etcd|flannel|docker|kube'
You'll see something like this on app-01, and app-02:
[email protected]:~$ ps -e -o pid,cmd | grep --color -E 'etcd|flannel|docker|kube'
9190 etcd
9250 flanneld
20746 /usr/bin/dockerd --bip=44.1.86.1/24 --mtu=1472 --raw-logs
20756 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc
20899 kubelet --api-servers=44.0.0.103:8888 --experimental-flannel-overlay=true --logtostderr=true
20926 kube-proxy --master=http://44.0.0.103:8888 --proxy-mode=iptables --logtostderr=true
21087 grep --color=auto --color -E etcd|flannel|docker|kube
Or on app-03 (our master node):
[email protected]:~$ ps -e -o pid,cmd | grep --color -E 'etcd|flannel|docker|kube'
9105 etcd
9164 flanneld
20634 /usr/bin/dockerd --bip=44.1.58.1/24 --mtu=1472 --raw-logs
20643 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc
20861 kubelet --api-servers=44.0.0.103:8888 --experimental-flannel-overlay=true --logtostderr=true
20887 kube-proxy --master=http://44.0.0.103:8888 --proxy-mode=iptables --logtostderr=true
20928 kube-apiserver --advertise-address=44.0.0.103 --storage-backend=etcd3 --service-cluster-ip-range=107.0.0.0/16 --logtostderr=true --etcd-servers=http://127.0.0.1:2379 --insecure-bind-address=44.0.0.103 --insecure-port=8888 --kubelet-https=false
20964 kube-controller-manager --cluster-cidr=107.0.0.0/16 --cluster-name=vagrant --master=http://44.0.0.103:8888 --port=8890 --service-cluster-ip-range=107.0.0.0/16 --logtostderr=true
20991 kube-scheduler --master=http://44.0.0.103:8888 --logtostderr=true
21026 grep --color=auto --color -E etcd|flannel|docker|kube
Provided your machines output similiar results, kubernetes should be running! We can spin up the dashboard using kubectl
. To do this we'll run the following command:
kubectl create -s 44.0.0.103:8888 -f /vagrant/dashboard.yaml
Thats it! Give kubernetes a couple minutes to schedule and start the dashboard Pod, and then you should be able to navigate to http://44.0.0.103:8888/ui/. It will redirected you to the dashboard and look something like this:

Next post in this series will go through using our kubernetes cluster to setup a simple highly available react/redux application!
*Bonus points from this post, if you can:*
- Pre-download kubernetes binaries and modify the
Vagrantfile
to share them with each machine so that we don't have to download the release on each machine, separately. - Install the
.kubeconfig
bonus file into each machine manually or modifying theVagrantfile
to remove the need to append-s 44.0.0.103:8888
to eachkubectl
call.