This post is Part 2 of a series on creating a custom Kubernetes cluter from scratch. As such, you will need to have finished the previous Part 1 to be able to move on.

Last time, we set up etcd, flannel, and docker to be our distributed configuration, network layer, and container tooling. Today, we'll be setting up the kubernetes' binaries: kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy, and kubelet. With which we'll run the kubernetes dashboard inside using kubernetes itself (so meta!)

As with the previous post in this series, it's recommended that you have an OK understanding of:

• Linux, bash, general command line experience (cd, cp, tar, mv, mkdir, etc)
• Networking, specifically CIDR notation, network interfaces, and basic understanding of firewalls and maybe port mapping too
• Docker
• Ruby, very basic familiarity with the programming language, specifically basic syntax and condition flow

Quick recap of the versions we'll be working with:

• Vagrant 1.8.5
• Ubuntu 14.04
• etcd 3.0.1
• flanneld 0.5.5
• Docker 1.11.2
• kubernetes 1.3.0

Note: I'm sure there will eventually be newer versions so feel free to use updated binaries of each, just be aware of breaking changes.

Okay, lets get started

From last time, we should have a three machine vagrant setup that has etcd and flannel running with docker using flannel's network as its bridge interface. You can follow along using the same Vagrantfile or clone the repository:

git clone https://github.com/mihok/kubernetes-scratch --branch part.1

Note: I always like to start from a blank slate so if you already have those machines provisioned from last time you'll want to vagrant destroy before we provision again

Okay, our Vagrantfile should look something like this:

# -*- mode: ruby -*-
# vi: set ft=ruby :

$instances = 3
$instance_name_prefix = "app"

$app_cpus = 1
$app_memory = 1024

Vagrant.configure(2) do |config|
  config.vm.box = "ubuntu/trusty64"

  (1..$instances).each do |i|
    config.vm.define vm_name = "%s-%02d" % [$instance_name_prefix, i] do |config|
      config.vm.hostname = vm_name

      # Set the VM memory and cpu allocations
      config.vm.provider :virtualbox do |vb|
        vb.memory = $app_memory
        vb.cpus = $app_cpus
      end

      # Create a private network, which allows host-only access to the machine
      # using a specific IP.
      ip = "44.0.0.#{i+100}"
      config.vm.network :private_network, ip: ip

      # Section (A) -- etcd

      # First machine gets a new state, and is the first in our cluster list
      state = "new"
      cluster = "app-01=http:\\/\\/44.0.0.101:2380"
      if i > 1
        # All other machines get an existing state since they're set up sequentially
        state = "existing"

        # Add each additional machine to our cluster list
        (2..i).each do |j|
          cluster = "#{cluster},app-0#{j}=http:\\/\\/44.0.0.#{j+100}:2380"
        end
      end

      # The actual vagrant provision call
      config.vm.provision "shell", path: "etcd.sh", name: "etcd", env: {"IP" => ip, "CLUSTER_STATE" => state, "CLUSTER" => cluster}

      # Section (B) -- flannel

      # Provision flannel binaries and services
      config.vm.provision "shell", path: "flanneld.sh", name: "flannel"

      if i == 1
        # Create our flannel configuration in etcd
        config.vm.provision "shell", name: "flannel-config", inline: "etcdctl mkdir /network; etcdctl mk /network/config </vagrant/flanneld.json"
      end

      # Start flannel
      config.vm.provision "shell", name: "flannel", inline: "start flanneld"

      # Add the next node if we aren't the last node
      if $instances > 1 && i < $instances
        config.vm.provision "shell", name: "etcd-add", inline: "etcdctl member add app-0#{i+1} http://44.0.0.#{i+101}:2380"
      end

      # Section (C) -- docker

      config.vm.provision "docker"

      config.vm.provision "shell", name: "docker", path: "docker.sh"

      # Section (D) -- kubernetes (NEW)

    end
  end
end

Note the addition of section (D), which will be our focus for today.

Lets go over the different binaries that come with kubernetes. Each one has a different function, but only the kublet and kube-proxy are required on each machine. Leave kube-scheduler, kube-controller-manager, and kube-apiserver for our master node.

kubelet

The kubelet is our bread and butter. If you want a machine to be orchestrated by kubernetes, you will need to run the kubelet. From kubernetes:

The kubelet is the primary “node agent” that runs on each node. The kubelet works in terms of a PodSpec. A PodSpec is a YAML or JSON object that describes a pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms (primarily through the apiserver) and ensures that the containers described in those PodSpecs are running and healthy.

http://kubernetes.io/docs/admin/kubelet/

kube-proxy

Next, is the kube-proxy, this little guy's sole purpose is to facilitate networking between the Pods/Services and each node:

The Kubernetes network proxy runs on each node. This reflects services as defined in the Kubernetes API on each node and can do simple TCP,UDP stream forwarding or round robin TCP,UDP forwarding across a set of backends.

http://kubernetes.io/docs/admin/kube-proxy/

kube-scheduler

The kube-scheduler is the mechanism that orchestrates what Pods go where:

The Kubernetes scheduler is a policy-rich, topology-aware, workload-specific function that significantly impacts availability, performance, and capacity. The scheduler needs to take into account individual and collective resource requirements, quality of service requirements, hardware/software/policy constraints, affinity and anti-affinity specifications, data locality, inter-workload interference, deadlines, and so on.

http://kubernetes.io/docs/admin/kube-scheduler/

kube-controller-manager

The kube-controller-manager is the ever relenting guard of our infrastructure. Making sure we're in a state that is always desired, specifically:

The Kubernetes controller manager is a daemon that embeds the core control loops shipped with Kubernetes... In Kubernetes, a controller is a control loop that watches the shared state of the cluster through the apiserver and makes changes attempting to move the current state towards the desired state

http://kubernetes.io/docs/admin/kube-controller-manager/

kube-apiserver

Last, but not least we have our kube-apiserver. The apisever is exactly what you think it is, an API daemon that we'll use to interact with our kubernetes infrastructure:

The API Server services REST operations and provides the frontend to the cluster’s shared state through which all other components interact.

http://kubernetes.io/docs/admin/kube-apiserver/

Each of these components makes up the kubernetes infrastructure (along with etcd, flannel, and docker too.) Similar to what we did in Part 1, we'll install kubelet and kube-proxy on each vagrant machine and leave the other binaries for the last machine that gets created.

In section (D) we'll create a shell provision for kubernetes as a whole, and then one for each binary:

      config.vm.provision "shell", name: "kubernetes", path: "kubernetes.sh"

      config.vm.provision "shell", name: "kubernetes", path: "kubelet.sh"
      config.vm.provision "shell", name: "kubernetes", path: "kube-proxy.sh"

      # Only provision these if we're the last node
      if i == $instances
        config.vm.provision "shell", name: "kubernetes", path: "kube-apiserver.sh"
        config.vm.provision "shell", name: "kubernetes", path: "kube-controller-manager.sh"
        config.vm.provision "shell", name: "kubernetes", path: "kube-scheduler.sh"
      end

The first shell script will download the binaries, and the following scripts will make them accessible from anywhere. They will also install our Upstart definitions as well. I've made quite a big gist for these files here, but they'll also are available in the repo which is tagged as part.2.

In addition, I've also included the dashboard.yaml file in the gist too, we'll use this in the next section once the Vagrant machines are up and running.

OK, vagrant up and it should provision everything~

Kubernetes Dashboard

Ah the fun part :)

Once you've watched our cluster get built, confirm that:

• kubelet and kube-proxy are running on each node
• etcd, flannel, and docker should be running too.
• kube-scheduler, kube-controller-manager, and kube-apiserver are running on app-03 (our master node.)

You can confirm this by vagrant ssh into each machine and running:

ps -e -o pid,cmd | grep --color -E 'etcd|flannel|docker|kube'

You'll see something like this on app-01, and app-02:

[email protected]:~$ ps -e -o pid,cmd | grep --color -E 'etcd|flannel|docker|kube'
 9190 etcd
 9250 flanneld
20746 /usr/bin/dockerd --bip=44.1.86.1/24 --mtu=1472 --raw-logs
20756 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc
20899 kubelet --api-servers=44.0.0.103:8888 --experimental-flannel-overlay=true --logtostderr=true
20926 kube-proxy --master=http://44.0.0.103:8888 --proxy-mode=iptables --logtostderr=true
21087 grep --color=auto --color -E etcd|flannel|docker|kube

Or on app-03 (our master node):

[email protected]:~$ ps -e -o pid,cmd | grep --color -E 'etcd|flannel|docker|kube'  
 9105 etcd
 9164 flanneld
20634 /usr/bin/dockerd --bip=44.1.58.1/24 --mtu=1472 --raw-logs
20643 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc
20861 kubelet --api-servers=44.0.0.103:8888 --experimental-flannel-overlay=true --logtostderr=true
20887 kube-proxy --master=http://44.0.0.103:8888 --proxy-mode=iptables --logtostderr=true
20928 kube-apiserver --advertise-address=44.0.0.103 --storage-backend=etcd3 --service-cluster-ip-range=107.0.0.0/16 --logtostderr=true --etcd-servers=http://127.0.0.1:2379 --insecure-bind-address=44.0.0.103 --insecure-port=8888 --kubelet-https=false
20964 kube-controller-manager --cluster-cidr=107.0.0.0/16 --cluster-name=vagrant --master=http://44.0.0.103:8888 --port=8890 --service-cluster-ip-range=107.0.0.0/16 --logtostderr=true
20991 kube-scheduler --master=http://44.0.0.103:8888 --logtostderr=true
21026 grep --color=auto --color -E etcd|flannel|docker|kube

Provided your machines output similiar results, kubernetes should be running! We can spin up the dashboard using kubectl. To do this we'll run the following command:

kubectl create -s 44.0.0.103:8888 -f /vagrant/dashboard.yaml

Thats it! Give kubernetes a couple minutes to schedule and start the dashboard Pod, and then you should be able to navigate to http://44.0.0.103:8888/ui/. It will redirected you to the dashboard and look something like this:

Next post in this series will go through using our kubernetes cluster to setup a simple highly available react/redux application!

• Pre-download kubernetes binaries and modify the Vagrantfile to share them with each machine so that we don't have to download the release on each machine, separately.
• Install the .kubeconfig bonus file into each machine manually or modifying the Vagrantfile to remove the need to append -s 44.0.0.103:8888 to each kubectl call.