Warning: This is a technical post.

We decided to adopt Kubernetes as the company doubles down on container-based deployment.

I am probably the last code-savvy person in the company to understand Kubernetes since we migrated our deployments to Kubernetes. Given that I have some code to deploy, I spent three days learning Kubernetes.

This post will serve as a reference for myself or anyone that will like to pick up Kubernetes and get deploying on it as soon as possible.

Problem: To push code into Gitlab, and have it automatically be deployed via Kubernetes

As a developer, I want to deploy changes made to code. We use Gitlab as

  • our code repository
  • our continuous integration/delivery (CI/CD)
  • docker container registry

Instead of ansible, I will like to have our docker containers deployed via Kubernetes.

Why

If you are not sure why Kubernetes can be helpful for your specific use-case, don't use Kubernetes. A simple ansible playbook will solve needs and scale up to an intermediate complexity level.

Kubernetes (server software)

Kubernetes as an orchestration server software is complex. You need a team to maintain Kubernetes. This is why we opted for managed Kubernetes instances.

I can recommend DigitalOcean or Scaleway for their managed Kubernetes. Scaleway if you need cheap resources and latency matters less.

Digitalocean for a slightly premium offering but with local servers in Singapore and hopefully your area of business.

Terminologies

  • Cluster - a group of servers managed by Kubernetes
  • Pod - a set of docker containers running on a specific volume and network interface
  • Service - a policy defining access to a set of pods
  • Deployment - A set of Pod replicas

Installing kubectl and connecting to a cluster

You use kubectl to interact with an instance of Kubernetes. You have to install it. As a mac user, I use homebrew to install it as such:

$ brew install kubernetes-cli

Thereafter, you need to interact with a Kubernetes cluster. To achieve that, you need a kubectl file which is usually stored in ~/.kube/config.

I used Scaleway's Kubernete's implementation known as Kapsule. In Kapsule's case, they provided me with a .yaml file known as kubeconfig-steven-private.yaml right after I created a Kapsule cluster. To immediately get started, I can replace the default config by doing the following:

$ mv ~/.kube/config 
$ mv ./kubeconfig-steven-private.yaml ~/.kube/config

To test if this is working, all I need to do is to

$ kubectl get pods

And it should show an empty list. If you are not authenticated properly or connected to a Kubernetes cluster correctly, you will get an error instead.

Testing it locally

Instead of a managed Kubernetes cluster which costs money, you can test Kubernetes locally with minikube. As a mac user, I can install it as such:

$ brew install minikube

Mac quirks

minikube is weird on Mac. It's default virtual machine driver is something that is really buggy. So this is the way I recommend that you start minikube on a mac.

$ minikube start --vm-driver=virtualbox

Testing docker locally

Sometimes, you want to test docker images locally on minikube without pushing it to a remote container registry. To do that:

$ eval $(minikube docker-env)
$ eval $(minikube docker-env -u) # to stop

Writing Kubernetes manifests

You need two different types of manifests to successfully deploy an application that is accessible.

The first type of manifest is called a Deployment. This is how it looks like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-hello-world
  labels:
    app: node-hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: node-hello-world
  template:
    metadata:
      labels:
        app: node-hello-world
    spec:
      containers:
      - name: node-hello-world
        image: node-hello-world
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
      imagePullSecrets:
      - name: gitlab-registry

This Deployment manifest

  • provides meta data on the application.
  • tells Kubernetes how many times we want to replicate it identically
  • tells Kubernetes what the Docker container image is
  • tells Kubernetes what port it listens on
  • tells Kubernetes how to authenticate with the private registry.

The second type of manifest is called a Service .

apiVersion: v1
kind: Service
metadata:
  name: node-hello-world-svc
  labels:
    app: node-hello-world-svc
spec:
  selector:
    app: node-hello-world
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

This Service manifest

  • describes itself with a name and some labels.
  • tells Kubernetes which Deployment it wants to connect to (via selector)
  • tells Kubernetes that it wants to interact with the outside world via TCP protocol. And the ports it will be listening/routing to.

Making Kubernetes and Gitlab play nice

To deploy a code base, first, I have to compile the code into a Docker image. This is how it looks like in my gitlab-ci.yml.

variables:
    IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG

build:
    stage: build
    image: docker:stable
    services:
        - docker:18-dind
    variables:
        DOCKER_TLS_CERTDIR: ""
    before_script:
        - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    script:
        - docker build -t $IMAGE_TAG .
        - docker push $IMAGE_TAG

The next step is to deploy it in the Kubernetes cluster that I have created on Scaleway. To do that.

deploy_on_scaleway:
    stage: deploy
    image: ${CI_REGISTRY}/nubela/k8s-ci-tools:latest
    before_script:
        - export KUBECONFIG=$KUBE_CONFIG
    script:
        - cd k8s
        - kustomize edit set image node-hello-world=$IMAGE_TAG
        - kustomize build . > build.yml
        - kubectl replace --force -f ./build.yml

This pipeline in Gitlab's CI tells it to use a docker image known as k8s-ci-tools:latest. This is a custom image that we have built for internal use. But really you can replicate it by creating an alpine docker image with kubectl and kustomize installed.

Dynamic manifest declarations via kustomize

Every code change results in a new docker image with a fresh hash. So I have to update the deployment.yaml file dynamically on every code push.
kustomize is an abstraction for  Kubernetesyaml manifests so I can dynamically update manifests. (What do you call an abstraction on top of an abstraction of an abstraction?)

To do that, I have a k8s folder in my project, with three files:

  • kustomization.yml
  • deployment.yml
  • nginx_route.yml

kustomization.yml contains the following:

- deployment.yml
- nginx_route.yml

To change an image name from node-hello-world to a Gitlab CI variable called $IMAGE_TAG, I run the command:

$ kustomize edit set image node-hello-world=$IMAGE_TAG

Then I can compile the changes as such:

$ kustomize build

Provisioning kube-config file

kubectl needs a config file and the default file is always located at ~/.kube/config. But you can change the default with your own file, by providing an environment variable known as $KUBECONFIG.

To provide that to the build context in Gitlab's CI, I have a Gitlab CI CI variable set as File, with the contents of kubeconfig-steven-private.yaml provided within.

This is why I set this line in the Gitlab pipeline to provide the CI variable file to kubectl:  export KUBECONFIG=$KUBE_CONFIG

Getting a remote server to access Gitlab's private registry.

There are two types of credentials. There is a temporary Gitlab credential that is only valid during the execution of the pipeline. This is not good enough as Kubernete's orchestration happens in the background. So even if kubectl commands return successful, it does not signify that any orchestration work has begun. This means the temporary credentials are not good enough.

The other type is a Deploy Token. You can generate one in the project settings in Gitlab.

Once you have created a deploy token, you can do that like this:

$ kubectl create secret docker-registry gitlab-registry --docker-server=<PATH_TO_REGISTRY> --docker-username=<DEPLOY_TOKEN_USERNAME> --docker-password=<DEPLOY_TOKEN_PASSWD> --docker-email=<YOUR_GITLAB_EMAIL>

In this case, I have named the credential as gitlab-registry.

And this credential needs to be added under spec in the Deployment manifest as such:

      imagePullSecrets:
      - name: gitlab-registry

Using nginx to route http traffic

To use nginx to route http traffic amongst pods in my cluster, I have to install nginx first as a Ingress Controller. Ingress is a service, and an Ingress Controller is software to enable this particular Ingress service. In my case, with nginx.

To do so, I need to install nginx in the Kubernetes cluster. I will do this with helm. helm is a "package manager" for Kubernetes. First, I need to install helm locally.

$ brew install helm

Then I have to install tiller in the Kubernetes cluster. I need tiller in Kubernetes cluster for helm to work.

$ kubectl -n kube-system create serviceaccount tiller
$ kubectl create clusterrolebinding tiller \
  --clusterrole=cluster-admin \
  --serviceaccount=kube-system:tiller
$ helm init --service-account tiller
$ helm install stable/nginx-ingress --name hello-nginx # install nginx

Finally, I declare my route with an Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: node-hello-world-ingress
spec:
  rules:
  - host: k8s-nginx-test.nubela.co
    http:
      paths:
      - path: /
        backend:
          serviceName: node-hello-world-svc
          servicePort: 80

Kubernetes is hard comprising of a serious amount of abstraction on top of abstractions. I will say avoid Kubernetes unless you know exactly what you are doing.