Today, we’ll talk about some deployment tools in the Kubernetes world. We’ll talk a little bit about why tools are used on top of kubectl. We’ll compare 3 different tools in the Kubernetes world focused on the deployment side of things: Kustomize, Helm, and Kubes. Then we’ll elaborate on the features that Kubes offers.

Kubectl with Simple Wrappers

Most folks start off with kubectl commands to create their Kubernetes resources. It’s simple. It’s also important to learn how to use kubectl commands to establish fundamentals. Eventually, you grow tired of typing the same commands repeatedly, though. So you write a wrapper bash script. Example:

kubectl-wrapper.sh

kubectl apply -f service.yaml
kubectl apply -f deployment.yaml

Bash shines for simple scripts and light glue, but it can quickly get messy as the script takes on more things to do. For example, what happens when you need another env like dev and prod?

You may start structuring things like this:

├── dev
│   ├── deployment.yaml
│   └── service.yaml
└── prod
    ├── deployment.yaml
    └── service.yaml

And write a wrapper script that selects the folder:

kubectl-wrapper.sh

KUBE_ENV=${1:-dev}
kubectl apply -f $KUBE_ENV/service.yaml
kubectl apply -f $KUBE_ENV/deployment.yaml

The issue is duplication of service.yaml and deployment.yaml. Instead, it’ll be nice if we use the same YAML and create a different env like dev and prod with it. Things like envsubst to replace variables from the same “template” YAML files can help. As requirements increases though, the simple bash glue scripts end up getting messy.

Using kubectl with simple wrapper scripts is like using a simple manual screwdriver. Eventually, you want to be more efficient. We want a power drill. We’ll discuss some power tools next.

• Source Code: Kubectl Examples

Kubectl vs Kustomize

Kustomize started off as a project outside of kubectl. A version of it is now built into the kubectl command. Kustomize allows you to write a kustomization.yaml that decorate existing YAML Kubernetes files.

Kubectl Structure

Here’s an example Kubectl project structure:

├── base
│   ├── deployment.yaml
│   ├── kustomization.yaml
│   └── service.yaml
└── overlays
    ├── dev
    │   ├── deployment.yaml
    │   ├── kustomization.yaml
    │   └── namespace.yaml
    └── prod
        ├── deployment.yaml
        ├── kustomization.yaml
        └── namespace.yaml

Using Kustomize to Create Different Environments

The provided structure allows you to use the same code to create different environments. To create different dev and prod environments, we use overlays:

kubectl apply -k overlays/dev
kubectl apply -k overlays/prod

Kustomize has a purist perspective. It uses YAML only to decorate and build new YAML files.

While we can appreciate the good intentions of trying to keep everything in YAML and avoid context switching, Kustomize puts too much logic into YAML. In a Kubernetes world where the amount of YAML we have to use can sometimes become embarrassing, Kustomize opts to use even more YAML. Feel like when we start seeing YAML that contains verbs representing method calls, we may be going in the wrong direction. It’s hard to solve the problem of YAML with even more YAML.

Additionally, Kustomize features generator methods as a way to remove duplication, but you can only get so far with the methods. You end up with some duplication in the kustomization.yaml and namespace.yaml files.

Source code: Kustomize Examples

Kustomize vs Helm

The Helm approach to building YAML files takes an entirely different direction. Instead of a purist YAML approach, Helm uses templating logic.

Helm Structure

Here’s an example Helm project structure:

├── Chart.yaml
├── templates
│   ├── _helpers.tpl
│   ├── deployment.yaml
│   └── service.yaml
└── values.yaml

Templates like deployment.yaml and service.yaml reside in the templates folder. The values.yaml file provides default variables values to substitute. Let’s take a look at part of a template to understand how the templating works:

templates/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "mychart.fullname" . }}
  labels:
    {{- include "mychart.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
{{- end }}
  selector:
    matchLabels:
      {{- include "mychart.selectorLabels" . | nindent 6 }}
  template:
    metadata:
    {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
    {{- end }}
      labels:
        {{- include "mychart.selectorLabels" . | nindent 8 }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: 80
              protocol: TCP

The templating language is a mixture of the Go template language and the Sprig template library.

Helm’s templating approach allows us to use conditional logic like if statements and methods. It’s much more powerful. At the same time, it’s quite difficult to read. It reminds me of PHP and gets spaghetti-like quickly. Also, since YAML must be properly aligned, it can get error-prone.

Helm Different Environments

To achieve different envs with the same code, we can use different variables. Here are example commands:

helm install chart-dev . --namespace chart-dev --create-namespace -f values/dev.yaml
helm install chart-prod . --namespace chart-prod --create-namespace -f values/prod.yaml

Helm creates the namespace outside of YAML and it’s lifecycle is not managed by Helm. The --create-namespace option is only necessary once. To use different variable values, you use the -f option. You can specify as many variables files as you wish.

You have to remember to type the CLI options. The command becomes verbose, and it’s easy to forget to type the right options. Typically, you end up writing bash wrapper script to reduce the risk of errors. For example, it could be something like this:

helm-wrapper.sh

HELM_ENV=${1:-dev}
helm install chart-$HELM_ENV . --namespace chart-$HELM_ENV --create-namespace -f values/$HELM_ENV.yaml

Usage would be like this:

helm-wrapper.sh dev
helm-wrapper.sh prod

• Helm Examples

Helm’s Greater Scope

Helm does a lot more than build YAML files from templates. Helm also supports hooks, rollbacks, packaging, and server for distribution. It’s a full package manager. So Helm’s scope is far greater than Kustomize, we’re somewhat comparing apples to oranges here.

Kustomize vs Helm vs Kubes

Kubes is another tool that handles deployment. Kubes has some similar concepts to both Kustomize and Helm and improves on them.

Kubes Structure

Here’s a Kubes directory structure.

.kubes/resources
├── base
│   ├── all.yaml
│   └── deployment.yaml
├── shared
│   └── namespace.yaml
└── web
    ├── deployment
    │   ├── dev.yaml
    │   └── prod.yaml
    ├── deployment.yaml
    └── service.yaml

Kubes introduces a conventional folder structure. Conventions takes you a long way. Instead of spending time configuring and wiring files together with kustomization.yaml like with Kustomize files or specifying CLI --namespace and -f options like with Helm, commands can become a lot shorter and more memorable.

How Kubes Works

Kubes works in a transparent and straightforward manner. The kubes deploy command first builds the Docker image. Then it compiles Kubernetes YAML files. Lastly, it merely calls out to kubectl.

In fact, you can use Kubes to build the files first and then run kubectl directly. Example:

kubes docker build
kubes docker push
kubes compile  # compiles the .kubes/resources files to .kubes/output

Now, use kubectl directly and apply them in the proper order:

kubectl apply -f .kubes/output/shared/namespace.yaml
kubectl apply -f .kubes/output/web/service.yaml
kubectl apply -f .kubes/output/web/deployment.yaml

The deploy command simply do all 3 steps: build, compile, and apply.

kubes deploy

Layering: Multiple Environments like dev and prod

To deploy and create multiple environments like dev and prod with the same YAML, we use a different KUBES_ENV value:

KUBES_ENV=dev  kubes deploy
KUBES_ENV=prod kubes deploy

The same code is used to create different environments. Kubes achieves this with a feature called Layering. The concept is similar to Kustomize overlays. Here’s the general layering processing order that Kubes takes.

1. The .kubes/resources/base folder is treated as a base layer. It gets processed as pre-layers by Kubes.
2. Then Kubes will process your .kubes/resources/ROLE definitions.
3. Lastly, Kubes processes any post-layers in the .kubes/resources/ROLE/KIND folders.

Let’s focus on deployment.yaml to explain and understand layering. Here are the files that get layered.

.kubes/resources/base/all.yaml
.kubes/resources/base/deployment.yaml
.kubes/resources/web/deployment.yaml
.kubes/resources/web/deployment/dev.yaml

Each file is merged together and produces a resulting YAML file:

.kubes/output/web/deployment.yaml

The final output deployment.yaml is the combined layered YAML files.

The same layering processing logic runs for the other files too. Here are all the built output files:

.kubes/output/shared/namespace.yaml
.kubes/output/web/deployment.yaml
.kubes/output/web/service.yaml

With Kustomize, you must write kustomization.yaml files to stitch together YAML files and define the overlays. With Kubes, it just works. The conventions allow the concept of multiple environments to be baked-in right off the bat. All you do is specify KUBES_ENV.

• Source Code: Kubes Examples

ERB Templating Support

Whereas Kustomize does not allow any templating logic, Helm goes all-in on templating logic. Kubes allows for both. Kubes merges YAML together via layering. Kubes also allows for templating logic via ERB. Here is an example of ERB usage:

.kubes/resources/shared/namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: demo-<%= Kubes.env %>
  labels:
    app: demo

Notice the <%= Kubes.env %> templating logic. When is KUBES_ENV=dev, then name: demo-dev. When is KUBES_ENV=prod, then name: demo-prod.

Kubes templating support allows you to use ERB where it makes sense. This provides the best of each world.

Extendable

The templating logic with Kubes is simply Ruby ERB. Kubes has some built-in helpers. For example, Kubes uses the built-in helper docker_image to automatically substitute the Docker image built from your Dockerfile. You can also extend Kubes and add user-defined custom helper methods.

With Helm, you can also add templating methods with custom helpers. The helper method definitions are awkward looking, though. Example:

templates/_helpers.tpl

{{- define "demo.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "demo.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}

With Kubes, custom template helper definitions is just Ruby code. Example:

.kubes/helpers/my_helpers.rb

module MyHelpers
  def database_endpoint
    case Kubes.env
    when "dev"
      "dev-db.cbuqdmc3nqvb.us-west-2.rds.amazonaws.com"
    when "prod"
      "prod-db.cbuqdmc3nqvb.us-west-2.rds.amazonaws.com"
    end
  end
end

The Power of Ruby

Kubes is written in Ruby. This fact is transparent to the end-user. The starter learning guide take you through a gentle path, where you are using YAML just like you usually would. You also have access to Ruby, but it’s lightly added on top of YAML. Think about it as “Ruby Sprinkles.”

Modern-day DevOps shops use a variety of tools like bash, python, ruby, and go to achieve their goals. As the adage goes: use the right tool for the job. Language shouldn’t matter, but it does. Ruby is one of the most powerful languages to craft tools and glue things together to make your life easier. Ruby is a versatile language and is well-suited to achieve tools like Kubes.

Docker Build

You may have notice that Kubes also handles building the Docker image. Kustomize and Helm do not. Building the Docker image is one less thing for you to do. It streamlines the deploy workflow.

Also, if you wish to use a prebuilt docker image instead, you can also do that with Kubes. See the --image option or .kubes/config.rb in the Docker Image Docs. Kubes provides you options.

Hooks: Finer-Grain Control

Kustomize does not support hooks. Helm supports hooks. Like Helm, Kubes also supports hooks, but they provide finer-grain control.

A key difference between these tools is how kubectl apply gets called. Essentially, both Kustomize and Helm generate a single YAML file and then runs kubectl apply on it. Instead, Kubes generates separate YAML files and calls kubectl apply on each file individually. Because of this, kubes hooks can run at fine-grain points. Example:

# hook can run here
kubectl apply -f .kubes/output/shared/namespace.yaml
# hook can run here
kubectl apply -f .kubes/output/web/service.yaml
# hook can run here
kubectl apply -f .kubes/output/web/deployment.yaml
# hook can run here

Note: Kubes also generates a single full.yaml for your convenience and debugging.

The hooks are more fine-grain. More docs: Kubes Kubectl Hooks

Comparison Table

Here’s a summary comparison table.

Summary

In this post, we covered the differences between Kustomize, Helm, and Kubes. Kustomize is built into the kubectl command is more like a feature. You use kustomization.yaml files to glue things together. Helm is a full package manager tool that also builds YAML files. Helm uses templating logic. Kubes allows for both YAML merging and templating. It provides additional conveniences like building the Docker image.

We did not cover all the features of these tools, for more info check out their docs sites:

• Helm
• Kubes
• Kustomize

Lastly, source code examples for each of these tools is available here:

• Helm Examples
• Kubectl Examples
• Kubes Examples
• Kustomize Examples