The Guide to Kubernetes DevOps Tools

It’s fascinating to think that merely 10-15 years ago, all we needed to deploy an application were a set of Bash scripts. Then, containerization took hold, and Docker containers running on Kubernetes orchestrator became standard. In this article, we will aim to provide an overview of the suite of DevOps tools available for Kubernetes and its ecosystem. For the sake of brevity, we will divide Kubernetes tooling into feature groups and take a closer look at each one.

Roughly speaking, all of the tools present within the Kubernetes ecosystem can be divided into the following groups:

• cluster provisioning, tools used to create Kubernetes clusters
• cluster administration, tools used to control and instruct Kubernetes to run things
• application provisioning, tools used to provision Kubernetes applications onto the clusters
• local Kubernetes distributions, tools used to deploy local development environments and stripped-down Kubernetes clusters without the need to pay for cloud resources
• application debugging, tools used to develop and debug applications directly on Kubernetes clusters

The most famous representatives of each category can be seen in the table below

Selecting the right tools for the provisioning of Kubernetes clusters can often lead to more questions than answers. To help you, the following questions and considerations should prove useful during your decision making:

• Is your infrastructure described as code (IaC), and does it use managed services? If so, then the likes of Terraform and Pulumi are your best choice as they avoid the need to use additional tools. IaC tools alone are not sufficient if you require the use of standard clusters, as you will need to manage the entire installation and configuration process yourself
• Are you creating production-ready clusters? Then kubespray or kops would be a good fit; otherwise, the simplicity of kubeadm is the way to go.
• Do you intend to create standard clusters across multiple cloud providers? Then again, kubespray is a good offering.

Evaluating each tool against your organization’s existing infrastructure and future roadmap is always a good practice. This will save your organization and its engineers from needless migration between tools and dran-out adaptation periods.

As an alternative to creating a fully-fledged Kubernetes cluster in the cloud, and for simple tasks such as local development or testing, it might make more sense to run Kubernetes locally, for example on a laptop.

All of the tools we’ve listed above are relatively similar, and are capable of delivering a lightweight version of Kubernetes. For example, creating a cluster via Docker with kind is as simple as running:

kind create cluster

With minikube:

minikube start

You may choose from plenty of configuration options too, such as the Kubernetes version, its runtime, add-ons, or even the virtualization driver used for cluster node creation. The key point remains: a few commands can get you up and running with a fully operational Kubernetes cluster.

Having a running cluster means nothing without the ability to interact with it. To this end we can take two routes: terminal or graphical interface.

kubectl is the official command-line tool and allows us to interact with the Kubernetes APIs that are provided by the cluster control plane. Any operation, ranging from describing a payload to deploying an application can be performed via Kubectl.

In our example below, we easily return a list of pods running in a ‘demo’ namespace and obtain a set of logs from a pod:

kubectl get pods -n demo
kubectl logs my-pod -n demo

kubectx and kubens on the other hand, allows for the pre-selection of a cluster and a namespace where commands can be run respectively. This is mostly a cosmetic improvement, but one that saves the excessive typing of commands.

For instance, in our previous example above we used commands that referenced a namespace - this can be omitted when using kubens:

kubens demo
# all further commands will now apply to demo namespace
kubectl get pods
kubectl logs my-pod

k9s is a significantly different tool - a terminal UI application that provides a way to look in-depth at a selected cluster. It has built-in dashboards and metric aggregation that allows us to track both cluster and applications health. It’s also able to perform simple day-to-day tasks such as listing running pods or getting the logs from a payload.

List of pods and its resource footprint (source)

Lens IDE is a powerful, standalone GUI application that can control and explore Kubernetes clusters graphically. Whether running commands against a cluster, deploying an application via Helm, or just exploring Ingress configurations, Lens does it all visually.

List of pods and details of a particular pod (source)

No matter how different these tools may appear at first glance, they are all based upon one source of truth, the official Kubernetes APIs. Choosing one tool over another might be a matter of practicality, use case limitations (for example, usage of CLIs in CI/CD pipelines) or just personal preference.

Kubecost is a tool designed to assist teams in monitoring and managing costs and capacity within Kubernetes environments. The tool integrates with your cluster to enable efficient tracking, management, and reduction of expenses. It offers real-time cost visibility and insights, aiding in the reduction of cloud costs for Kubernetes users.

The features encompass flexible cost breakdowns, unified monitoring of Kubernetes and external spending, personalized optimization suggestions, customizable alerts and governance. Kubecost can be installed via a Helm Chart on your infrastructure within minutes (free for individual clusters), but is also available as a Software as a Service (SaaS) for those who prefer a cloud-hosted solution. Kubecost also founded the OpenCost open-source project worth noting.

Deploying applications into a Kubernetes cluster is typically achieved via manifests - files written in YAML that contain the instructions sent to the Kubernetes API. Maintaining these files, especially at scale and when working with multiple app versions and environments is tricky and requires good organization. This is where tools such as Helm and Kustomize can come to the rescue.

Kustomize is a declarative templating engine that uses the concept of a base manifest and overlays, which are specific to the environment or configuration. Kustomize is simple and Kubernetes native, but its inability to handle operations at scale (multiple applications, multiple overlays and so on) make it unsuitable for larger organizations.

Helm is a more feature-rich tool that manages and packages manifests. Helm uses charts, which are a combination of templates and value files. Charts can be stored centrally into a chart museum and this provides a way to easily use and share them. Helm also handles the full lifecycle of an application. It’s a mature tool with plenty of features and good documentation.

It’s worth mentioning at this point a set of practices called GitOps. These provide a means to configure clusters and deploy applications using declarative definitions stored in something called a Git repo. Two representative tools that can implement GitOps are Flux and Argo CD. There are many similarities between them (reconciliation mode, secrets management, alerting and monitoring) but also quite a few differences (deployment model, support for Helm and Kustomize, web UI).

Lastly, we will examine some tools which are particularly useful when debugging applications running on Kubernetes. Some of these tools are IDE extensions, such as Bridge to Kubernetes, and CloudCode, where others are standalone applications, such as squash and telepresence. These inject agents into Kubernetes clusters and capture what’s happening via either network proxies or process thread debugging. Choosing between them is really just a case of convenience, functionality or personal choice.

During this article, we looked at some of the various tools available to aid the deployment of both Kubernetes clusters and the applications within. We provided a few practical demonstrations and even a little advice on which tools might be suitable for your specific use-cases.

If you found this article useful, and we hope that you did, we’d like to draw your attention to others that have been written as part of an overall guide to Kubernetes DevOps Tools. By following the links below, you can find articles that dive further into some of the concepts and topics covered during this piece. This includes several IaC tools and their differences (Pulumi vs Terraform), GitOps operators (ArgoCD) and tools used for the provisioning of Kubernetes (kubeadm, kubespray). See you next time!

We have organized this guide in the following articles:

Chapter 1: ArgoCD Tutorial

Chapter 2: Istio & Envoy: How they work together

Chapter 3: Pulumi vs Terraform

Chapter 4: Traefik vs NGINX

Chapter 5: cAdvisor Tutorial

Chapter 6: Kubespray

Chapter 7: Kubernetes Monitoring Tools

Chapter 8: Kube-Prometheus

Chapter 9: Kubectl Cheat Sheet

Chapter 10: Kubernetes External Secrets

Chapter 11: Kustomize vs Helm

Each chapter includes detailed explanations of the feature, along with configuration examples so you can see them implemented in practice.