Automate All the Boring Kubernetes Operations With Python

Learn how you can use Python’s Kubernetes Client library to automate all the boring Kubernetes tasks and operations

Background Photo by Sharon McCutcheon on Unsplash

Kubernetes became a de-facto standard in recent years, and many of us — both DevOps engineers and developers alike — use it on a daily basis. Many of the tasks we perform are, however, the same, boring, and easy to automate.

Often, it’s simple enough to whip up a quick shell script with a bunch of kubectl commands, but for more complicated automation tasks bash just isn't good enough, and you need the power of proper languages, such as Python.

So, in this article, we will look at how you can leverage the Kubernetes Python Client library to automate whatever annoying Kubernetes task you might be dealing with!

Playground

Before we start playing with the Kubernetes client, we first need to create a playground cluster where we can safely test things out. We will use KinD (Kubernetes in Docker), which you can install from here.

We will use the following cluster configuration:

To create a cluster from the above configuration, you can run:

With the cluster up and running, we also need to install the client library (optionally, inside the virtual environment):

Authentication

To perform any action inside our Kubernetes cluster, we first need to authenticate.

We will use long-lived tokens so we don’t need to go through the authentication flow repeatedly. Long-lived tokens can be created by creating a ServiceAccount:

Using a service account also has the benefit that it’s not tied to any single person, which is always preferable for automation purposes.

The token from the output above can be then used in requests:

We’re authenticated now, but not authorized to do much of anything. Therefore, next, we need to create a Role and bind it to the ServiceAccount so we can perform actions on resources. Here’s the code:

The above gives our service account permission to perform any action on pods, limited to default namespace.

You should always keep your roles very narrow and specific, but playing around in KinD, it makes sense to apply a cluster-wide admin role, as you can see below:

Raw Requests

To get a better understanding of what is kubectl and also the client doing under the hood, we will start with raw HTTP requests using curl.

The easiest way to find out what requests are made under the hood is to run the desired kubectl command with -v 10. This will output complete curl commands, which are shown below:

The output with loglevel 10 will be very verbose, but somewhere in there, you will find the above curl command.

Add a Bearer token header in the above curl command with your long-lived token, and you should be able to perform the same actions as kubectl, such as the following:

In case there’s a request body needed, look up which fields need to be included in the request. For example, when creating a Pod, we can use the API described here. When we do, we get the following request:

Refer to the Kubernetes API reference for object attributes. Additionally, you can also view the OpenAPI definition with this command:

Interacting with Kubernetes directly using REST API might be a bit clunky, but there are situations where it might make sense to use it. Some situations include interacting with APIs that have no equivalent kubectl command when you're using a different distribution of Kubernetes — such as OpenShift — which exposes additional APIs not covered by either kubectl or client SDK.

Python Client

Moving onto the Python client itself now. We need to go through the same step as with kubectl or curl. The first step is authentication, which is shown below:

First, we define a configuration object, which tells the client that we will authenticate using the Bearer token. Considering that our KinD cluster doesn’t use SSL, we disable it in the real cluster. However, you should never do that.

To test out the configuration, we use list_namespaced_pod method of API client to get all pods in the default namespace, and then we print out their name, namespace and IP.

Now, for a more realistic task, let’s create a Deployment below:

In addition to creating the Deployment, we also wait for its pods to become available. We do that by querying the Deployment status and checking a number of available replicas.

Also, notice the pattern in function names, such as create_namespaced_deployment. To make it more obvious, let's look at a couple more:

• replace_namespaced_cron_job
• patch_namespaced_stateful_set
• list_namespaced_horizontal_pod_autoscaler
• read_namespaced_daemon_set
• read_custom_resource_definition

All of these are in the format operation_namespaced_resource or just operation_resource for global resources. They can be additionally suffixed with _status or _scale for methods that perform operations on resource status, such as read_namespaced_deployment_status or resource scale such as patch_namespaced_stateful_set_scale.

Another thing to highlight is that in the above example, we performed the actions using client.AppsV1Api which allows us to work with all the resources that belong to apiVersion: apps/v1. If we wanted to use CronJob, we would instead choose BatchV1Api (which is apiVersion: batch/v1 in YAML format), or for PVCs, we would choose CoreV1Api because of apiVersion: v1 . You get the gist.

As you can imagine, that’s a lot of functions to choose from. Luckily, all of them are listed in docs, and you can click on any one of them to get an example of its usage.

Beyond basic CRUD operations, it’s also possible to watch objects for changes continuously. The obvious choice is to watch Events:

Here we chose to watch events in default namespace. We take the first 10 events and then close the stream. If we wanted to monitor the resources continuously, we would just remove the timeout_seconds and the w.stop() call.

In the first example, you saw that we used plain Python dict to define the Deployment object, which we passed to the client. Alternatively, we can use a more OOP style by using API Models (classes) provided by the library, shown below:

Trying to figure out which model you should use for each argument is a losing battle, and it’s tough. When creating resources like those shown above, you should always use documentation for models and traverse the links as you create the individual subobjects to figure out what values/types are expected in each field.

Handy Examples

You should now have a basic idea about how the client works, so let’s take a look at some handy examples and snippets that might help you automate daily Kubernetes operations.

A very common thing you might want to perform is a Deployment rollout — usually done with kubectl rollout restart. However, there’s no API to do this. The way kubectl does it is by updating Deployment Annotations. More specifically, setting kubectl.kubernetes.io/restartedAt to the current time. This works because any change made to Pod spec causes a restart.

If we want to perform a restart using the Python client, we need to do the following:

Another common operation is scaling a Deployment. Fortunately, this one has an API function we can use, which you can see below:

For troubleshooting purposes, it often makes sense to exec into a Pod, take a look around, and possibly grab the environment variable to verify the correct configuration. Here’s the code for that:

The snippet above also allows you to run whole shell scripts if need be.

So, let’s say you want to apply a Taint onto a node that has some issue. We can focus on cluster administration-oriented tasks. Since there’s no direct API for Node Taints, we can find a way. Here’s the code to help us:

You might also want to monitor cluster resource utilization to automate cluster scaling. Usually, you’d use kubectl top to get the information interactively, but with the client library, you can do the following:

The above example assumes that you have metrics-server installed in your cluster. You can run kubectl top to verify that. Use the comment in the snippet to install it if you're working with KinD.

Last but not least, if you have a bunch of YAML or JSON files you want to use to deploy or modify objects in your cluster, or you want to export and backup what you’ve created with the client, there’s an easy way. Here’s how you can convert YAML/JSON files to Kubernetes objects and back to files again:

The first way to convert an existing object into a Python dictionary (JSON) is to use sanitize_for_serialization which produces raw output with all the generated/default fields. A better option is to use utility methods of the kopf library, which will remove all unnecessary fields. From there, it's simple enough to convert a dictionary into a proper YAML or JSON file.

If you want to reverse this process — go from the dictionary to a Client Object Model — we can use deserialize method of API Client. This method, however, expects its argument to have a data attribute, so we pass it a container class instance with such an attribute.

If you already have YAML files that you’d like to use with the Python client, you can use the utility function, kubernetes.utils.create_from_yaml.

To get a complete overview of all the library features, I recommend you take a look at the examples directory in the repository.

I’d also encourage you to look through the issues in the library repository, as it has a lot of great examples of client usage, such as processing events in parallel or watching ConfigMaps for updates.

Conclusion

The Python client library contains hundreds of functions, so it’s difficult to cover every little feature or use case there is. Most of them follow a common pattern which should make the library’s usage pretty natural after a couple of minutes.

If you’re looking for more examples beyond what was shown and referenced above, I recommend exploring other popular tools that make use of the Python Kubernetes client, such as the library for creating Kubernetes operators. I also find it very useful to take a look at tests of the library itself, as it showcases its intended usage. Here’s an excellent client test suite to study.

You may also like

Building GitHub Apps With Golang

Keeping Kubernetes Clusters Clean and Tidy

Want to Connect?
This article was originally posted at martinheinz.dev