Writing Custom Kubernetes Controller and Webhooks

Create a Kubernetes API, controller, validate webhooks, and test.

Photo by Benjamin Davies on Unsplash

In the previous article, we tried to understand the Unit tests in go language using mocks. In this article, we will cover the following topics

1. Creating a Kubernetes API
2. Creating a Kubernetes controller
3. Creating a Validating webhook
4. Testing the changes with kind cluster
5. Bonus tip

Before starting with writing our own controller, we should have the basic knowledge about what is controller, webhooks in Kubernetes.

Writing custom Kubernetes controller

Let us try to develop a controller for adding a two numbers, It may sound simple but here the intention is to understand the steps required to create API or controller. We will use kubebuilder for initial scaffolding.

1. Create a new directory and initialize the project

$ mkdir custom-k8-controller

$ cd custom-k8-controller 

$ go mod init github.com/Karthik-K-N/custom-k8-controller

2. Use kubebuilder for initial scaffolding

$ kubebuilder init --domain sample.domain --repo github.com/Karthik-K-N/custom-k8-controller

$ kubebuilder create api --group calculator --version v1 --kind Sum
Create Resource [y/n]
y
Create Controller [y/n]
y

3. Open the newly created project in the editor of your choice. Now we can see various directories created by Kubebuilder for us. The full project can be found in this GitHub repository.

4. Now let's add the required members to the SumSpec and SumStatus structs under API directory. Since our intention is to add two numbers, we need to add two members (NumberOne, NumberTwo) in SumSpec representing the input and one member (Result) to store the result in SumStatus.

// SumSpec defines the desired state of Sum
type SumSpec struct {
   NumberOne int `json:"numberOne,omitempty"`
   NumberTwo int `json:"numberTwo,omitempty"`
}

// SumStatus defines the observed state of Sum
type SumStatus struct {
   Result int `json:"result,omitempty"`
}

5. Now let us add the main controller logic in Reconcile method to find the sum of two numbers. Make sure to add enough logging — it will be helpful in debugging and also understanding the code workflow.

func (r *SumReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {

   var sum calculatorv1.Sum
   if err := r.Get(ctx, req.NamespacedName, &sum); err != nil {
      klog.Error(err, "unable to fetch sum")
      return ctrl.Result{}, client.IgnoreNotFound(err)
   }
   klog.Infof("Found the sum object %v", sum)

   klog.Infof("Calculating the sum of %d and %d", sum.Spec.NumberOne, sum.Spec.NumberTwo)
   result := sum.Spec.NumberOne + sum.Spec.NumberTwo
   sum.Status.Result = result

   klog.Info("Updating the result of calculation")
   if err := r.Status().Update(ctx, &sum); err != nil {
      klog.Error(err, "Unable to update sum status")
      return ctrl.Result{}, err
   }

   klog.Infof("Successfully updated the sum status with result %d", result)
   return ctrl.Result{}, nil
}

6. Once the business logic is completed, now it is time to test the controller. As a prerequisite, we need a Kubernetes cluster for easy testing. Let us create a kind cluster and deploy cert-manager which is required when we have to test our webhook changes.

$ kind create cluster --name custom-controller
$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.yaml

7. Create the manifests of our controller, install them onto the cluster and run the controller.

$ make manifests
$ make install
$ make run

8. Create and apply Sum resource:

$ cat custom-k8-controller/config/samples/calculator_v1_sum.yaml
apiVersion: calculator.sample.domain/v1
kind: Sum
metadata:
  name: sum-sample
spec:
  numberOne: 10
  numberTwo: 20
$ kubectl create -f custom-k8-controller/config/samples/calculator_v1_sum.yaml
$ kubectl get sum
NAME         AGE
sum-sample   5m8s
$ kubectl get sum sum-sample -o yaml                                                           
apiVersion: calculator.sample.domain/v1
kind: Sum
metadata:
  creationTimestamp: "2022-02-27T17:17:03Z"
  generation: 1
  name: sum-sample
  namespace: default
  resourceVersion: "4562"
  uid: 9d25777c-d695-485f-9ba0-9c65526a8737
spec:
  numberOne: 10
  numberTwo: 20
status:
  result: 30

9. Add the kubebuilder tags to view more information on fetching sum resources.

//+kubebuilder:printcolumn:name="Number One",type="integer",JSONPath=".spec.numberOne",description="Input number one"
//+kubebuilder:printcolumn:name="Number Two",type="integer",JSONPath=".spec.numberTwo",description="Input number two"
//+kubebuilder:printcolumn:name="Result",type="integer",JSONPath=".status.result",description="Sum of two numbers"

// Sum is the Schema for the sums API
type Sum struct {
   metav1.TypeMeta   `json:",inline"`
   metav1.ObjectMeta `json:"metadata,omitempty"`

   Spec   SumSpec   `json:"spec,omitempty"`
   Status SumStatus `json:"status,omitempty"`
}

10. Create and install the new manifest and observe the changes in the result

$ make manifests
$ make install
$ make run
$ kubectl get sum
NAME         NUMBER ONE   NUMBER TWO   RESULT
sum-sample   10           20           30

In this way, we can easily get the inputs and their corresponding results.

Writing Kubernetes Webhook

As our sum controller is ready now, let us add a webhook. Here we will only add validation webhook — we are not going to add mutating or defaulting webhooks.

1. Again let us use the kubebuilder for webhook scaffloding.

$ kubebuilder create webhook --group calculator --version v1 --kind Sum  --programmatic-validation

2. Kubebuilder will create the webhook file under api directory. Now let us add the validation webhook logic. For simplicity let us invalidate the controller on negative numbers

func (r *Sum) ValidateCreate() error {
   klog.Infof("In validate create of %s", r.Name)
   if r.Spec.NumberOne < 0 || r.Spec.NumberTwo < 0 {
      return fmt.Errorf("The input values Number One: %d Number Two: %d cannot be negative ", r.Spec.NumberOne, r.Spec.NumberTwo)
   }
   return nil
}

3. Before testing the changes, we need to uncomment a few files to make the webhook work:

1. config/crd/kustomization.yaml
2. config/default/kustomization.yaml

Since we are using only validation webhook, we need to remove the Mutating block from the following file.

$ cat config/default/webhookcainjection_patch.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: validating-webhook-configuration
  annotations:
    cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)

4. Now we are good to test the changes. Let us first build the docker image. When testing the webhook changes, it is better to run it as a container so that we do need not deal with certificate management on a local machine.

$ make docker-build IMG=karthik/custom-controller:v1

5. Since we are using a kind cluster, we need not push the docker image to the dockerhub. Instead, we can load it onto the kind cluster

kind load docker-image karthik/custom-controller:v1 --name custom-controller

6. We are ready to deploy the image

make deploy IMG=karthik/custom-controller:v1

7. Check the controller pod status

$ kubectl -n custom-k8-controller-system  get pods
NAME                                                       READY   STATUS    RESTARTS   AGE
custom-k8-controller-controller-manager-784f74cdf6-45hpq   2/2     Running   0          66s

8. Now, let us try creating an invalid sum resource with negative numbers and observe what the controller and webhook does.

$ kubectl create -f config/samples/calculator_v1_sum.yaml
Error from server (The input values Number One: -10 Number Two: 20 cannot be negative ): error when creating "config/samples/calculator_v1_sum.yaml": admission webhook "vsum.kb.io" denied the request: The input values Number One: -10 Number Two: 20 cannot be negative

As expected the validation webhook rejected the request for creating new resource. Also, we can see what controller logs say:

$ kubectl -n custom-k8-controller-system logs -f custom-k8-controller-controller-manager-784f74cdf6-45hpq
I0304 15:45:49.386571       1 sum_webhook.go:39] In validate create of sum-sample
1.646408749386699e+09    DEBUG    controller-runtime.webhook.webhooks    wrote response    {"webhook": "/validate-calculator-sample-domain-v1-sum", "code": 403, "reason": "The input values Number One: -10 Number Two: 20 cannot be negative ", "UID": "2343b586-dc3f-45b5-9c75-df28b1c30b40", "allowed": false}

In this way, we can test both our controller and the webhook changes.

Bonus Tip — Creating a webhook file in a different directory other than the default API directory

If we observe carefully, when we used kubebuilder for webhook scaffolding, it created the file sum_webhook.go file under API directory and added one more dependency to the API directory.

If it is a simple independent controller it does not matter. But, what if the newly created APIs are imported into another controller? In that case, the imported API will contain unnecessary webhook dependencies.

In order to avoid this, we can use webhooks custom validator. Let us move the generated webhook files to a new directory named webhook.

Create a simple struct, implement Customvalidator methods and update the SetupWebhookWithManager method. The new method looks like this and the file content can be seen here.

type SumWebhook struct {
}
func (r *SumWebhook) SetupWebhookWithManager(mgr ctrl.Manager) error {
   return ctrl.NewWebhookManagedBy(mgr).
      For(&v1.Sum{}).
      WithValidator(r).
      Complete()
}

Remember to change the webhook validator to a custom validator:

// var _ webhook.Validator = &Sum{}
var _ webhook.CustomValidator = &SumWebhook{}

Finally, let us update the main.go file to use the newly created webhook instead of the default webhook from the API directory.

//if err = (&calculatorv1.Sum{}).SetupWebhookWithManager(mgr); err != nil {
// setupLog.Error(err, "unable to create webhook", "webhook", "Sum")
// os.Exit(1)
//}

if err = (&webhook.SumWebhook{}).SetupWebhookWithManager(mgr); err != nil {
   setupLog.Error(err, "unable to create webhook", "webhook", "Sum")
   os.Exit(1)
} 

Once it is done, we are ready to rock! We can test this change as we did earlier — by building the docker image, loading it on the kind cluster, and deploying it.

Hope this article helped in understanding how to create a simple Kubernetes controller and webhooks!