Member-only story
Airflow Series
Running Airflow Using Kubernetes Executor and Kubernetes Pod Operator with Istio
Overcoming obstacles
What is Kubernetes Executor
The Kubernetes executor runs each task instance in its own pod on a Kubernetes cluster. That means for each task in a dag, there is a separated pod generated in the Kubernetes cluster. The image of the pod is an Airflow base image. The pod will run your task, PythonOperator, BashOperator, etc. It basically runs the Airflow command “airflow task run …” using the Airflow base image inside the pod.
What is Kubernetes Pod Operator
Kubernetes Pod Operator also creates a new pod on a Kubernetes cluster, but the image is defined by you using the image arguments in the operator. It basically calls the Kubernetes API to launch a pod and run your image, and then record your result.
What is Istio
In most simple languages, it is a service networking layer. It controls how the traffic route through each service. As modern microservice architecture often consists of hundred or thousand of services, it needs another layer to handle all the network traffic. It is called service mesh.
Istio will inject a sidecar called istio-proxy inside your pod and handle the traffic for you. But this hinders the normal workflow of the Kubernetes executor and Kubernetes Pod Operator. It is because both of them use the Pod Phase to determine the status of the dag task. When the Pod Phase is successful and failed, Airflow will consider that the task is finished. Because of the istio-proxy sidecar, the pod will always be running. Just the task container itself is completed. This will lead to…
