Member-only story
Create Avro Producers With Python and the Confluent Kafka Library
A command-line executable Python script to send an Avro record to a Kafka topic
Overview
In this tutorial, we will learn how to write an Avro producer using Confluent’s Kafka Python client library.
The script we will write will be executable from the command line and takes a few arguments as its input. Writing it as a command-line executable gives us the flexibility to call it from anywhere we want.
The code for this tutorial can be found in this GitHub repo.
Dependency
Firstly, let’s create the requirements.txt file and write the packages we need for this project. There’s actually only one package that we need:
Then, let’s set up and activate our virtual environment so the packages we will install are isolated just for our project. For this tutorial, virtualenv is used.
Assuming you already have virtualenv installed, let’s execute the following commands from the terminal:
~ ❯ mkdir python-avro-producer~ ❯ cd python-avro-producer~/python-avro-producer ❯~/python-avro-producer ❯ virtualenv ./venv~/python-avro-producer ❯ source ./venv/bin/activate~/python-avro-producer (venv) ❯
Now, let’s go ahead and install the dependencies we need:
~/python-avro-producer (venv) ❯ pip install -r requirements.txtAvro Producer
Alright, now the fun part begins!
To make this tutorial easy to follow, we’re going to break it down into a few sections and each will discuss one…
