Member-only story
An Introduction to Message Queues With RabbitMQ and Python
Write your first event-driven workers
This article was originally published on my personal blog here.
What Is a Message Queue in the First Place?
Message queues (MQ) are a fundamental concept in programming and software development. In a distributed system, a message queue is the backbone of the system. A message queue allows inter-process communication between services/applications in your system (eg. Service A can talk to Service B).
In MQ terminology, the service that emits the message is called the producer worker, while the service that listens and reacts to messages is called the consumer worker. This is how communication happens between the services.
You can scale up or down the number of producer and consumer workers running depending on their loads. For example, you may have two producers running in two VMs and ten consumers running CPU-intensive tasks across 10 VMs. You can also increase the number of workers during the day and shut down the workers at night (provided that your application traffic is a diurnal pattern).
Imagine a World Without Message Queues
Without a message queue, your system runs synchronously. Even though synchronous programming comes with ease in mind, it’s not really convenient for the end-user experience. Let’s take an example of signing up for an online service. Once you’re done filling up your details and press the “Sign up” button, the system will send you an email and create a database row for you. Let’s say you put a wrong, unreachable email address. The system will retry to send you the activation email for maybe ten seconds. As the new user, you will have to wait for at least ten seconds for the system to finish attempting to send an email before you’re redirected to the next page. Isn’t that a bad user experience? Waiting is a very big deal for a web app!
Here is the ideal scenario is: You press the “Sign up” button, you are redirected into the homepage, and you wait for the activation email to reach your inbox. In that case, you do not have to wait for the system to successfully send you the email. This is because there is a…
