Member-only story
Create Zero-Point Failure Distributed Tasks With Python and ZeroMQ
A guide on how to build a distributed pub-sub architecture without a broker
A common solution for facilitating communication between disparate application services is to use a message queue. Most traditional message queues act as hubs to facilitate the exchange of messages over the network between distributed software components. This is usually done through applications via a centralized queue using an API.
Recently I started exploring a ZeroMQ (aka ØMQ, 0MQ, or zmq), an opinionated, lightweight library that bills itself as a high-performance, asynchronous messaging library that goes without a middleware broker, aimed for use in distributed or concurrent applications.
When you think of communication between microservices or services, for some of us, the first thing that comes to mind is a REST API with service discovery or message brokers that facilitate intercommunication between services. ØMQ takes a different approach and goes back to the networking roots.
In this piece, we will be looking at
- What ØMQ is
- Exploring ZeroMQ design patterns
- Quirks and issues ØMQ presents when building distributed applications
- How to distribute your Python tasks, with a peek at the possibilities
Wrapping Your Head Around ZeroMQ
To paraphrase what the ØMQ docs have to say when describing this library
“ØMQ is a ground-up redesign of messaging based on specific design principles of uniformity, scalability, and interjection, and inspired by the Internet Protocol (IP).”
ZeroMQ is a library that allows you to perform low-level message passing, but unlike message-oriented middleware, an ØMQ system can run without a dedicated message broker. To understand ØMQ, you need to think in the sense of network sockets that carry atomic messages across various transports. You can connect network sockets N-to-N, 1-to-N, with patterns like fan-out, pub-sub, task distribution, and two-way…
