Member-only story
Build a Cloud-Native Multiprocessing Framework
How to convert a local multiprocessing framework to the cloud
Cloud technology continues to boggle my mind. The power, the ease, and the cost are all favorable — workloads complete 100 times faster, web interfaces and programmatic interactivity provide means for the novice and the expert, and free tiers, training, and examples flatten up-front costs and learning curves. There are many cloud platforms, Amazon Web Services (AWS) is likely the best place to start.
Table of Contents
- Review
- Background
- The Two Services
- Getting Started with AWS
- Setup SQS
- Setup Lambda
- Summation
1. Review
In this article, we are going to walk-through how to create a framework for multiprocessing in the cloud. I wish to extend the content from a previous article I wrote where I discuss multiprocessing on a PC or laptop. You may find reading that article first gives you added context, it is called A Simple Multiprocessing Framework Within Python.
2. Background
AWS provides two services when used in tandem act as a multiprocessing engine which can be fully automated — and if your jobs are small enough, then you won’t pay a dime because of their free tier.
Multiprocessing on a PC
Let’s say your laptop or PC has, at best, 4 cores and 8 logical processors. This would allow you (if you don’t do anything else on your computer at the same time) to run 8 tasks simultaneously. You are therefore limited to logical processors. Not to mention, you may also run into memory (RAM) constraints based on what you are trying to achieve. Let’s assume your PC also has 8 GB of memory.
Multiprocessing in the Cloud
Regarding “processors” and RAM in the cloud, you are limited to 1K “processors” running simultaneously and 10 GB of memory per “processor” — that is 10K GB. You cannot match that anywhere.
