Member-only story
When DynamoDB is 10x Cheaper Than S3
Surprisingly enjoying both low latency and low costs with DynamoDB
As big-data architects using managed AWS solutions, we tend to map storage technologies in our heads into rules of thumb. S3 will be in the category of “cheap and slow” while DynamoDB, relatively speaking, will sit in the “fast and expensive” category. Well, in certain use cases as I discovered, you can get cheap and fast using DynamoDB.
Eat the Cake and Leave it Whole
In our event stream processing system, we process 300 million events per day that are ingested from a source queue.
Our service runs some business logic that includes transformation and enrichment on the source events and in some cases outputs a message to a destination queue.
Each event has a unique GUID and the first requirement is to store them, after the transformation, in a way we could later retrieve them by their GUID.
The second requirement is that those retrievals could only occur on events that were ingested in the last 30 days.
Sounds easy right? We basically need a key-value store that would map the GUID to the transformed event with a 30 days retention.
Let’s break down the costs of doing this with S3 VS. the costs of using DynamoDB.
I’ll start with the pure managed costs, i.e., how much would we pay AWS, and then take a broader look at more hidden costs (development, maintenance par complexity, etc.).
All costs will be calculated based on the us-east region.
First, we need to make an assumption about the average event size we process.
In my use case it was 5 KB, but after GZIP compression I got it to 1.6 KB. Compressing the events and calculating costs based on the compressed size is something many engineers tend to miss, and it could easily shift the pendulum from one technology to the other.
