Member-only story
Build Better Serverless APIs By Going Storage First
Serverless offers many options when it comes to API design. Storage first is an async pattern that is changing the game
Last week my team was having a fiery debate. They were talking about the usage of VTL exclusively for APIs.
One side of the argument debated that it enabled you to get the fastest possible response times for your API. The other side argued that it is near impossible to troubleshoot if an error occurs in your integration. The data is lost and you are forced to rely on API Gateway logs (which is a bit of a treasure hunt to set up).
Both sides make great arguments. But there has to be some middle ground somewhere. You want as low API latency as possible, but you absolutely do not want to lose any data.
Good news, there is a compromise.
Eric Johnson has been an advocate for the storage first pattern for years. Storage first refers to a set of serverless design patterns that get your data to land somewhere before running compute. That way if something goes wrong you have the ability to retry or send to a dead letter queue for manual processing in the future.
2022 is the year of async. We’re putting our focus into making faster, more reliable, idempotent APIs by embracing asynchronous architectures and patterns in our applications. By going storage first, we take a huge step in that direction.
Start With Direct Saves
Step one in a storage first pattern is to save the incoming data to a persistent location. With AWS, there are many options to choose from. You can go straight to DynamoDB, Amazon Kinesis, SQS, and even EventBridge. Each one of these services can keep a record of the incoming event so if something does go wrong, you can replay it.
Our example today is going to do a direct save to DynamoDB. The save will trigger a stream which starts a Step Function.
By going to DynamoDB, we know our data will not be lost. It goes straight to storage by proxying straight from API Gateway to Dynamo. Unfortunately this does use VTL, but just a little bit. Once HTTP APIs support integrations directly to Dynamo…
