Member-only story
How One Rogue User Took Down Our API
And how to prevent it from happening to you
It was 1 a.m. on a dreary December morning, on the cusp of a code freeze for the holiday season, when my team received the page. We were alerted that the API for our CDN product was experiencing a partial outage that was impacting all of our users.
If you’re a software engineer, reading that intro paragraph probably gave you heart palpitations. You aren’t the only one.
For any readers who aren’t aware, CDN stands for Content Delivery Network, and they are crucial for eCommerce businesses. They allow websites to cache frequently accessed assets — such as pictures, videos, or JavaScript files — to make the website more performant and avoid downtime.
Websites like Amazon lose hundreds of thousands of dollars every minute it’s unavailable, and even more than that during the holiday season. As a CDN provider, our customers depend on our product for their own success.
Hearing that our API was nearly unusable just before the beginning of the holiday season was serious cause for concern. With our pulses racing, we took a deep breath and got to work.
With bloodshot eyes and double shot espressos in hand, we began investigating where our API woes were coming from. The first thing we noticed was that the errors were originating from the Purge endpoint. This endpoint allowed our users to flush any and all items from their CDN cache.
On a normal day, this endpoint received a fair amount of traffic. On this night, however, traffic had spiked. But the most frightening and intriguing aspect of it was that nearly all of the API requests were originating from a single user. They were patient zero.
But as you will come to see, it wasn’t as simple as pointing a finger at this rogue user and declaring them the stem of all of our troubles. In reality, they were merely the catalyst to a situation that would soon spiral out of control.
The real culprit was the assumptions we made about our users. And to truly understand how my team had gotten…
