Continuous Performance Improvement of HTTP API

We developed an analytics HTTP API in Python. This is how we discovered what to improve to serve requests faster

Burning down. Image licensed CC0.

In my previous post, I detailed a few code tricks to improve backend performance. How did I know where to focus and what to optimize, though? Indeed, joining Cython and other low-level gizmos to the party should have solid reasoning.

I work at Athenian. Athenian offers a SaaS that helps engineering leaders build a continuous improvement software development culture. We have pretty strict performance targets dictated by the UX. It’s hard to achieve great P95 response times without proper tooling. So we’ve wrapped ourselves with high-quality apps and services:

• Sentry Distributed Tracing allows us to investigate why a particular API request executed slow in production. This tool works in the Python domain.
• Prodfiler gives an independent zoom into the native CPU performance, including all the shared libraries.
• py-spy is an excellent low-overhead Python profiler by Ben Frederickson.
• Prometheus + Grafana help to monitor the immediate situation and trigger performance disaster recovery.
• Google log-based metrics augment the previous…