Better Programming

Advice for programmers.

Follow publication

Member-only story

Every Python Programmer Should Know LRU_cache From the Standard Library

Fabian Bosler
Better Programming
Published in
3 min readFeb 24, 2020

--

Image by Pexels from Pixabay

Not so long ago, I built an ETL-pipeline that was running daily, pulling data from an external service to enrich my input data and then loading the result into a database.

As my input data grew, my ETL-process became slower and slower because waiting for the responses from the external service was time-consuming. After some investigation, I realized I didn’t have that many different input values (~500) compared to the number of total records (~500k).

So, in other words, I was calling the external service with the same argument roughly 1000 times per record.

A situation like this is a prime use case of using caching. Caching a function means that whenever we calculate a return value of the function for the first time, we put the input and the result into a dictionary.

For every subsequent function call, we first check if the result has already been calculated by looking into our cache. If we find it there, perfect, no calculation needed! If not, we calculate the result and store the input and result in the cache, so that the next function call can look…

--

--

Fabian Bosler
Fabian Bosler

Written by Fabian Bosler

EX-Consultant turned tech geek! Business intelligence, marketing, advanced analytics, and machine learning. 👉 https://medium.com/@fabianbosler/membership 👈

Write a response