Use Perflint — A Performance Linter for Python

Detect performance problems early on and learn how to write better-performing code on the way

Photo by Chander R on Unsplash

In contemporary software development, performance takes a front sit, both in the testing efforts but also in the development process it self.

In this article I’ll demonstrate how you can easily integrate performance linting into your python application development and come up with a better code.

Performance Engineering

Performance engineering is a growing field of software engineering where performance considerations are being proactively accounted for.

A more intuitive approach to performance would be to run performance tests during the end of the project and make this performance ends meet after all the functional requirements are fulfilled.

However, this approach appears to be ineffective and costly.

A recent ethnographic study, published in the Empirical Software Engineering Journal, was aimed to explore the common practices of performance assurance.

Having several tech professionals continuously interviewed for 6 months, the researcher concluded that:

The study shows that the case organization still relies on a waterfall-like approach for performance assurance. Such an approach showed to be inadequate for ASD[Agile Software Development], thereby leading to a sub-optimal management of performance assessment activities. We distilled three key challenges when trying to improve the performance assurance process: (i) managing performance assessment activities, (ii) continuous performance assessment and (iii) defining the performance assessment effort.

This illustrates the importance of performance assurance taking precedence in the life cycle of software development and adherent to agile methodologies instead of the waterfall approach.

Premature optimization

The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.

This quote is taken from the 1968 book ‘The Art of Computer Programming’, written by the 1974 Turing award laureate Donald Knuth.

Knuth argued (remember, this quote is over 60 years old…) that programmers tend to think intuitively about performance.

They would typically assume that the parts of code that are harder to understand are the parts of code that would be the most time-consuming for the machine to execute.

And overall his statement was meant to caution programmers from blindly optimizing their code without profiling.

Many programmers though take his statement out of context and interpret it as if any efforts to optimize a program must take place at the final stages of application development and this argument is just false.

Performance optimization is part of software design and implementation, and it, like all other software development efforts, should be priorities in terms of risks vs benefits.

Introducing perflint

I talked about linters in general and pylint on a previous article.

Performance linter is a more specific linter, aimed to detect performance anti-patterns in the code.

As I demonstrated in my previous article, pylint can be very easily extended with plugins to add more checkers functionality.

So the awesome python developer and author Anthony Shaw, has been developing a plugin to extend pylint to detect performance issues.

Using this linter, python developers can boost their application performance and also learn about the python internals and how it affects performance.

Hands-on

Let's take a look at a not very useful piece of code:

So it has a global variable set to 2, the main function and the main clause.
The main function has a local list of integers variable, it iterates over the list and prints each of these numbers powered by the global variable.

Let's run the code:

python example.py

Output:

1
4 
9 
16

Now lets lint this code, first, we need to install pylint using pip:

pip install pylint

Now let's run pylint:

pylint example.py

And the result is:

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

No errors were found and my code scored 10/10.

Now let's add perflint to the party :)

pip install perflint

Now we can run pylint and use perflint as a plugin!

pylint --load-plugins perflint example.py

Result:

************* Module __main__
myapp\__main__.py:8:27: W8202: Lookups of global names within a loop is inefficient, copy to a local variable outside of the loop first. (loop-invariant-global-usage)
myapp\__main__.py:6:16: W8301: Use tuple instead of list for a non-mutated sequence (use-tuple-over-list)
------------------------------------------------------------------
Your code has been rated at 7.14/10 (previous run: 5.00/10, +2.14)

Perflint found 2 issues and my code score dropped to 7.14!

Let's examine these two issues:

Issue 1 — Lookups of global names within a loop

Let's focus on the loop, we are iterating over the list, and then we use the global variable.

Lookup for global variables requires the python interpreter to perform a lookup over the entire realm of names and indexes, this is the STORE_NAME opcall of the python bytecode interpreter

However, when you define a function, it has a fixed number of local variables, so that python can set a fixed array and easily find the value associated with the local variable name, this is the STORE_FAST opcall of the python bytecode interpreter.

So a quick fix for that would be to store the global variable in a local variable and then we can use that local variable inside our loop.

Now lets perflint again:

pylint --load-plugins perflint example.py

Result:

************* Module __main__
myapp\__main__.py:6:16: W8301: Use tuple instead of list for a non-mutated sequence (use-tuple-over-list)
------------------------------------------------------------------
Your code has been rated at 8.75/10 (previous run: 7.14/10, +1.61)

Cool, the first problem is solved.

Now let's fix the second issue.

Issue 2— Use a tuple instead of a list

The list variable in our code never mutates, so it is advised to use immutable objects in this case.
The preferred immutable object here would be a tuple

So let's fix this problem:

and now perflint:

pylint --load-plugins perflint example.py

Result:

-------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 8.75/10, +1.25)

Perfect score!

Bonus

Let's make a comparison between fast and slow examples of our code:

Using the timeit we can evaluate how much faster the ‘fast’ version really is:

Run:

python example.py

Result:

fast_main execution time = 7.155288799898699 usec
slow_main execution time = 7.3095553000457585 usec

7.15 useconds for the ‘fast’ version vs 7.31 useconds for the ‘slow’ version, quite noticeable.

Thanks for reading.