7 Advanced Python Concepts You Might Want To Know

Multiple inheritance, metaclasses, and more

Photo by Chris Ried on Unsplash.

Python is the most popular programming language in the world. Most applications only need a small subset of Python’s diverse arsenal. However, if you want to become a true Python expert, you should learn and master these concepts as well.

Multiple Inheritance

Classes can inherit from more than just one main class. They can inherit from multiple. This allows you to combine the strengths of multiple base classes to create a very diverse and powerful class.

Multiple inheritance

While true multiple inheritance (i.e. inheriting from two or more fleshed-out classes) is frowned upon, inheriting from one base class and then inheriting some other very specialized classes is often used. These specialized classes are called mixins and often only work when combined with certain base classes.

A framework that uses mixins very well is the Django Rest framework. You take one of the base View classes and then you can easily add GET, POST, or PUT mixins.

Multiple inheritance in Django Rest framework. Source: Django Rest framework

The ListModelMixin and CreateModelMixin will have the logic to get and update the correct data in the database based on the Django model — and just with those two words for inheritance.

One thing to keep in mind is that any resolution will be done from left to right. So if a method is defined both in BaseClassA and BaseClassB, it will take the method from BaseClassB, as it is defined first. This is the same for any super() call in the class. This method resolution is the reason that inheritance of multiple expansive classes is discouraged, as it can create a lot of confusion. Keep it to one base class and several specialized mixins.

Dunder Methods

Dunder methods are Python methods that start and end with a double underscore. These are also sometimes called magic methods. You can find a complete list in the docs.

Some dunder methods are well known, like __init__ or __str__. But there are so many others that are worth looking at.

Call

__call__ allows you to execute an object as a function. This is useful if you have some default functionality to use in the class or when you want to use methods and objects interchangeably.

__call__ example with the function element providing the rounding amount

With the call method, you can combine easy access to attributes of objects with the simple use of functions.

Operators

You can also overwrite any operators you like, such as add, subtract, multiply and divide. There is also power, floored division, matrix multiplication, and modulo (@). The comparison statements like greater than, smaller than, or equal can also be implemented. Lastly, you have the logical operators like and, invert, and or. Keep in mind that the __and__, __invert__, and __or__ methods refer to bitwise ands (a & b), inverts (~a), and ors (a | b). The and, not, and or statements will convert the object to a boolean and then evaluate the expression.

Operators magic methods

There is a corresponding __r(method)__ implementation for a lot of these methods. This is used when the left-to-right implementation is not implemented. This would be called when we would do 5 + a. That is why it’s important not to error out but to return NotImplemented. Something similar also exists for __i(method)__ when the calculation would be done in place, such as a += 5.

Overwriting these magic methods is very popular in libraries. In NumPy, the matrices overwrite the standard implementations to provide powerful and concise usage of matrix calculus. In Django, ors and ands can be used to combine QuerySets or filters.

NumPy and Django — Usage of magic methods

Generators

Generators are objects that can be iterated upon. The most common example is the range function. The benefit of using generators is that you don’t need to keep the whole data structure in memory. The generator just keeps track of what is needed to provide the next value.

Generators come in two types. The first type is a function that contains the yield statement. The second type is an object that has implemented the __iter__ and __next__ methods.

Any function that contains the yield statement will become a generator. Iterating over the function will execute the function until the yield statement and return that as the first value. For the second value, it will continue from the keyword and carry on until the next yield. This will happen until the function ends or the StopIteration error is raised.

Function generators

Object-based iterators need two methods to work: __iter__ and __next__. __iter__ will initialize the iteration and needs to return the generator (generally itself). __next__ will be called to get the next value.

Object generators

You have to keep two things in mind with generators. Once iterated over, they must be initialized again. You cannot use them to iterate over twice. Secondly, you can also manually iterate over them using the built-innext function.

Context Managers

Every self-respecting Python developer knows how they need to work with files. Either you make sure to close the file after opening it or you use the with statement. The with statement is called a context manager. It can be used anytime you have something you need to initialize when starting or clean up after you stop using it — especially when the cleanup also needs to happen when an error occurs.

Creating your own context manager is easy. You need to implement the __enter__ and the __exit__ dunder methods. Then you can use the with statement to make sure you safely clean up after yourself. The value returned by the enter method is given to the as statement.

Context managers

These are great for I/O operations, as you want to make sure that you close the used resources afterward. This will avoid dangling connections and open files.

Lastly, the exit method can also handle raised errors. The type, value, and trackback are all error information. You can use these to handle the exception. If you return a True value from the exit method, Python will not re-raise the exception and continue as normal.

A generator function can also be used as a context manager with the yield keyword if you use the contextlib.contextmanager decorator. But I’ll leave that to you to figure out.

Asyncio

asyncio is the Python library allowing your code to run asynchronously. Make sure to know the difference between threading, async, and multi-processors before starting with this. I suggest reading this article. In short, asyncio allows you to process other stuff while you are waiting for slow operations (generally IO, databases, or networking). If you don’t have any of these operations, asyncio won’t improve code performance.

Implementing asyncio is very easy. You can make any function asynchronous by putting async in front of the definition. When you call the function, it will return a Future object. And to get the result of that future, you need to await it. Python will schedule the right execution of Future objects by executing other code while it is waiting.

Asyncio

Here, we get some records from the database. As long as we have an asyncio-compatible database, it will be able to get the records in parallel. We initialize all the requests first before we await them. While the code is waiting for one row, it will already request the next one, and so on.

To compare, we also have a bad example that does a list comprehension with await. This will not improve performance, as the event loop will wait until the previous record is fetched before requesting the next one. So make sure to use gather or similar methods to improve performance.

With asyncio come some extra under methods. You can implement the await magic method to make an object awaitable. You can use __aenter__ and __aexit__ for asynchronous context managers and implement __aiter__ and __anext__ for async iterators.

__new__ and Object Initialization

The __new__ dunder class method is called whenever you create a new instance of a class before __init__. It is a class method and should return an instance of the class. You can control the instance that is returned when an object from the class is created. You can even return another class object! __init__ will be called on the instance you return.

The new dunder method

While often unconventional and even controversial, this method does have its uses. The most commonly provided example is a Singleton implementation. Using the class as a base class will make sure only one of them will be initialized at a time. Singletons can also be enforced in other ways, but I feel like the __new__ method provides a very convenient implementation.

A second example is similar to the factory pattern in other languages. Based on the input, the new method can return the appropriate class. As with the previous example, a factory object or static function could also be used with the same outcome. But I find the new method quite a clean and concise solution for the same problem.

Metaclasses

Metaclasses take classes one step further. They allow you to adjust the attributes or methods of your class dynamically. Text won’t do metaclasses justice, so here is an example:

Metaclasses for the environment

The Environment class will load the environment variables in a useful class. It works well with typing hints and avoids using the same code of loading environment variables over and over again. You can even set defaults or use it to cast values to the right type!

To define a metaclass, you need to set it with named inheritance. The output of the new function must be a class (which is a metaclass instance). It will define what the class is and how it behaves.

The input arguments are:

• The metaclass (as __new__ is a class method by default)
• The name of the (non-meta) class
• The subclasses/bases of the (non-meta) class
• The namespace of the non-meta class

The namespace is the interesting part here. It will contain all fields defined on the (non-meta) class if they have values. It also provides some other info such as annotations. Here, we use the annotations to set environment variables to the class attributes. With some extra logic, you could also cast them to the right type or raise appropriate errors.

For another example, Django uses this on its models to convert the model attributes to its relevant values from the database. And Pydantic uses a similar approach to retrieve the annotations of its models.

A very important note, though: The metaclass new method is executed as soon as the class is defined and not on every object initialization. So make sure your class is complete as written and the metaclass does not need extra information afterward.

Conclusion

Not all of these are equally useful. Metaclasses, in particular, are not commonplace — and for good reason. And you can be a serious programmer without having in-depth knowledge of all these concepts.

But I think this article provides you with the options to get it done better sometimes. Understanding Python on a deeper level allows you to understand what is happening, and you will also be able to better understand some advanced libraries (e.g. Django, SQLAlchemy, and Pydantic) that make heavy use of these advanced methods.