Meta-Programming in Python

A short tutorial on decorators and meta-classes

Recently, I encountered a very fascinating concept which is meta-programming in Python. I would like to share my findings on this topic in this article. I hope that it may help you to wrap your head around this because they say it is a tough nut to crack.

What Is Meta-Programming?

So, in one line: “Meta-programming is an act of writing code that manipulates code.”

Wait, what? Yes, you read it right. Code that manipulates code. Doesn’t it sound fascinating and powerful? Well, actually it is.

In the context of Python, meta-programming can be stated as: “Meta-programming is an act of building functions and classes that can manipulate code by modifying, wrapping existing code, or generating code.”

Meta-programming in Python can be achieved with:

1. Decorators
2. Meta-classes

Let’s get familiar with them one-by-one.

Decorators

A decorator is a way of adding new functionality to an existing function without modifying its original structure.

For instance, we have these three functions:

def add(x, y):
    return x + y    

def sub(x, y):
    return x - y
    
def mul(x, y):
    return x * y

Now we need to print the function name and parameter values when the function gets called. This should be applicable to all three functions above.

The native way is to add print/log statements to all three functions. But this sounds like very repetitive work and we’d also need to modify each function body.

def add(x, y):
    print("add is called with parameter {0},{1}".format(x,y))
    return x + y    

def sub(x, y):
    print("sub is called with parameter {0},{1}".format(x,y))
    return x - y
    
def mul(x, y):
    print("mul is called with parameter {0},{1}".format(x,y))
    return x * y    

print(add(5,3))
print(sub(5,3))
print(mul(5,3))

*********************** output *********************

add is called with parameter 5, 3
8
sub is called with parameter 5, 3
2
mul is called with parameter 5, 3
15

Can we do better? Of course we can, because by the grace of God, we are programmers and programmers are intelligent. We can achieve this by writing a decorator function and by not modifying any of the existing function body.

def my_decorator(func):
    def wrapper_function(*args):
        print("{0} is called with parameter {1}".format(func.__name__, args))
        return func(*args)
    return wrapper_function

@my_decorator
def add(x, y):
    return x + y
    
@my_decorator
def sub(x, y):
    return x - y

@my_decorator    
def mul(x, y):
    return x * y 

*********************** output *********************

add is called with parameter (5, 3)
8
sub is called with parameter (5, 3)
2
mul is called with parameter (5, 3)
15

Bingo! In the above code snippet, my_decorator is a decorator function. We decorate all three functions with @my_decorator and we have not touched the existing function body to add this print functionality.

So, basically, decorators are higher-order functions that take a function as an argument and returns another function. Here, my_decorator takes a function as an argument and returns wrapper_function as a result, where wrapper_function adds our print functionality to func.

There is more to decorators but this is a brief introduction to decorators in Python.

Meta-Classes

Now that we’ve seen decorators, they are for decorating functions. But there is more to meta-programming than decorators, such as meta-classes.

Meta-classes are special types of classes, rather than ordinary classes in Python. Where an ordinary class defines behavior of its own instance, a meta-class defines the behavior of an ordinary class and its instance.

A meta-class can add or subtract a method or field to an ordinary class. Python has one special class, the type class, which is by default a meta-class. All custom type classes must inherit from the type class.

For instance, if we have class Calc, with three class methods, and we want to provide debug functionality to all the methods in one class then we can use a meta-class for this.

class Calc():
    def add(self, x, y):
        return x + y
    
    def sub(self, x, y):
        return x - y
    
    def mul(self, x, y):
        return x * y

First, we need to create a meta-class MetaClassDebug, with debug functionality, and make the Calc class inherit from MetaClassDebug.

And, when we call any method from the Calc class, it will get invoked with our debug_function.

def debug_function(func):

    def wrapper(*args, **kwargs):
        print("{0} is called with parameter {1}".format(func.__qualname__, args[1:]))
        return func(*args, **kwargs)
    
    return wrapper


def debug_all_methods(cls):

    for key, val in vars(cls).items():
        if callable(val):
            setattr(cls, key, debug_function(val))
    return cls


class MetaClassDebug(type):

    def __new__(cls, clsname, bases, clsdict):
        obj = super().__new__(cls, clsname, bases, clsdict)
        obj = debug_all_methods(obj)
        return obj


class Calc(metaclass=MetaClassDebug):
    def add(self, x, y):
        return x + y

    def sub(self, x, y):
        return x - y

    def mul(self, x, y):
        return x * y


calc = Calc()
print(calc.add(2, 3))
print(calc.sub(2, 3))
print(calc.mul(2, 3))


**************** output ****************

Calc.add is called with parameter (2, 3)
5
Calc.sub is called with parameter (2, 3)
-1
Calc.mul is called with parameter (2, 3)
6

Bingo! In the above snippet, we created a meta-class MetaClassDebug and wrote a new method which is responsible for creating an instance of class and applies our decorator function debug_function to the object (instance), which will get created for every class that inherits MetaClassDebug.

Calc is inherited from MetaClassDebug, hence every method has been decorated by debug_function from debug_all_methods.

This way, we can add new behavior to all the methods within a class and also control the instance creation of a class using a meta-class. We can achieve a lot with a meta-class, such as adding a method or field to class, removing a method or field from a class, and many more.

I wanted you to have a quick look at meta-programming in Python, so I wasn’t able to cover all the things in this post.

I hope that this article has helped you to familiarize yourself with the concept of meta-programming. Criticism is always welcome!