Magical Metaclasses

  • python

Continuing on from the previous post about decorators, we should visit metaclasses before we get to the good stuff.

For those who haven't worked with them, metaclasses are essentially invisible decorators for classes.

The metaclass describes how your class is made. You define a base class with a custom metaclass, then every class which inherits from your base class will have the same metaclass.

Whereas a normal class's __init__ method will let you control what happens when your class is instantiated, the __init__ method on a metaclass will let you control what happens when your class is defined:

class MyMetaclass(type):
    def __init__(self, name, bases, dct):
        super().__init__(name, bases, dct)
        print("MyClass defined")

class MyClass(metaclass=MyMetaclass):
    def __init__(self):
        print("MyClass initialised")

This means you can manipulate class attributes and methods just after it has been defined. This can be useful for logic to set and update defaults, or enforce restrictions like checking or validating required class attributes.

As an aside, Python 3.6 introduced the class method __init_subclass__, which is called when someone subclasses that class - essentially a shortcut for defining __init__ on a metaclass, and thus covers 95% of the use cases for metaclasses. In other words, if you're reaching for a metaclass, you probably want __init_subclass__ instead. But in these articles we're working towards the 5% of use cases that it doesn't help with, because they're more fun, so we're going to ignore it and focus on metaclasses.

Metaclasses also have a __new__ method, which lets you manipulate the class before it is defined. This is a lot more fun - it lets you do things like dynamically define class attributes and methods, or strip off attributes and use them to build something far more complicated before the class exists:

class MyMetaclass(type):
    def __new__(cls, name, bases, attrs):
        meta = attrs.pop("Meta")
        if meta and isinstance(meta, type):
            print("This had a `class Meta` definition, but doesn't now")
            attrs = do_something_exciting(attrs, meta)

        # Create the class with the modified attributes
        new_cls = super().__new__(cls, name, bases, attrs)
        return new_cls

One common use of a metaclass is for automatic class registration - we want a list of all subclasses of our class.

First create our base class with our metaclass, but as it isn't going to do anything itself we don't want it in our registry. Lets set an attribute abstract=True, then look for that in our metaclass when we're trying to register. We want to work with the class once it has been defined, so we'll put this in the metaclass's __init__ method:

registry = {}

# Metaclass
class RegistryType(type):
    def __init__(cls, name, bases, attrs):
        super().__init__(name, bases, attrs)
        if attrs.get("abstract", False):
            # cls.abstract is True, so don't register
            return

        # cls.abstract is either False or missing, register
        registry[name] = cls

# Our base class - we don't want to register this
class RegistryBase(metaclass=RegistryType):
    abstract = True

# User's subclass - we do want to register this
class First(RegistryBase):
    """Not abstract"""

This is a pretty common pattern (or at least was until __init_subclass__ arrived), and you've probably come across it without realising. A great example is in Django's models, which looks something like this:

from django.apps import apps

class ModelBase(type):
    def __init__(cls, name, bases, attrs):
        super().__init__(name, bases, attrs)
        if not attrs.get("abstract", False):
            apps.register(name, cls)

class Model(metaclass=ModelBase):
    abstract = True

class Cat(Model):
    name = models.CharField(...)

Obviously this is very simplified, but this is essentially what you'll find if you dig through django.db.models.base.

This works well for Django, as it can register the models for relationships and migrations; and it makes things simple for the user, who doesn't have to worry about manually registering their model classes.

But why not register them manually? After all, Django does that elsewhere - the admin for example:

# Register with a function call...
class CatAdmin(admin.ModelAdmin):
    ...
admin.register(Cat, CatAdmin):

# ... or using a decorator
@admin.register(Dog)
class DogAdmin(admin.ModelAdmin)

The main reason: it saves the user a line of boilerplate. The actual metaclass does a lot more, but it could be written as a decorator or function call. The metaclass is a sprinkling of syntactic sugar to save the user from having to remember to do something, which sounds great - the less code you need to write, the less code someone else needs to read, and the less scope for mistakes. And it is great for Django's models. But if it's so good, why not use it everywhere?

The problem with metaclasses is they are a bit too magical for every-day use. Without knowing what we've just talked about, from a standard model defininition you'd have no idea how Django is going to find out about your model.

More importantly, it's not clear how you'd control registration. For a start, you don't necessarily want everything to register with the same thing; it would be awkward when you want to register a ModelAdmin against multiple admin sites, for example. When you introduce a metaclass to your code, you need to be confident that your users will never need to work around the defaults and understand the "how". This is why the Zen of Python discourages this sort of thing in the first place.

Metaclasses exist to hide explicit logic - in this case, registration is now implicit, done entirely behind the scenes. You define a model and suddenly Django just magically knows about it, and if you have a problem with that, good luck - but as nobody sensible has a problem with how Django's models work, they're a perfect fit.

Normally. If you're sensible. This brings me onto the topic for next week: nanodjango.

Leave a comment