Radiac's Blog: python

Monkeypatching Django

Wed, 15 Jan 2025 08:30:00 +0000

I haven't yet posted here about my project nanodjango. If you haven't heard of it yet, it is a package which lets you write Django in a single file. I gave a lightning talk at Djangocon US and have written an introductory blog post over at Lincoln Loop, if you want to find out more from a user's perspective - but here I'm going to talk about how it works.

A couple of years ago I took over yet another project where the previous developers had heard "Flask is easier than Django", and I saw they had spent a long time and a lot of money accidentally building their own poor version of Django, burying the project under years of technical debt. It became clear why they had handed the project off saying they couldn't support it any more - even simple changes required hours of wading through spaghetti code, and upgrading the very outdated packages was daunting, if not impractical. We got frustrated working on it, the client got frustrated paying so much for small changes, and it became clear the only practical option was a rewrite - which the client couldn't afford.

Choosing Flask had been a mistake. Don't get me wrong, Flask has its place, but you need experience to make good decisions that will help your project grow - experience which the original developers didn't have. Django's startproject may be offputting to beginners, but it does force you to follow a consistent structure which scales and makes it easier to hand off to another team.

And when your project grows to need a database, forms or admin, Flask devs will be rolling their own solutions or pulling in third-party libraries with varying levels of support and compatibility, while Django developers will be using core batteries which will always work for the latest version of Django, and we won't look to pull in third-party libraries until we get to much higher-level functionality. This leads to a more stable platform to develop against, and for those reasons I'm firmly of the opinion that Django is the better option for complex projects.

But I thought to myself, why can't Django be a good fit for smaller projects and prototypes too? Why can't Django work like Flask, in a single file?

Django in a single file in itself isn't particularly novel or special - boiled down, Django just routes requests to functions which return responses, so getting it to handle views in a single file had been done, and done well - first in 2009 with djing (Simon Willison) and Django Inside Tornado (Yann Malet), and more recently Using Django as a Micro-Framework (Carlton Gibson), django-microframework (Will Vincent and Peter Baumgartner), μDjango (Paolo Melchiorre), django-singlefile (Andrew Godwin) and Django from first principles (Eric Matthes) - and many more in between.

Most of these do what Flask does - they implement views and routing, and sensibly stop there. I enjoy doing silly things with Python though, so wanted to see if I could give you access to all of Django's batteries from a single file - the bits which expect you to structure things properly in separate modules, so they can control what order things are loaded in to let them sprinkle the syntactic sugar that makes Django so great. The things that use metaclasses and implicit registration. I wanted models.

This was fun. Models need to register with your app in the apps registry, so you need to get the current module into INSTALLED_APPS - but you can't just add the current module to INSTALLED_APPS to get it in there, because your module hasn't finished importing. If it hasn't finished importing, the apps registry can't resolve a reference to it, and it will try to load it again - putting you into an infinite import loop.

I tried various silly things, like monkeypatching Django's models and apps to delay model registration until import was complete, disabling them completely and parsing the AST to rewrite the single file into a proper app structure in-memory on demand, but then I realised I could just trick apps into not importing the model if it already looked like it was imported.

The first problem is that Django expects to load apps itself during setup, but we're going to run setup from within our app, so we need to make Django think we've already loaded it. To do this we need an AppConfig with a hard-coded path:

class NanodjangoAppConfig(AppConfig):
    path = str(get_script_path())

and then we need to manually add that to the app registry, before we call django.setup(), which will in turn call apps.populate() to pick it up:

app_config = NanodjangoAppConfig(app_name=get_script_name(), app_module=get_script_module())
apps_registry.app_configs[app_config.label] = app_config
app_config.apps = apps_registry
app_config.models = {}

That's solved app load order - we've now force-registered an app using our script name, pointing at our script module, and it knows where to look for anything it might want to look at. In nanodjango, that code is run as part of the app = Django() intialisation step, just before it calls django.setup() - which is why models can't be defined in Nanodjango before the app object exists.

But now we've got another problem - Django's model metaclass magic has expectations which we don't want to meet - in particular, it looks for the app name in MyModel.__module__ - but because of how we're running our script, it will get the string __main__, rather than the name of the app we're trying to dynamically create.

The metaclass then uses this module name to do a lookup in the apps registry, but of course doesn't find anything - we registered our AppConfig under a different name - so we'll get a fatal error saying our app __main__ isn't in INSTALLED_APPS.

To solve this, we need to change the behaviour of Django's ModelBase metaclass. If you need a refresher, we talked about metaclasses last week.

For those who haven't come across it before, monkeypatching is the practice of changing other people's code at runtime. Python makes this possible because everything is an object - we can put a reference to a function in a variable, and overwrite the original just as easily.

The most common time to do this is during testing, where you want to make a temporary change to stub something out - for example, this would be a way to ensure that os.path.exists says that every file starting test:// exists:

import os

# Create a reference to the old exists() method
old_exists = os.path.exists

def fake_exists(path):
    # Our override logic
    if path.startswith('test://'):
        return True

    # Otherwise fall back to the old behaviour
    return old_exists(path)

# Overwrite the exists() method with our new function
os.path.exists = fake_exists

The other reason is to make your code work with someone else's - their code doesn't do what you want, but for whatever reason you don't want to fork it or submit a PR. That's what we have with nanodjango.

Essentially all we're doing is overwriting a variable so that when other code tries to use it, it finds our new value or function which does what we want, but this introduces potential problems.

You do have to be careful not to break other people's code. In a test we can get away with functions not doing what they were originally meant to (either intentionally or accidentally), but if you do it in production code things are going to go very wrong very quickly.

It's important that you write unit tests to check that the old and new functions behave the same under normal circumstances - if nothing else, it will tell you when the upstream package changes. You also have to be extra careful that your code doesn't introduce bugs or exceptions - monkeypatched code can be a pain to work past in a stack trace.

But as long as you're aware of the risks, and you take care around them, monkeypatching can be another powerful ally when trying to make your code easier to use.

Which brings me back to nanodjango's problem with the ModelBase metaclass. If we look at the source, we'll see the error comes from ModelBase.new:

class ModelBase(type):
    def __new__(cls, name, bases, attrs, **kwargs):
        ...
        if getattr(meta, "app_label", None) is None:
            if app_config is None:
                if not abstract:
                    raise RuntimeError(
                        "Model class %s.%s doesn't declare an explicit "
                        "app_label and isn't in an application in "
                        "INSTALLED_APPS." % (module, name)
                    )

            else:
                app_label = app_config.label
        ...

The app_config is None, it's not abstract, and we don't have an app_label in our class Meta definition, so we get the error. That last condition looks promising though - could we just tell nanodjango users that every model they define needs an app_label?

class MyModel(models.Model):
    ...
    class Meta:
        app_label = "myscript"

If you try it you'll see that would work, but it's messy - I don't want to write that every time I define a model, and neither will my users. Monkeypatching to the rescue:

# Collect a reference to the old __new__
old_new = ModelBase.__new__

def new_new(cls, name, bases, attrs, **kwargs):
    # See if this is a nanodjango model, if it is defined in __main__
    module = attrs["__module__"]
    if module == "__main__":
        # It's a nanodjango model, so let's say it's from our new module...
        attrs["__module__"] = app_name

        # ... update a Meta class if it exists ...
        attr_meta = attrs.get("Meta")
        if attr_meta:
            if not getattr(attr_meta, "app_label", None):
                attr_meta.app_label = app_name

        # ... or create one if it doesn't
        else:
            class attr_meta:
                app_label = app_name

            attrs["Meta"] = attr_meta

    # Call the original ModelBase.__new__
    return old_new(cls, name, bases, attrs, **kwargs)

# Swap our function in so any new model will be created using our code
ModelBase.__new__ = new_new

This is the actual code from the current nanodjango patch_modelbase() function. If you look in that file you'll notice that I like to put my monkeypatches in functions which need to be called explicitly, rather than applying themselves during import - it makes it slightly clearer when and where things happen.

The approach I've taken here is very defensive - I only make changes if I know it's something I'm in charge of (a model defined in the __main__ module can only ever come from nanodjango), and even then I'm careful not to disturb any Meta.app_label value which may already exist.

I disturb the original code as little as possible - I was lucky here that I found a route to fix it by setting Meta.app_label before it's checked; that's not always going to be the case, but you can usually find a way to accomplish your goals while still calling the original function. What you don't want to do is rewrite and replace the original function entirely - you could if you have no other option, but then you'll own it forever, and have to keep a much closer eye on it every time a new upstream version is released.

If you dig in nanodjango's monkeypatch file you'll spot there's also a patch to get migrations working - and you'll see it uses a very similar approach.

So that's metaclasses and monkeypatching. But that's still pretty tame, and I promised you dark secrets. Next week it's time to look at my tagging library, django-tagulous.

Magical Metaclasses

Wed, 08 Jan 2025 08:30:00 +0000

Continuing on from the previous post about decorators, we should visit metaclasses before we get to the good stuff.

For those who haven't worked with them, metaclasses are essentially invisible decorators for classes.

The metaclass describes how your class is made. You define a base class with a custom metaclass, then every class which inherits from your base class will have the same metaclass.

Whereas a normal class's __init__ method will let you control what happens when your class is instantiated, the __init__ method on a metaclass will let you control what happens when your class is defined:

class MyMetaclass(type):
    def __init__(self, name, bases, dct):
        super().__init__(name, bases, dct)
        print("MyClass defined")

class MyClass(metaclass=MyMetaclass):
    def __init__(self):
        print("MyClass initialised")

This means you can manipulate class attributes and methods just after it has been defined. This can be useful for logic to set and update defaults, or enforce restrictions like checking or validating required class attributes.

As an aside, Python 3.6 introduced the class method __init_subclass__, which is called when someone subclasses that class - essentially a shortcut for defining __init__ on a metaclass, and thus covers 95% of the use cases for metaclasses. In other words, if you're reaching for a metaclass, you probably want __init_subclass__ instead. But in these articles we're working towards the 5% of use cases that it doesn't help with, because they're more fun, so we're going to ignore it and focus on metaclasses.

Metaclasses also have a __new__ method, which lets you manipulate the class before it is defined. This is a lot more fun - it lets you do things like dynamically define class attributes and methods, or strip off attributes and use them to build something far more complicated before the class exists:

class MyMetaclass(type):
    def __new__(cls, name, bases, attrs):
        meta = attrs.pop("Meta")
        if meta and isinstance(meta, type):
            print("This had a `class Meta` definition, but doesn't now")
            attrs = do_something_exciting(attrs, meta)

        # Create the class with the modified attributes
        new_cls = super().__new__(cls, name, bases, attrs)
        return new_cls

One common use of a metaclass is for automatic class registration - we want a list of all subclasses of our class.

First create our base class with our metaclass, but as it isn't going to do anything itself we don't want it in our registry. Lets set an attribute abstract=True, then look for that in our metaclass when we're trying to register. We want to work with the class once it has been defined, so we'll put this in the metaclass's __init__ method:

registry = {}

# Metaclass
class RegistryType(type):
    def __init__(cls, name, bases, attrs):
        super().__init__(name, bases, attrs)
        if attrs.get("abstract", False):
            # cls.abstract is True, so don't register
            return

        # cls.abstract is either False or missing, register
        registry[name] = cls

# Our base class - we don't want to register this
class RegistryBase(metaclass=RegistryType):
    abstract = True

# User's subclass - we do want to register this
class First(RegistryBase):
    """Not abstract"""

This is a pretty common pattern (or at least was until __init_subclass__ arrived), and you've probably come across it without realising. A great example is in Django's models, which looks something like this:

from django.apps import apps

class ModelBase(type):
    def __init__(cls, name, bases, attrs):
        super().__init__(name, bases, attrs)
        if not attrs.get("abstract", False):
            apps.register(name, cls)

class Model(metaclass=ModelBase):
    abstract = True

class Cat(Model):
    name = models.CharField(...)

Obviously this is very simplified, but this is essentially what you'll find if you dig through django.db.models.base.

This works well for Django, as it can register the models for relationships and migrations; and it makes things simple for the user, who doesn't have to worry about manually registering their model classes.

But why not register them manually? After all, Django does that elsewhere - the admin for example:

# Register with a function call...
class CatAdmin(admin.ModelAdmin):
    ...
admin.register(Cat, CatAdmin):

# ... or using a decorator
@admin.register(Dog)
class DogAdmin(admin.ModelAdmin)

The main reason: it saves the user a line of boilerplate. The actual metaclass does a lot more, but it could be written as a decorator or function call. The metaclass is a sprinkling of syntactic sugar to save the user from having to remember to do something, which sounds great - the less code you need to write, the less code someone else needs to read, and the less scope for mistakes. And it is great for Django's models. But if it's so good, why not use it everywhere?

The problem with metaclasses is they are a bit too magical for every-day use. Without knowing what we've just talked about, from a standard model defininition you'd have no idea how Django is going to find out about your model.

More importantly, it's not clear how you'd control registration. For a start, you don't necessarily want everything to register with the same thing; it would be awkward when you want to register a ModelAdmin against multiple admin sites, for example. When you introduce a metaclass to your code, you need to be confident that your users will never need to work around the defaults and understand the "how". This is why the Zen of Python discourages this sort of thing in the first place.

Metaclasses exist to hide explicit logic - in this case, registration is now implicit, done entirely behind the scenes. You define a model and suddenly Django just magically knows about it, and if you have a problem with that, good luck - but as nobody sensible has a problem with how Django's models work, they're a perfect fit.

Normally. If you're sensible. This brings me onto the topic for next week: nanodjango.

The Corruption of Python

Wed, 01 Jan 2025 13:02:57 +0000

I enjoy doing silly things with code - having an idea that makes me chuckle and then figuring out how to make it happen. These are often fun diversions, an exercise in pushing my limits and Python's - but sometimes they turn into proper projects which I release, other people use, and I then need to maintain. As a result, a lot of my projects harbour a dark secret or two, and I've used a lot of techniques to hide them away and protect my users from the troubles they can bring. I'm going to talk about these techniques in this and the next few posts.

The Zen of Python says that simple is better than complex, explicit is better than implicit, and beautiful is better than ugly. I sometimes say that my projects follow The Corruption of Python: complex makes the simple possible, implicit is simpler than explicit, and beauty is in the eye of the beholder.

The most obvious way for a library to move code away from the user is to put it in a function for them to call. Python takes this a step further and gives us decorators - not controversial or particularly complicated, but it's a good place to start. They're a quick and easy way to move boilerplate functionality away from the code you work on regularly. A contrived example would be checking an argument is a string:

def write_to_screen(var):
    if not isinstance(var, str):
        raise ValueError("Expected a string")
    print(var)

Simple enough, but if you do that 10 times, you need to write those 2 lines 10 times. It starts to get messy - the complexity is now with the person writing this code. Decorators let you shift that complexity to somewhere else in your codebase, like a function call but separate from your code, with a neater syntax:

@enforce_string
def write_to_screen(var):
    print(var)

Nice and clean, nothing cluttering your function logic, difficult to mess up. But this comes at a cost - those 2 lines of checking logic are now 6:

def enforce_string(fn):
    def wrap(var):
        if not isinstance(var, str):
            raise ValueError("Expected a string")
        return fn(var)
    return wrap

And this is a contrived example - things can get much worse in the real world.

Mara is my asynchronous networking library, which uses decorators to register event handlers:

@server.on(events.Receive)
async def echo(event: events.Receive):
    event.connection.write(event.data)

Here we have an echo function which receives an event and sends the same data back to the same client. We bind that to the server using the server.on decorator, which says "listen for the Receive event and pass it to this function". It's about as simple and clear as networking can get.

But it comes at a cost - the code behind that simple decorator involves two classes and multiple functions. I didn't need to do it like this - I could have exposed the event lookup dictionary on the server object, and told Mara users to append their listener callbacks to that directly.

By doing it this way I've made the library easier to use, but that one line decorator is now hiding significant complexity, and if there's a problem in there it's going to be pretty difficult for someone using Mara to figure out what has gone wrong. By making the choice to simplify Mara's API, I've raised the barrier to entry for any potential contributors.

I'm happy with that - my goal is to write a library which makes networking easier, and this achieves that. With tests I can mitigate the risk and minimise the hassle. But the point I'm making is that this has increased my maintenance burden - and this burden is only going to get worse as we progress through this series of posts.

So as I said, decorators are neither controversial nor all that complicated, but this does set the scene for where we're going. Next time, we'll talk about metaclasses.

Syntactic Sugar vs Maintainability

Fri, 13 Sep 2019 16:04:16 +0000

I've just given a talk at PyCon UK, called Syntactic Sugar vs Maintainability, looking at balancing helping your users at the cost of your sanity.

The synopsis was:

Is it ever worth committing coding sins for the greater good? We'll look at techniques which can make your code easier to use at the cost of being harder to maintain, and when the effort is worth the reward.

There are plenty of ways in which you can use and abuse the power of python to make your library code easier for your users to work with. I'm going to talk you through some techniques to design clean and simple library interfaces for your users, and explain how they can make things both easier and harder at the same time.

Using real world examples we'll touch on topics such as automatic registration using metaclasses; changing base classes at runtime to save your users a line of code; and the joys and pitfalls of monkey patching things which should probably never be monkey patched.

By the end of the talk you'll know why doing these things is usually a bad idea, and why I think it's worth doing them anyway.

I've also uploaded the slides and links to resources - if you came to listen, thanks very much! And if you didn't, the video should be up soon.

Mara - a Python network service framework

Sun, 13 Dec 2015 13:34:44 +0000

I've released a new version of Mara, my network service framework written in Python. It aims to make it easy to build TCP/IP services, such as echo servers, flash policy servers, chatrooms, talkers and MUDs.

It's event-based; that is to say you write event listener functions which you bind to events that your service raises - like Connect, Receive or Disconnect.

Mara is on pypi, so you can pip install mara, then start writing your service. An echo server in Mara looks like this:

from mara import Service
service = Service()

@service.listen(mara.events.Receive)
def receive(event):
    event.client.write(event.data)

if __name__ == '__main__':
    service.run()

You can then save it as echo.py and run the service by calling it:

python echo.py
* Server listening on 127.0.0.1:9000

That's a pretty simple example, but Mara can do a bunch more. Its core has support for things like telnet negotiation, timers, a storage system, and seamless restarts (where client connections and storage objects persist, but your code is reloaded cleanly), and it ships with a contrib module which has a lot of optional extras, such as a command manager and dispatcher, basic natural language tools, user accounts and rooms.

Although there's a focus on talkers and muds in the contrib modules at the moment, Mara should be a reasonable base for writing any network service. To get a feel for what you can do with it, take a look at the examples, which include an IRC-style chat server, a simple talker, and the start of a basic mud. There's also fairly comprehensive documentation.

I've always enjoyed writing this sort of thing, so this is a fun side project for me. At its heart Mara is a rewrite of my old perl chat server Cletus, which I wrote in 2001 - in fact if you dive back a few months through git, you'll see Mara was called Cletus until I realised that name was taken on pypi.

It's still missing a few glaringly obvious features at the moment - most notably unicode and python 3 support, an example of using threads through events and timers, and more contrib modules for the mud like items, combat and NPCs. That said, it should make a solid starting point for any network service that you'd want to write, and as always, contributions are welcome.

Introducing Tagulous

Fri, 09 Oct 2015 07:22:53 +0000

Tagulous is a tagging library for Django which is based on ManyToManyField and ForeignKey relationships. I've been developing and using it internally for several years, and have recently tidied it up for release; it supports Django 1.4 to 1.9a, on Python 2.7 to 3.5.

It started with a simple enough idea - rather than use generic relations like other tagging libraries, use a subclass of ManyToManyField which supports assignment using tag strings, to allow things like this:

class Person(models.Model):
  name = models.CharField(max_length=255)
  skills = TagField()

person = Person.objects.create(name='Bob', skills='run, jump')
person.skills = 'run, "kung fu", jump'

And because the underlying relationship is a ManyToManyField, you can build queries exactly as you would expect:

runners = Person.objects.filter(skills='run')
user_skills = Pet.skills.tag_model.objects.filter(pet__owner=request.user)

In this example the related tag model is generated automatically (and accessible on field.tag_model), but you can create custom tag models and share them between fields if you prefer. See the usage examples for more examples of how you can use Tagulous.

The first version wasn't particularly complex, but as I started using it more it quickly became a more substantial project, with admin support, integrated autocomplete, support for hierarchical trees, and a ForeignKey version - a SingleTagField which essentially operates as a CharField with a list of choices which can be customised by users at runtime.

It has a comprehensive set of tests, and has been in use for several years on several projects, so I'm reasonably confident that it's stable and the API won't need to be changed significantly now. That said, I'm releasing it as a beta version until it's been out in the wild for a bit - so please give it a try, and let me know how you get on.

Tagulous is available on pypi and github, and this site hosts the documentation and a demo of the front-end.