Radiac's Blog

Monkeypatching Django

Wed, 15 Jan 2025 08:30:00 +0000

I haven't yet posted here about my project nanodjango. If you haven't heard of it yet, it is a package which lets you write Django in a single file. I gave a lightning talk at Djangocon US and have written an introductory blog post over at Lincoln Loop, if you want to find out more from a user's perspective - but here I'm going to talk about how it works.

A couple of years ago I took over yet another project where the previous developers had heard "Flask is easier than Django", and I saw they had spent a long time and a lot of money accidentally building their own poor version of Django, burying the project under years of technical debt. It became clear why they had handed the project off saying they couldn't support it any more - even simple changes required hours of wading through spaghetti code, and upgrading the very outdated packages was daunting, if not impractical. We got frustrated working on it, the client got frustrated paying so much for small changes, and it became clear the only practical option was a rewrite - which the client couldn't afford.

Choosing Flask had been a mistake. Don't get me wrong, Flask has its place, but you need experience to make good decisions that will help your project grow - experience which the original developers didn't have. Django's startproject may be offputting to beginners, but it does force you to follow a consistent structure which scales and makes it easier to hand off to another team.

And when your project grows to need a database, forms or admin, Flask devs will be rolling their own solutions or pulling in third-party libraries with varying levels of support and compatibility, while Django developers will be using core batteries which will always work for the latest version of Django, and we won't look to pull in third-party libraries until we get to much higher-level functionality. This leads to a more stable platform to develop against, and for those reasons I'm firmly of the opinion that Django is the better option for complex projects.

But I thought to myself, why can't Django be a good fit for smaller projects and prototypes too? Why can't Django work like Flask, in a single file?

Django in a single file in itself isn't particularly novel or special - boiled down, Django just routes requests to functions which return responses, so getting it to handle views in a single file had been done, and done well - first in 2009 with djing (Simon Willison) and Django Inside Tornado (Yann Malet), and more recently Using Django as a Micro-Framework (Carlton Gibson), django-microframework (Will Vincent and Peter Baumgartner), μDjango (Paolo Melchiorre), django-singlefile (Andrew Godwin) and Django from first principles (Eric Matthes) - and many more in between.

Most of these do what Flask does - they implement views and routing, and sensibly stop there. I enjoy doing silly things with Python though, so wanted to see if I could give you access to all of Django's batteries from a single file - the bits which expect you to structure things properly in separate modules, so they can control what order things are loaded in to let them sprinkle the syntactic sugar that makes Django so great. The things that use metaclasses and implicit registration. I wanted models.

This was fun. Models need to register with your app in the apps registry, so you need to get the current module into INSTALLED_APPS - but you can't just add the current module to INSTALLED_APPS to get it in there, because your module hasn't finished importing. If it hasn't finished importing, the apps registry can't resolve a reference to it, and it will try to load it again - putting you into an infinite import loop.

I tried various silly things, like monkeypatching Django's models and apps to delay model registration until import was complete, disabling them completely and parsing the AST to rewrite the single file into a proper app structure in-memory on demand, but then I realised I could just trick apps into not importing the model if it already looked like it was imported.

The first problem is that Django expects to load apps itself during setup, but we're going to run setup from within our app, so we need to make Django think we've already loaded it. To do this we need an AppConfig with a hard-coded path:

class NanodjangoAppConfig(AppConfig):
    path = str(get_script_path())

and then we need to manually add that to the app registry, before we call django.setup(), which will in turn call apps.populate() to pick it up:

app_config = NanodjangoAppConfig(app_name=get_script_name(), app_module=get_script_module())
apps_registry.app_configs[app_config.label] = app_config
app_config.apps = apps_registry
app_config.models = {}

That's solved app load order - we've now force-registered an app using our script name, pointing at our script module, and it knows where to look for anything it might want to look at. In nanodjango, that code is run as part of the app = Django() intialisation step, just before it calls django.setup() - which is why models can't be defined in Nanodjango before the app object exists.

But now we've got another problem - Django's model metaclass magic has expectations which we don't want to meet - in particular, it looks for the app name in MyModel.__module__ - but because of how we're running our script, it will get the string __main__, rather than the name of the app we're trying to dynamically create.

The metaclass then uses this module name to do a lookup in the apps registry, but of course doesn't find anything - we registered our AppConfig under a different name - so we'll get a fatal error saying our app __main__ isn't in INSTALLED_APPS.

To solve this, we need to change the behaviour of Django's ModelBase metaclass. If you need a refresher, we talked about metaclasses last week.

For those who haven't come across it before, monkeypatching is the practice of changing other people's code at runtime. Python makes this possible because everything is an object - we can put a reference to a function in a variable, and overwrite the original just as easily.

The most common time to do this is during testing, where you want to make a temporary change to stub something out - for example, this would be a way to ensure that os.path.exists says that every file starting test:// exists:

import os

# Create a reference to the old exists() method
old_exists = os.path.exists

def fake_exists(path):
    # Our override logic
    if path.startswith('test://'):
        return True

    # Otherwise fall back to the old behaviour
    return old_exists(path)

# Overwrite the exists() method with our new function
os.path.exists = fake_exists

The other reason is to make your code work with someone else's - their code doesn't do what you want, but for whatever reason you don't want to fork it or submit a PR. That's what we have with nanodjango.

Essentially all we're doing is overwriting a variable so that when other code tries to use it, it finds our new value or function which does what we want, but this introduces potential problems.

You do have to be careful not to break other people's code. In a test we can get away with functions not doing what they were originally meant to (either intentionally or accidentally), but if you do it in production code things are going to go very wrong very quickly.

It's important that you write unit tests to check that the old and new functions behave the same under normal circumstances - if nothing else, it will tell you when the upstream package changes. You also have to be extra careful that your code doesn't introduce bugs or exceptions - monkeypatched code can be a pain to work past in a stack trace.

But as long as you're aware of the risks, and you take care around them, monkeypatching can be another powerful ally when trying to make your code easier to use.

Which brings me back to nanodjango's problem with the ModelBase metaclass. If we look at the source, we'll see the error comes from ModelBase.new:

class ModelBase(type):
    def __new__(cls, name, bases, attrs, **kwargs):
        ...
        if getattr(meta, "app_label", None) is None:
            if app_config is None:
                if not abstract:
                    raise RuntimeError(
                        "Model class %s.%s doesn't declare an explicit "
                        "app_label and isn't in an application in "
                        "INSTALLED_APPS." % (module, name)
                    )

            else:
                app_label = app_config.label
        ...

The app_config is None, it's not abstract, and we don't have an app_label in our class Meta definition, so we get the error. That last condition looks promising though - could we just tell nanodjango users that every model they define needs an app_label?

class MyModel(models.Model):
    ...
    class Meta:
        app_label = "myscript"

If you try it you'll see that would work, but it's messy - I don't want to write that every time I define a model, and neither will my users. Monkeypatching to the rescue:

# Collect a reference to the old __new__
old_new = ModelBase.__new__

def new_new(cls, name, bases, attrs, **kwargs):
    # See if this is a nanodjango model, if it is defined in __main__
    module = attrs["__module__"]
    if module == "__main__":
        # It's a nanodjango model, so let's say it's from our new module...
        attrs["__module__"] = app_name

        # ... update a Meta class if it exists ...
        attr_meta = attrs.get("Meta")
        if attr_meta:
            if not getattr(attr_meta, "app_label", None):
                attr_meta.app_label = app_name

        # ... or create one if it doesn't
        else:
            class attr_meta:
                app_label = app_name

            attrs["Meta"] = attr_meta

    # Call the original ModelBase.__new__
    return old_new(cls, name, bases, attrs, **kwargs)

# Swap our function in so any new model will be created using our code
ModelBase.__new__ = new_new

This is the actual code from the current nanodjango patch_modelbase() function. If you look in that file you'll notice that I like to put my monkeypatches in functions which need to be called explicitly, rather than applying themselves during import - it makes it slightly clearer when and where things happen.

The approach I've taken here is very defensive - I only make changes if I know it's something I'm in charge of (a model defined in the __main__ module can only ever come from nanodjango), and even then I'm careful not to disturb any Meta.app_label value which may already exist.

I disturb the original code as little as possible - I was lucky here that I found a route to fix it by setting Meta.app_label before it's checked; that's not always going to be the case, but you can usually find a way to accomplish your goals while still calling the original function. What you don't want to do is rewrite and replace the original function entirely - you could if you have no other option, but then you'll own it forever, and have to keep a much closer eye on it every time a new upstream version is released.

If you dig in nanodjango's monkeypatch file you'll spot there's also a patch to get migrations working - and you'll see it uses a very similar approach.

So that's metaclasses and monkeypatching. But that's still pretty tame, and I promised you dark secrets. Next week it's time to look at my tagging library, django-tagulous.

Magical Metaclasses

Wed, 08 Jan 2025 08:30:00 +0000

Continuing on from the previous post about decorators, we should visit metaclasses before we get to the good stuff.

For those who haven't worked with them, metaclasses are essentially invisible decorators for classes.

The metaclass describes how your class is made. You define a base class with a custom metaclass, then every class which inherits from your base class will have the same metaclass.

Whereas a normal class's __init__ method will let you control what happens when your class is instantiated, the __init__ method on a metaclass will let you control what happens when your class is defined:

class MyMetaclass(type):
    def __init__(self, name, bases, dct):
        super().__init__(name, bases, dct)
        print("MyClass defined")

class MyClass(metaclass=MyMetaclass):
    def __init__(self):
        print("MyClass initialised")

This means you can manipulate class attributes and methods just after it has been defined. This can be useful for logic to set and update defaults, or enforce restrictions like checking or validating required class attributes.

As an aside, Python 3.6 introduced the class method __init_subclass__, which is called when someone subclasses that class - essentially a shortcut for defining __init__ on a metaclass, and thus covers 95% of the use cases for metaclasses. In other words, if you're reaching for a metaclass, you probably want __init_subclass__ instead. But in these articles we're working towards the 5% of use cases that it doesn't help with, because they're more fun, so we're going to ignore it and focus on metaclasses.

Metaclasses also have a __new__ method, which lets you manipulate the class before it is defined. This is a lot more fun - it lets you do things like dynamically define class attributes and methods, or strip off attributes and use them to build something far more complicated before the class exists:

class MyMetaclass(type):
    def __new__(cls, name, bases, attrs):
        meta = attrs.pop("Meta")
        if meta and isinstance(meta, type):
            print("This had a `class Meta` definition, but doesn't now")
            attrs = do_something_exciting(attrs, meta)

        # Create the class with the modified attributes
        new_cls = super().__new__(cls, name, bases, attrs)
        return new_cls

One common use of a metaclass is for automatic class registration - we want a list of all subclasses of our class.

First create our base class with our metaclass, but as it isn't going to do anything itself we don't want it in our registry. Lets set an attribute abstract=True, then look for that in our metaclass when we're trying to register. We want to work with the class once it has been defined, so we'll put this in the metaclass's __init__ method:

registry = {}

# Metaclass
class RegistryType(type):
    def __init__(cls, name, bases, attrs):
        super().__init__(name, bases, attrs)
        if attrs.get("abstract", False):
            # cls.abstract is True, so don't register
            return

        # cls.abstract is either False or missing, register
        registry[name] = cls

# Our base class - we don't want to register this
class RegistryBase(metaclass=RegistryType):
    abstract = True

# User's subclass - we do want to register this
class First(RegistryBase):
    """Not abstract"""

This is a pretty common pattern (or at least was until __init_subclass__ arrived), and you've probably come across it without realising. A great example is in Django's models, which looks something like this:

from django.apps import apps

class ModelBase(type):
    def __init__(cls, name, bases, attrs):
        super().__init__(name, bases, attrs)
        if not attrs.get("abstract", False):
            apps.register(name, cls)

class Model(metaclass=ModelBase):
    abstract = True

class Cat(Model):
    name = models.CharField(...)

Obviously this is very simplified, but this is essentially what you'll find if you dig through django.db.models.base.

This works well for Django, as it can register the models for relationships and migrations; and it makes things simple for the user, who doesn't have to worry about manually registering their model classes.

But why not register them manually? After all, Django does that elsewhere - the admin for example:

# Register with a function call...
class CatAdmin(admin.ModelAdmin):
    ...
admin.register(Cat, CatAdmin):

# ... or using a decorator
@admin.register(Dog)
class DogAdmin(admin.ModelAdmin)

The main reason: it saves the user a line of boilerplate. The actual metaclass does a lot more, but it could be written as a decorator or function call. The metaclass is a sprinkling of syntactic sugar to save the user from having to remember to do something, which sounds great - the less code you need to write, the less code someone else needs to read, and the less scope for mistakes. And it is great for Django's models. But if it's so good, why not use it everywhere?

The problem with metaclasses is they are a bit too magical for every-day use. Without knowing what we've just talked about, from a standard model defininition you'd have no idea how Django is going to find out about your model.

More importantly, it's not clear how you'd control registration. For a start, you don't necessarily want everything to register with the same thing; it would be awkward when you want to register a ModelAdmin against multiple admin sites, for example. When you introduce a metaclass to your code, you need to be confident that your users will never need to work around the defaults and understand the "how". This is why the Zen of Python discourages this sort of thing in the first place.

Metaclasses exist to hide explicit logic - in this case, registration is now implicit, done entirely behind the scenes. You define a model and suddenly Django just magically knows about it, and if you have a problem with that, good luck - but as nobody sensible has a problem with how Django's models work, they're a perfect fit.

Normally. If you're sensible. This brings me onto the topic for next week: nanodjango.

The Corruption of Python

Wed, 01 Jan 2025 13:02:57 +0000

I enjoy doing silly things with code - having an idea that makes me chuckle and then figuring out how to make it happen. These are often fun diversions, an exercise in pushing my limits and Python's - but sometimes they turn into proper projects which I release, other people use, and I then need to maintain. As a result, a lot of my projects harbour a dark secret or two, and I've used a lot of techniques to hide them away and protect my users from the troubles they can bring. I'm going to talk about these techniques in this and the next few posts.

The Zen of Python says that simple is better than complex, explicit is better than implicit, and beautiful is better than ugly. I sometimes say that my projects follow The Corruption of Python: complex makes the simple possible, implicit is simpler than explicit, and beauty is in the eye of the beholder.

The most obvious way for a library to move code away from the user is to put it in a function for them to call. Python takes this a step further and gives us decorators - not controversial or particularly complicated, but it's a good place to start. They're a quick and easy way to move boilerplate functionality away from the code you work on regularly. A contrived example would be checking an argument is a string:

def write_to_screen(var):
    if not isinstance(var, str):
        raise ValueError("Expected a string")
    print(var)

Simple enough, but if you do that 10 times, you need to write those 2 lines 10 times. It starts to get messy - the complexity is now with the person writing this code. Decorators let you shift that complexity to somewhere else in your codebase, like a function call but separate from your code, with a neater syntax:

@enforce_string
def write_to_screen(var):
    print(var)

Nice and clean, nothing cluttering your function logic, difficult to mess up. But this comes at a cost - those 2 lines of checking logic are now 6:

def enforce_string(fn):
    def wrap(var):
        if not isinstance(var, str):
            raise ValueError("Expected a string")
        return fn(var)
    return wrap

And this is a contrived example - things can get much worse in the real world.

Mara is my asynchronous networking library, which uses decorators to register event handlers:

@server.on(events.Receive)
async def echo(event: events.Receive):
    event.connection.write(event.data)

Here we have an echo function which receives an event and sends the same data back to the same client. We bind that to the server using the server.on decorator, which says "listen for the Receive event and pass it to this function". It's about as simple and clear as networking can get.

But it comes at a cost - the code behind that simple decorator involves two classes and multiple functions. I didn't need to do it like this - I could have exposed the event lookup dictionary on the server object, and told Mara users to append their listener callbacks to that directly.

By doing it this way I've made the library easier to use, but that one line decorator is now hiding significant complexity, and if there's a problem in there it's going to be pretty difficult for someone using Mara to figure out what has gone wrong. By making the choice to simplify Mara's API, I've raised the barrier to entry for any potential contributors.

I'm happy with that - my goal is to write a library which makes networking easier, and this achieves that. With tests I can mitigate the risk and minimise the hassle. But the point I'm making is that this has increased my maintenance burden - and this burden is only going to get worse as we progress through this series of posts.

So as I said, decorators are neither controversial nor all that complicated, but this does set the scene for where we're going. Next time, we'll talk about metaclasses.

Syntactic Sugar vs Maintainability

Fri, 13 Sep 2019 16:04:16 +0000

I've just given a talk at PyCon UK, called Syntactic Sugar vs Maintainability, looking at balancing helping your users at the cost of your sanity.

The synopsis was:

Is it ever worth committing coding sins for the greater good? We'll look at techniques which can make your code easier to use at the cost of being harder to maintain, and when the effort is worth the reward.

There are plenty of ways in which you can use and abuse the power of python to make your library code easier for your users to work with. I'm going to talk you through some techniques to design clean and simple library interfaces for your users, and explain how they can make things both easier and harder at the same time.

Using real world examples we'll touch on topics such as automatic registration using metaclasses; changing base classes at runtime to save your users a line of code; and the joys and pitfalls of monkey patching things which should probably never be monkey patched.

By the end of the talk you'll know why doing these things is usually a bad idea, and why I think it's worth doing them anyway.

I've also uploaded the slides and links to resources - if you came to listen, thanks very much! And if you didn't, the video should be up soon.

Tagulous 0.13.0

Mon, 30 Apr 2018 19:04:57 +0000

Tagulous v0.13.0 is now available - it adds support for Django 1.11 and addresses several issues - see the changelog for full details.

Note that to support for Django 1.11, the names of automatically generated tag models has had to change - they're no longer allowed to start with an underscore. There is a simple fix, but this version does therefore require additional upgrade steps - see the upgrade notes for more information.

Tagulous 0.12.0 released

Sun, 26 Feb 2017 18:58:18 +0000

Tagulous v0.12 is now available - it adds support for Django 1.10 and addresses several issues - see the changelog for full details.

Note that this version may require additional upgrade steps - see the upgrade notes for more information.

Mara - a Python network service framework

Sun, 13 Dec 2015 13:34:44 +0000

I've released a new version of Mara, my network service framework written in Python. It aims to make it easy to build TCP/IP services, such as echo servers, flash policy servers, chatrooms, talkers and MUDs.

It's event-based; that is to say you write event listener functions which you bind to events that your service raises - like Connect, Receive or Disconnect.

Mara is on pypi, so you can pip install mara, then start writing your service. An echo server in Mara looks like this:

from mara import Service
service = Service()

@service.listen(mara.events.Receive)
def receive(event):
    event.client.write(event.data)

if __name__ == '__main__':
    service.run()

You can then save it as echo.py and run the service by calling it:

python echo.py
* Server listening on 127.0.0.1:9000

That's a pretty simple example, but Mara can do a bunch more. Its core has support for things like telnet negotiation, timers, a storage system, and seamless restarts (where client connections and storage objects persist, but your code is reloaded cleanly), and it ships with a contrib module which has a lot of optional extras, such as a command manager and dispatcher, basic natural language tools, user accounts and rooms.

Although there's a focus on talkers and muds in the contrib modules at the moment, Mara should be a reasonable base for writing any network service. To get a feel for what you can do with it, take a look at the examples, which include an IRC-style chat server, a simple talker, and the start of a basic mud. There's also fairly comprehensive documentation.

I've always enjoyed writing this sort of thing, so this is a fun side project for me. At its heart Mara is a rewrite of my old perl chat server Cletus, which I wrote in 2001 - in fact if you dive back a few months through git, you'll see Mara was called Cletus until I realised that name was taken on pypi.

It's still missing a few glaringly obvious features at the moment - most notably unicode and python 3 support, an example of using threads through events and timers, and more contrib modules for the mud like items, combat and NPCs. That said, it should make a solid starting point for any network service that you'd want to write, and as always, contributions are welcome.

Introducing Tagulous

Fri, 09 Oct 2015 07:22:53 +0000

Tagulous is a tagging library for Django which is based on ManyToManyField and ForeignKey relationships. I've been developing and using it internally for several years, and have recently tidied it up for release; it supports Django 1.4 to 1.9a, on Python 2.7 to 3.5.

It started with a simple enough idea - rather than use generic relations like other tagging libraries, use a subclass of ManyToManyField which supports assignment using tag strings, to allow things like this:

class Person(models.Model):
  name = models.CharField(max_length=255)
  skills = TagField()

person = Person.objects.create(name='Bob', skills='run, jump')
person.skills = 'run, "kung fu", jump'

And because the underlying relationship is a ManyToManyField, you can build queries exactly as you would expect:

runners = Person.objects.filter(skills='run')
user_skills = Pet.skills.tag_model.objects.filter(pet__owner=request.user)

In this example the related tag model is generated automatically (and accessible on field.tag_model), but you can create custom tag models and share them between fields if you prefer. See the usage examples for more examples of how you can use Tagulous.

The first version wasn't particularly complex, but as I started using it more it quickly became a more substantial project, with admin support, integrated autocomplete, support for hierarchical trees, and a ForeignKey version - a SingleTagField which essentially operates as a CharField with a list of choices which can be customised by users at runtime.

It has a comprehensive set of tests, and has been in use for several years on several projects, so I'm reasonably confident that it's stable and the API won't need to be changed significantly now. That said, I'm releasing it as a beta version until it's been out in the wild for a bit - so please give it a try, and let me know how you get on.

Tagulous is available on pypi and github, and this site hosts the documentation and a demo of the front-end.

A Tiny Web Font Loader

Mon, 21 Sep 2015 09:51:12 +0000

Today I'm releasing TinyWFL, my tiny web font loader which is about 95% smaller than other popular loaders.

When web fonts started to gain adoption around 2010, the problem people had was FOUT - the flash of unstyled text while you waited for the browser to download the font. I think most people would agree that this has since been solved very comprehensively by webfontloader from Google and Typekit, and that it's now the de-facto standard loader - but back in 2010 or 2011 FOUT was still an issue, which is why I wrote my own.

To be accurate, my loader doesn't actually load anything - it's more of a anti-FOUT aid. The idea behind avoiding FOUT is simple enough: characters are very rarely exactly the same size in different fonts, so if we put a few of them together in an HTML element and then measure its width we should get different values depending on which font it's using. If we measure the width for a font we know they'll have (ie a default font like serif), we can then compare that to the width of the web font we want to use; if they match, the web font hasn't loaded, so we use setTimeout to wait a bit, then we check again. Once we know they're loaded we can either fire off JavaScript functions, or set a CSS class somewhere which can be used by CSS rules to only show the font once it has loaded.

This weekend I wanted to add a 2KB reduced character set font to a site, and thought about replacing my 5 year old code with webfontloader - but that's when I looked at the file sizes. The minified version of webfontloader that's currently on their git master is 11.5KB, and the version on Google's CDN is 16.6KB. Of course those sizes aren't huge, but a loader here and a polyfill there is how you end up with big slow pages. Besides, it seems silly to use a library that is 8x larger than the font it's trying to help show - at best it'll still be 50-100% of your average web font.

By comparison TinyWFL is just 852 bytes. The reason it manages such a big difference is that it leaves out a bunch of features you won't need in most circumstances - and if you do need them, we've already got webfontloader.

You can find TinyWFL on github

As a reward for reading this far, have a couple of useful web font-related links:

Font Squirrel web font generator - upload a font and they'll generate an excellent optimised web font for you to use with TinyWFL.
IcoMoon app - upload vector images (or pick from their selection) to generate an icon font. It probably doesn't make much sense to use TinyWFL for icon fonts, but it's still a great service.

Tips for SSL certificates

Mon, 18 May 2015 12:27:29 +0000

Display CSR information:

openssl req -text -noout -in foo.csr

Display signed cert information:

openssl x509 -in foo.crt.pem -noout -text

To remove a password from a key:

openssl rsa -in foo.key.pem -out foo-unlocked.key.pem

To decode a CRL:

openssl crl -text -in ca.crl.pem

To check a certificate against a CRL:

cat ca/ca.crl.pem ca/ca.crt.pem > crl-check.pem
openssl verify -CAfile crl-check.pem -crl_check foo.crt.pem

Self-Signing Certificate Authorities

Mon, 18 May 2015 12:08:00 +0000

Introduction

If you run a website which receives or displays personal information, passwords or other secrets, you need to encrypt your connections using SSL or TLS. This is what puts the "S" into HTTPS, FTPS, IMAPS, POPS etc, and requires private keys and public certificates. Your browser (or other SSL/TLS client) trusts certain CAs (certificate authorities), and they in turn are willing to trust you by issuing you a certificate, if you throw money at them.

This is necessary for public-facing production deployments, and these days the cheapest certificates don't cost the earth - for example, Namecheap's start at £6/$9, and Let's Encrypt should also be launching this summer, providing free domain-validated certificates. However, cheap certificates are domain validated, where the CA checks that you own the domain using automated tests; these can be slow and laborious, so it's often not practical or possible to get these for internal services or development environments.

The alternative is to set yourself up as a self-signing CA. You won't be trusted by the public, but it's a good option if you're in the position where you're not public-facing, or where you can tell your users to install your root CA before they use your service. You'll be able to issue as many certificates as you want, and your connection should be just as secure as with a "proper" certificate.

The easy way

I've written caman which will do everything for you - it's a bash script with all the commands I'm about to describe in excruciating detail, so if all you want to do is jump to getting your certificates, go over to the project page and start following its instructions.

Read on if you want to know how to do it yourself.

What we're aiming for

We're going to create a directory which contains your CA, your host keys, and the certificates you generate. The structure will be:

ca/                 Your CA information
  caconfig.cnf      CA configuration (modify this)
  ca.key.pem        CA private key (keep this safe)
  ca.crt.pem        CA public certificate (distribute this)
  ca.crl.pem        List of revoked keys (publish this)
  serial            Next serial number
  index.txt         Registered certificates
  crlnumber         Next CRL number
  newcerts/         A copy of each certificate signed
store/              Your certificates
  host.domain.tld/  One folder per host
    config.cnf      Config for the given host
    YYYY-MM-DD/     Build date for cert set
      host.domain.tld.key.pem       Key
      host.domain.tld.csr           Signing request
      host.domain.tld.crt.pem       Certificate
      host.domain.tld.keycrt.pem    Key + cert combo

Prepare your CA

Before we start, you'll need openssl to be installed on your system. In Ubuntu, that would be:

sudo apt-get install openssl

Now we'll create the CA dir, and start the serial number at 1:

cd my_ca
mkdir ca store
mkdir ca/certs ca/private
echo '01' > ca/serial
touch ca/index.txt

You will now need to create your ca/caconfig.cnf file. You can use the caman template:

curl https://raw.githubusercontent.com/radiac/caman/master/ca/caconfig.cnf.default > ca/caconfig.cnf

You will need to change some lines - look for the comments starting # >>.

Change the 6 values under [ req_distinguished_name ]:
- countryName: your two-character country code
- stateOrProvinceName: your state or province
- organizationName: the name of your organisation
- organizationUnitName: your department in the organisation
- commonName: the name of your organisation
- emailAddress: your e-mail address
Change the CRL distribution points URL under [ usr_cert ] and [ v3_ca ]:
- crlDistributionPoints: URL where you will publish your ca.crl.pem
- The CRL is a list of revoked certificates which you'll need to update each time you revoke a host certificate; if you don't want to be bothered with this, you can just comment these lines out, as well as crl_extensions and crlnumber under [ CA_default ].
You can ignore the value for default_days here - caman uses it, but OpenSSL won't; we'll be passing it directly on the command line.

Create the root certificate

Your CA is identified by its root certificate. First we need to create the private CA key:

openssl genrsa -aes256 -out ca/ca.key.pem 4096

The openssl genrsa command generates an RSA key, and the other options tell it to encrypt it with AES 256 (-aes256), to use 4096 bits, and to write it to the file ca/ca.key.pem.

When you run the command, it will ask you for a password - keep it safe, you'll need it every time you want to generate and sign a new certificate.

This will generate the private CA key, ca/ca.key.pem. This is important:

Do not lose this key. Without it, certificates can't be signed or renewed
Do not disclose this key to anyone. If it is compromised, others will be able to impersonate the CA.

If you're really paranoid, you could take further steps to protect your key, by creating an intermediate authority to sign host certificates, and keeping your root key somewhere more secure, eg on a USB key locked in a safe; that way if your intermediate authority key is compromised, you could revoke it and create a new one, without having to get people to install your new CA certificate. However, we'll assume that's not an issue for you, and won't be using an intermediate authority here.

Now we can use the key to generate the public CA certificate:

openssl req -x509 -new -key ca/ca.key.pem -days 36500 -out ca/ca.crt.pem -config ca/caconfig.cnf

The openssl req command with -x509 tells OpenSSL to generate a self-signed certificate; the other options tell it to generate a new certificate (-new) and how long the certificate should be valid for - in this case, -days 36500 means about 100 years. It then uses the caconfig.cnf which we configured earlier, and puts the certificate in the file ca/ca.crt.pem.

Publishing your CA Certificate and CRL

To get clients to trust certificates signed by the new self-signing authority, they must install the root certificate, ca.crt.pem.

The easiest way to do this is to put it on your web server as a normal file for download, called ca.crt. Most browsers will know what to do with it from there.

If you're using Apache to serve your page, it may need to know the MIME type - add this to your configuration or .htaccess:

AddType application/x-x509-ca-cert .crt

You can then generate your CRL with the following command:

openssl ca -gencrl -out ca/ca.crl.pem -config ca/caconfig.cnf

To publish it, just put it at the URL you defined in your caconfig.cnf above. Remember to update it each time you renew or revoke a certificate.

Preparing for a new host

Before we start creating hosts, lets set up a directory to store them.

In this article we'll be creating a certificate for my.example.com:

mkdir store/my.example.com

To make a wildcard certificate to cover multiple hosts, just use an asterisk - for example, *.example.com would catch any subdomain of example.com.

You will now need to create a configuration for this host. Again, you can use the caman template:

curl https://raw.githubusercontent.com/radiac/caman/master/ca/host.cnf.default > store/my.example.com/config.cnf

You will need to change some lines - look for the comments starting # >>.

Change 4 of the values under [ host_distinguished_name ]:
- countryName: the two-character country code for this host
- stateOrProvinceName: the state or province for this host
- organizationName: the name of the organisation for this host
- emailAddress: the e-mail address for the admin for this host
- Do not change commonName or organizationUnitName - these are placeholders which will be set by caman
You can ignore the value for default_days here - caman uses it, but OpenSSL won't; we'll be passing it directly on the command line.

You're now ready to make the certificate itself.

Creating a host certificate

First create a new sub-directory to store this set of files - there will be a key, a CSR (certificate signing request) and a certificate. I like to use the date I'm generating the certificate:

mkdir store/my.example.com/2015-05-18

Now we'll create a new private key and CSR with this command:

openssl req -sha256 \
    -newkey rsa:2048 -nodes \
    -keyout store/my.example.com/2015-05-18/my.example.com.key.pem \
    -new -out store/my.example.com/2015-05-18/my.example.com.csr \
    -config store/my.example.com/config.cnf

This again calls openssl req, but without -x509 this time because we don't want it to be self-signed - we want to sign it using our own CA. We then tell it to use SHA256 (-sha256), and generate a new 2048 bit RSA private key (-newkey rsa:2048) which is not encrypted (-nodes - that's "no DES", not "nodes") in the file ending .key.pem. It will then use this to create a new CSR (-new) in the file ending .csr. It uses the host config we created earlier to provide any additional settings.

Next we need to sign the CSR to generate the certificate:

openssl ca \
    -in store/my.example.com/2015-05-18/my.example.com.csr \
    -out store/my.example.com/2015-05-18/my.example.com.crt.pem \
    -days 3650 -config ca/caconfig.cnf

This calls openssl ca to use the certificate authority to sign the CSR (-in) and generate the certificate (-out). Here it will be valid for 3650 days (-days), or approximately 10 years. We give it the CA config this time, so it knows where to find all the settings for our CA.

You will now have a private key ending .key.pem, and a public certificate ending .crt.pem. In most cases you'll use these separately, but some services may want them merged together in one file (such as Apache wth the SSLCertificateFile directive), so we'll concatenate them just in case it's needed:

cat store/my.example.com/2015-05-18/my.example.com.key.pem \
    store/my.example.com/2015-05-18/my.example.com.crt.pem \
    > store/my.example.com/2015-05-18/my.example.com.keycrt.pem

You can now copy those three files to the server (you won't need the .csr) and configure your services to use them. That's a bit beyond the scope of this article, but may be something I'll follow up with in a later one.

Revoking and renewing certificates

If a host's certificate has expired, is no longer needed, or has been compromised, you must revoke the current certificate for that host, and generate and publish a new CRL.

First check ca/index.txt for the index corresponding to the host you're revoking:

more index.txt | grep my.example.com

You will see something like this:

R       250513123511Z   150516124528Z   01      unknown /C=CN/ST=State/O=MyOrg/OU=my.example.com/CN=my.example.com/emailAddress=email@example.com
R       250513124545Z   150516124604Z   02      unknown /C=CN/ST=State/O=MyOrg/OU=my.example.com/CN=my.example.com/emailAddress=email@example.com
V       250513124606Z                   03      unknown /C=CN/ST=State/O=MyOrg/OU=my.example.com/CN=my.example.com/emailAddress=email@example.com

If a line starts with an R it means that certificate has been revoked; look for the one starting V with CN= which matches your host; it will almost certainly be the last one. Then take the short number in the fourth column; in the example above, the index is 03.

Now you have the index, you can revoke that certificate:

openssl ca -revoke ca/newcerts/03.pem" -config ca/caconfig.cnf

replacing 03 with the index for that certificate.

If you are replacing the certificate, you can now create a new host certificate exactly as you did in the Creating a host certificate section above.

Don't forget to update and publish a new CRL, as described under Publishing your CA Certificate and CRL.

POODLE

Wed, 15 Oct 2014 07:45:59 +0000

I have been meaning to get back on the blogging horse for some time, and what better way than with a new SSL vulnerability.

POODLE was announced this morning. It's a 5/10 on the panic scale, but both users and sysadmins should take action now.

This one isn't particularly exciting compared to the recent sky-is-falling heartbleed and shellshock - instead of giving away all your secrets and/or shell access to anyone with curl, the worst-case scenario with POODLE is that someone can read your SSL traffic; still bad, but for most people running small sites it's nowhere near the same scale.

Unlike shellshock and heartbleed though, this doesn't look like something which is likely to be fixed by a distro patch any time soon, because this vulnerability is inherent in SSLv3 (I'm no crypto expert, so I won't attempt to explain it myself). Because it is still used in the wild, distros are unlikely to disable it by default in the immediate future, so sysadmins will need to do something about it themselves.

This vulnerability doesn't come as a great surprise - SSLv3 is getting on a bit now, and was superseded by TLSv1 which has been supported by most browsers since IE6. However, most clients and servers are designed to fall back to earlier protocols if the newer ones fail - a good idea in theory, but in practice an attacker who wants you to use SSLv3 can force this to happen.

While this fallback strategy is likely to be disabled in future distro patches using TLS_FALLBACK_SCSV, it will still leave any use of SSLv3 vulnerable. Although Google is using SCSV for maximum compatibility, based on Cloudflare's numbers where a trivial amount of valid traffic actually uses SSLv3, most people seem to be advocating the removal of SSLv3 altogether.

As a user you're vulnerable if you use SSLv3, so you should disable it - you're extremely unlikely to need it anyway. I won't go into details, but Mozilla will disable SSLv3 in Firefox 34, or you can disable it now in about:config by setting security.tls.version.min == 1.

Steps for sysadmins

Systems' traffic is vulnerable if they're running SSLv3, so a reasonable course of action would therefore be to disable SSLv3 at the top level of your server config, then re-enable it on specific sites if you get reports of problems - although it would be best to wait for a stable TLS_FALLBACK_SCSV patch to land before re-enabling it.

Those running nginx can disable SSLv3 with Cloudflare's cipher configuration (add to http or server context):

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:EECDH+RC4:RSA+RC4:!MD5;
ssl_prefer_server_ciphers on;

Apache users can add (to server or virtual host context):

SSLProtocol all -SSLv2 -SSLv3

And dirty IIS users get what they deserve - a 17 step procedure where you have to fiddle with the registry.

You should of course also take a look at any other services which are likely to support SSLv3, like mail or VPNs.

Testing your connection

As with heartbleed, Qualys SSL Labs seems to be the place to go for an easy and comprehensive assessment of your SSL security. Alternatively you could use openssl:

openssl s_client -connect [server]:[port] -ssl3

If you run that command and connect successfully, you're still vulnerable.

Canvas bezier curves

Mon, 09 Dec 2013 11:42:11 +0000

The new Christmas design for this site uses bezier curves on canvas elements to generate random snowdrifts behind the header. Drawing a bezier curve is pretty simple, so seems like a reasonable place to start my new blog. First you need a canvas and a context:

<canvas id="myCanvas" width="400" height="200"></canvas>
<script>
var canvas = document.getElementById('myCanvas'),
    context = canvas.getContext("2d");
;
</script>

Think Turtles

A quick aside for those who haven't used the canvas before: a canvas element has a context property, which is what you draw on. Context methods to draw lines take destination coordinates, and draw them from the current position - think of pen up in Logo's turtle graphics, only with absolute positioning.

The canvas equivalent of pen up is context.moveTo(x, y), so to draw a line from (50, 50) to (150, 50), you would say:

context.beginPath();
context.moveTo(50, 50);
context.lineTo(150, 50);
c.strokeStyle = '#000';
c.stroke();

You'll notice I added some extra bits there - beginPath() says we're starting a new path, and stroke() draws it using the current style. Call beginPath() whenever you're about to change the style.

The canvas bezier curve takes 6 arguments; two sets of control points, and an end point:

context.bezierCurveTo(c1x, c1y, c2x, c2y, ex, ey);

Here's an example:

context.beginPath();
context.moveTo(50, 50);
context.bezierCurveTo(50, 150, 250, 150, 350, 50);
context.strokeStyle = '#00d';
context.stroke();

I've added some grey lines and red markers to show what's going on. First we moveTo(50, 50), then call bezierCurveTo with the two control points (50, 150) and (250, 150), and the end point (350, 50). The result is a line from the current position to the end point, pulled out of place by the control points.

Once you've got that far, building a randomly-generated snowdrift is trivial - a bit of Math.random(), and instead of context.stroke() use context.fillStyle and context.fill().

To finish the effect, I'm drawing two background snowdrifts on one canvas element, and a foreground snowdrift on a second, with snowflakes falling in between - although I cheated there and used an animated gif which I had rendered earlier. Your CPU thanks me.

TCMI 2.0

Tue, 03 Dec 2013 13:19:05 +0000

I am very proud to announce my finest work to date - the Tacky Christmas Music Interface 2.0!

The old TCMI played midi files, but most browsers seem to struggle with those these days, and even when it did work, modern soundfonts made the experience quite variable. To get around this, the new version of TCMI now uses HTML5 audio to play MP3 or OGG files (depending on browser support).

This also means that if you've got the relevant PPL/PRS license, you can now play non-tacky music to your visitors! Although I'm not quite sure what the point of that would be.

The old version also ran in an iframe, which made front page SEO and deep-linking difficult for 1/12th of the year. This version therefore has the option to use AJAX loads and history.pushState, with the controls in a styleable <div> overlay.

There's also a new cleaner design, but those who want a tacky UI to go with the tacky music can opt to use the classic design.

You can find out more on the TCMI project page, where you can also download some of the old TCMI midis which I've converted into MP3s and OGGs. Improvements welcome on github.

A New Blog

Sun, 01 Dec 2013 01:15:00 +0000

Welcome to my new tech blog, where I'll be writing about programming, sysadmin and anything else related to computers that I think may be of general interest to strangers on the internet.

I've been on the internet for about 16 years now, and I've had this site for almost 14 of them. I want to get back to sharing some of the stuff I'm working on, so this blog will be a mix of articles and tutorials covering various topics related to web development. I also want to learn things through this process too - as a freelance developer, I've spent a lot of the past 10 years working away on my own, and I'm sure I've missed some tricks - so please do send me feedback, either in the comments or through the contact form.

For those who have been visiting my site for years, I'll still keep my personal diary going, but from now on that will be where I ramble on about my cats, and things that annoy me which I can't fit into 140 characters on twitter.

For those who are new, welcome, and I hope that you will find my blog useful.