Radiac's Blog

Syntactic Sugar vs Maintainability

Fri, 13 Sep 2019 16:04:16 +0000

I've just given a talk at PyCon UK, called Syntactic Sugar vs Maintainability, looking at balancing helping your users at the cost of your sanity.

The synopsis was:

Is it ever worth committing coding sins for the greater good? We'll look at techniques which can make your code easier to use at the cost of being harder to maintain, and when the effort is worth the reward.

There are plenty of ways in which you can use and abuse the power of python to make your library code easier for your users to work with. I'm going to talk you through some techniques to design clean and simple library interfaces for your users, and explain how they can make things both easier and harder at the same time.

Using real world examples we'll touch on topics such as automatic registration using metaclasses; changing base classes at runtime to save your users a line of code; and the joys and pitfalls of monkey patching things which should probably never be monkey patched.

By the end of the talk you'll know why doing these things is usually a bad idea, and why I think it's worth doing them anyway.

I've also uploaded the slides and links to resources - if you came to listen, thanks very much! And if you didn't, the video should be up soon.

Tagulous 0.13.0

Mon, 30 Apr 2018 19:04:57 +0000

Tagulous v0.13.0 is now available - it adds support for Django 1.11 and addresses several issues - see the changelog for full details.

Note that to support for Django 1.11, the names of automatically generated tag models has had to change - they're no longer allowed to start with an underscore. There is a simple fix, but this version does therefore require additional upgrade steps - see the upgrade notes for more information.

Tagulous 0.12.0 released

Sun, 26 Feb 2017 18:58:18 +0000

Tagulous v0.12 is now available - it adds support for Django 1.10 and addresses several issues - see the changelog for full details.

Note that this version may require additional upgrade steps - see the upgrade notes for more information.

Mara - a Python network service framework

Sun, 13 Dec 2015 13:34:44 +0000

I've released a new version of Mara, my network service framework written in Python. It aims to make it easy to build TCP/IP services, such as echo servers, flash policy servers, chatrooms, talkers and MUDs.

It's event-based; that is to say you write event listener functions which you bind to events that your service raises - like Connect, Receive or Disconnect.

Mara is on pypi, so you can pip install mara, then start writing your service. An echo server in Mara looks like this:

from mara import Service
service = Service()

@service.listen(mara.events.Receive)
def receive(event):
    event.client.write(event.data)

if __name__ == '__main__':
    service.run()

You can then save it as echo.py and run the service by calling it:

python echo.py
* Server listening on 127.0.0.1:9000

That's a pretty simple example, but Mara can do a bunch more. Its core has support for things like telnet negotiation, timers, a storage system, and seamless restarts (where client connections and storage objects persist, but your code is reloaded cleanly), and it ships with a contrib module which has a lot of optional extras, such as a command manager and dispatcher, basic natural language tools, user accounts and rooms.

Although there's a focus on talkers and muds in the contrib modules at the moment, Mara should be a reasonable base for writing any network service. To get a feel for what you can do with it, take a look at the examples, which include an IRC-style chat server, a simple talker, and the start of a basic mud. There's also fairly comprehensive documentation.

I've always enjoyed writing this sort of thing, so this is a fun side project for me. At its heart Mara is a rewrite of my old perl chat server Cletus, which I wrote in 2001 - in fact if you dive back a few months through git, you'll see Mara was called Cletus until I realised that name was taken on pypi.

It's still missing a few glaringly obvious features at the moment - most notably unicode and python 3 support, an example of using threads through events and timers, and more contrib modules for the mud like items, combat and NPCs. That said, it should make a solid starting point for any network service that you'd want to write, and as always, contributions are welcome.

Introducing Tagulous

Fri, 09 Oct 2015 07:22:53 +0000

Tagulous is a tagging library for Django which is based on ManyToManyField and ForeignKey relationships. I've been developing and using it internally for several years, and have recently tidied it up for release; it supports Django 1.4 to 1.9a, on Python 2.7 to 3.5.

It started with a simple enough idea - rather than use generic relations like other tagging libraries, use a subclass of ManyToManyField which supports assignment using tag strings, to allow things like this:

class Person(models.Model):
  name = models.CharField(max_length=255)
  skills = TagField()

person = Person.objects.create(name='Bob', skills='run, jump')
person.skills = 'run, "kung fu", jump'

And because the underlying relationship is a ManyToManyField, you can build queries exactly as you would expect:

runners = Person.objects.filter(skills='run')
user_skills = Pet.skills.tag_model.objects.filter(pet__owner=request.user)

In this example the related tag model is generated automatically (and accessible on field.tag_model), but you can create custom tag models and share them between fields if you prefer. See the usage examples for more examples of how you can use Tagulous.

The first version wasn't particularly complex, but as I started using it more it quickly became a more substantial project, with admin support, integrated autocomplete, support for hierarchical trees, and a ForeignKey version - a SingleTagField which essentially operates as a CharField with a list of choices which can be customised by users at runtime.

It has a comprehensive set of tests, and has been in use for several years on several projects, so I'm reasonably confident that it's stable and the API won't need to be changed significantly now. That said, I'm releasing it as a beta version until it's been out in the wild for a bit - so please give it a try, and let me know how you get on.

Tagulous is available on pypi and github, and this site hosts the documentation and a demo of the front-end.

A Tiny Web Font Loader

Mon, 21 Sep 2015 09:51:12 +0000

Today I'm releasing TinyWFL, my tiny web font loader which is about 95% smaller than other popular loaders.

When web fonts started to gain adoption around 2010, the problem people had was FOUT - the flash of unstyled text while you waited for the browser to download the font. I think most people would agree that this has since been solved very comprehensively by webfontloader from Google and Typekit, and that it's now the de-facto standard loader - but back in 2010 or 2011 FOUT was still an issue, which is why I wrote my own.

To be accurate, my loader doesn't actually load anything - it's more of a anti-FOUT aid. The idea behind avoiding FOUT is simple enough: characters are very rarely exactly the same size in different fonts, so if we put a few of them together in an HTML element and then measure its width we should get different values depending on which font it's using. If we measure the width for a font we know they'll have (ie a default font like serif), we can then compare that to the width of the web font we want to use; if they match, the web font hasn't loaded, so we use setTimeout to wait a bit, then we check again. Once we know they're loaded we can either fire off JavaScript functions, or set a CSS class somewhere which can be used by CSS rules to only show the font once it has loaded.

This weekend I wanted to add a 2KB reduced character set font to a site, and thought about replacing my 5 year old code with webfontloader - but that's when I looked at the file sizes. The minified version of webfontloader that's currently on their git master is 11.5KB, and the version on Google's CDN is 16.6KB. Of course those sizes aren't huge, but a loader here and a polyfill there is how you end up with big slow pages. Besides, it seems silly to use a library that is 8x larger than the font it's trying to help show - at best it'll still be 50-100% of your average web font.

By comparison TinyWFL is just 852 bytes. The reason it manages such a big difference is that it leaves out a bunch of features you won't need in most circumstances - and if you do need them, we've already got webfontloader.

You can find TinyWFL on github

As a reward for reading this far, have a couple of useful web font-related links:

Font Squirrel web font generator - upload a font and they'll generate an excellent optimised web font for you to use with TinyWFL.
IcoMoon app - upload vector images (or pick from their selection) to generate an icon font. It probably doesn't make much sense to use TinyWFL for icon fonts, but it's still a great service.

Tips for SSL certificates

Mon, 18 May 2015 12:27:29 +0000

Display CSR information:

openssl req -text -noout -in foo.csr

Display signed cert information:

openssl x509 -in foo.crt.pem -noout -text

To remove a password from a key:

openssl rsa -in foo.key.pem -out foo-unlocked.key.pem

To decode a CRL:

openssl crl -text -in ca.crl.pem

To check a certificate against a CRL:

cat ca/ca.crl.pem ca/ca.crt.pem > crl-check.pem
openssl verify -CAfile crl-check.pem -crl_check foo.crt.pem

Self-Signing Certificate Authorities

Mon, 18 May 2015 12:08:00 +0000

Introduction

If you run a website which receives or displays personal information, passwords or other secrets, you need to encrypt your connections using SSL or TLS. This is what puts the "S" into HTTPS, FTPS, IMAPS, POPS etc, and requires private keys and public certificates. Your browser (or other SSL/TLS client) trusts certain CAs (certificate authorities), and they in turn are willing to trust you by issuing you a certificate, if you throw money at them.

This is necessary for public-facing production deployments, and these days the cheapest certificates don't cost the earth - for example, Namecheap's start at £6/$9, and Let's Encrypt should also be launching this summer, providing free domain-validated certificates. However, cheap certificates are domain validated, where the CA checks that you own the domain using automated tests; these can be slow and laborious, so it's often not practical or possible to get these for internal services or development environments.

The alternative is to set yourself up as a self-signing CA. You won't be trusted by the public, but it's a good option if you're in the position where you're not public-facing, or where you can tell your users to install your root CA before they use your service. You'll be able to issue as many certificates as you want, and your connection should be just as secure as with a "proper" certificate.

The easy way

I've written caman which will do everything for you - it's a bash script with all the commands I'm about to describe in excruciating detail, so if all you want to do is jump to getting your certificates, go over to the project page and start following its instructions.

Read on if you want to know how to do it yourself.

What we're aiming for

We're going to create a directory which contains your CA, your host keys, and the certificates you generate. The structure will be:

ca/                 Your CA information
  caconfig.cnf      CA configuration (modify this)
  ca.key.pem        CA private key (keep this safe)
  ca.crt.pem        CA public certificate (distribute this)
  ca.crl.pem        List of revoked keys (publish this)
  serial            Next serial number
  index.txt         Registered certificates
  crlnumber         Next CRL number
  newcerts/         A copy of each certificate signed
store/              Your certificates
  host.domain.tld/  One folder per host
    config.cnf      Config for the given host
    YYYY-MM-DD/     Build date for cert set
      host.domain.tld.key.pem       Key
      host.domain.tld.csr           Signing request
      host.domain.tld.crt.pem       Certificate
      host.domain.tld.keycrt.pem    Key + cert combo

Prepare your CA

Before we start, you'll need openssl to be installed on your system. In Ubuntu, that would be:

sudo apt-get install openssl

Now we'll create the CA dir, and start the serial number at 1:

cd my_ca
mkdir ca store
mkdir ca/certs ca/private
echo '01' > ca/serial
touch ca/index.txt

You will now need to create your ca/caconfig.cnf file. You can use the caman template:

curl https://raw.githubusercontent.com/radiac/caman/master/ca/caconfig.cnf.default > ca/caconfig.cnf

You will need to change some lines - look for the comments starting # >>.

Change the 6 values under [ req_distinguished_name ]:
- countryName: your two-character country code
- stateOrProvinceName: your state or province
- organizationName: the name of your organisation
- organizationUnitName: your department in the organisation
- commonName: the name of your organisation
- emailAddress: your e-mail address
Change the CRL distribution points URL under [ usr_cert ] and [ v3_ca ]:
- crlDistributionPoints: URL where you will publish your ca.crl.pem
- The CRL is a list of revoked certificates which you'll need to update each time you revoke a host certificate; if you don't want to be bothered with this, you can just comment these lines out, as well as crl_extensions and crlnumber under [ CA_default ].
You can ignore the value for default_days here - caman uses it, but OpenSSL won't; we'll be passing it directly on the command line.

Create the root certificate

Your CA is identified by its root certificate. First we need to create the private CA key:

openssl genrsa -aes256 -out ca/ca.key.pem 4096

The openssl genrsa command generates an RSA key, and the other options tell it to encrypt it with AES 256 (-aes256), to use 4096 bits, and to write it to the file ca/ca.key.pem.

When you run the command, it will ask you for a password - keep it safe, you'll need it every time you want to generate and sign a new certificate.

This will generate the private CA key, ca/ca.key.pem. This is important:

Do not lose this key. Without it, certificates can't be signed or renewed
Do not disclose this key to anyone. If it is compromised, others will be able to impersonate the CA.

If you're really paranoid, you could take further steps to protect your key, by creating an intermediate authority to sign host certificates, and keeping your root key somewhere more secure, eg on a USB key locked in a safe; that way if your intermediate authority key is compromised, you could revoke it and create a new one, without having to get people to install your new CA certificate. However, we'll assume that's not an issue for you, and won't be using an intermediate authority here.

Now we can use the key to generate the public CA certificate:

openssl req -x509 -new -key ca/ca.key.pem -days 36500 -out ca/ca.crt.pem -config ca/caconfig.cnf

The openssl req command with -x509 tells OpenSSL to generate a self-signed certificate; the other options tell it to generate a new certificate (-new) and how long the certificate should be valid for - in this case, -days 36500 means about 100 years. It then uses the caconfig.cnf which we configured earlier, and puts the certificate in the file ca/ca.crt.pem.

Publishing your CA Certificate and CRL

To get clients to trust certificates signed by the new self-signing authority, they must install the root certificate, ca.crt.pem.

The easiest way to do this is to put it on your web server as a normal file for download, called ca.crt. Most browsers will know what to do with it from there.

If you're using Apache to serve your page, it may need to know the MIME type - add this to your configuration or .htaccess:

AddType application/x-x509-ca-cert .crt

You can then generate your CRL with the following command:

openssl ca -gencrl -out ca/ca.crl.pem -config ca/caconfig.cnf

To publish it, just put it at the URL you defined in your caconfig.cnf above. Remember to update it each time you renew or revoke a certificate.

Preparing for a new host

Before we start creating hosts, lets set up a directory to store them.

In this article we'll be creating a certificate for my.example.com:

mkdir store/my.example.com

To make a wildcard certificate to cover multiple hosts, just use an asterisk - for example, *.example.com would catch any subdomain of example.com.

You will now need to create a configuration for this host. Again, you can use the caman template:

curl https://raw.githubusercontent.com/radiac/caman/master/ca/host.cnf.default > store/my.example.com/config.cnf

You will need to change some lines - look for the comments starting # >>.

Change 4 of the values under [ host_distinguished_name ]:
- countryName: the two-character country code for this host
- stateOrProvinceName: the state or province for this host
- organizationName: the name of the organisation for this host
- emailAddress: the e-mail address for the admin for this host
- Do not change commonName or organizationUnitName - these are placeholders which will be set by caman
You can ignore the value for default_days here - caman uses it, but OpenSSL won't; we'll be passing it directly on the command line.

You're now ready to make the certificate itself.

Creating a host certificate

First create a new sub-directory to store this set of files - there will be a key, a CSR (certificate signing request) and a certificate. I like to use the date I'm generating the certificate:

mkdir store/my.example.com/2015-05-18

Now we'll create a new private key and CSR with this command:

openssl req -sha256 \
    -newkey rsa:2048 -nodes \
    -keyout store/my.example.com/2015-05-18/my.example.com.key.pem \
    -new -out store/my.example.com/2015-05-18/my.example.com.csr \
    -config store/my.example.com/config.cnf

This again calls openssl req, but without -x509 this time because we don't want it to be self-signed - we want to sign it using our own CA. We then tell it to use SHA256 (-sha256), and generate a new 2048 bit RSA private key (-newkey rsa:2048) which is not encrypted (-nodes - that's "no DES", not "nodes") in the file ending .key.pem. It will then use this to create a new CSR (-new) in the file ending .csr. It uses the host config we created earlier to provide any additional settings.

Next we need to sign the CSR to generate the certificate:

openssl ca \
    -in store/my.example.com/2015-05-18/my.example.com.csr \
    -out store/my.example.com/2015-05-18/my.example.com.crt.pem \
    -days 3650 -config ca/caconfig.cnf

This calls openssl ca to use the certificate authority to sign the CSR (-in) and generate the certificate (-out). Here it will be valid for 3650 days (-days), or approximately 10 years. We give it the CA config this time, so it knows where to find all the settings for our CA.

You will now have a private key ending .key.pem, and a public certificate ending .crt.pem. In most cases you'll use these separately, but some services may want them merged together in one file (such as Apache wth the SSLCertificateFile directive), so we'll concatenate them just in case it's needed:

cat store/my.example.com/2015-05-18/my.example.com.key.pem \
    store/my.example.com/2015-05-18/my.example.com.crt.pem \
    > store/my.example.com/2015-05-18/my.example.com.keycrt.pem

You can now copy those three files to the server (you won't need the .csr) and configure your services to use them. That's a bit beyond the scope of this article, but may be something I'll follow up with in a later one.

Revoking and renewing certificates

If a host's certificate has expired, is no longer needed, or has been compromised, you must revoke the current certificate for that host, and generate and publish a new CRL.

First check ca/index.txt for the index corresponding to the host you're revoking:

more index.txt | grep my.example.com

You will see something like this:

R       250513123511Z   150516124528Z   01      unknown /C=CN/ST=State/O=MyOrg/OU=my.example.com/CN=my.example.com/emailAddress=email@example.com
R       250513124545Z   150516124604Z   02      unknown /C=CN/ST=State/O=MyOrg/OU=my.example.com/CN=my.example.com/emailAddress=email@example.com
V       250513124606Z                   03      unknown /C=CN/ST=State/O=MyOrg/OU=my.example.com/CN=my.example.com/emailAddress=email@example.com

If a line starts with an R it means that certificate has been revoked; look for the one starting V with CN= which matches your host; it will almost certainly be the last one. Then take the short number in the fourth column; in the example above, the index is 03.

Now you have the index, you can revoke that certificate:

openssl ca -revoke ca/newcerts/03.pem" -config ca/caconfig.cnf

replacing 03 with the index for that certificate.

If you are replacing the certificate, you can now create a new host certificate exactly as you did in the Creating a host certificate section above.

Don't forget to update and publish a new CRL, as described under Publishing your CA Certificate and CRL.

POODLE

Wed, 15 Oct 2014 07:45:59 +0000

I have been meaning to get back on the blogging horse for some time, and what better way than with a new SSL vulnerability.

POODLE was announced this morning. It's a 5/10 on the panic scale, but both users and sysadmins should take action now.

This one isn't particularly exciting compared to the recent sky-is-falling heartbleed and shellshock - instead of giving away all your secrets and/or shell access to anyone with curl, the worst-case scenario with POODLE is that someone can read your SSL traffic; still bad, but for most people running small sites it's nowhere near the same scale.

Unlike shellshock and heartbleed though, this doesn't look like something which is likely to be fixed by a distro patch any time soon, because this vulnerability is inherent in SSLv3 (I'm no crypto expert, so I won't attempt to explain it myself). Because it is still used in the wild, distros are unlikely to disable it by default in the immediate future, so sysadmins will need to do something about it themselves.

This vulnerability doesn't come as a great surprise - SSLv3 is getting on a bit now, and was superseded by TLSv1 which has been supported by most browsers since IE6. However, most clients and servers are designed to fall back to earlier protocols if the newer ones fail - a good idea in theory, but in practice an attacker who wants you to use SSLv3 can force this to happen.

While this fallback strategy is likely to be disabled in future distro patches using TLS_FALLBACK_SCSV, it will still leave any use of SSLv3 vulnerable. Although Google is using SCSV for maximum compatibility, based on Cloudflare's numbers where a trivial amount of valid traffic actually uses SSLv3, most people seem to be advocating the removal of SSLv3 altogether.

As a user you're vulnerable if you use SSLv3, so you should disable it - you're extremely unlikely to need it anyway. I won't go into details, but Mozilla will disable SSLv3 in Firefox 34, or you can disable it now in about:config by setting security.tls.version.min == 1.

Steps for sysadmins

Systems' traffic is vulnerable if they're running SSLv3, so a reasonable course of action would therefore be to disable SSLv3 at the top level of your server config, then re-enable it on specific sites if you get reports of problems - although it would be best to wait for a stable TLS_FALLBACK_SCSV patch to land before re-enabling it.

Those running nginx can disable SSLv3 with Cloudflare's cipher configuration (add to http or server context):

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:EECDH+RC4:RSA+RC4:!MD5;
ssl_prefer_server_ciphers on;

Apache users can add (to server or virtual host context):

SSLProtocol all -SSLv2 -SSLv3

And dirty IIS users get what they deserve - a 17 step procedure where you have to fiddle with the registry.

You should of course also take a look at any other services which are likely to support SSLv3, like mail or VPNs.

Testing your connection

As with heartbleed, Qualys SSL Labs seems to be the place to go for an easy and comprehensive assessment of your SSL security. Alternatively you could use openssl:

openssl s_client -connect [server]:[port] -ssl3

If you run that command and connect successfully, you're still vulnerable.

Canvas bezier curves

Mon, 09 Dec 2013 11:42:11 +0000

The new Christmas design for this site uses bezier curves on canvas elements to generate random snowdrifts behind the header. Drawing a bezier curve is pretty simple, so seems like a reasonable place to start my new blog. First you need a canvas and a context:

<canvas id="myCanvas" width="400" height="200"></canvas>
<script>
var canvas = document.getElementById('myCanvas'),
    context = canvas.getContext("2d");
;
</script>

Think Turtles

A quick aside for those who haven't used the canvas before: a canvas element has a context property, which is what you draw on. Context methods to draw lines take destination coordinates, and draw them from the current position - think of pen up in Logo's turtle graphics, only with absolute positioning.

The canvas equivalent of pen up is context.moveTo(x, y), so to draw a line from (50, 50) to (150, 50), you would say:

context.beginPath();
context.moveTo(50, 50);
context.lineTo(150, 50);
c.strokeStyle = '#000';
c.stroke();

You'll notice I added some extra bits there - beginPath() says we're starting a new path, and stroke() draws it using the current style. Call beginPath() whenever you're about to change the style.

The canvas bezier curve takes 6 arguments; two sets of control points, and an end point:

context.bezierCurveTo(c1x, c1y, c2x, c2y, ex, ey);

Here's an example:

context.beginPath();
context.moveTo(50, 50);
context.bezierCurveTo(50, 150, 250, 150, 350, 50);
context.strokeStyle = '#00d';
context.stroke();

I've added some grey lines and red markers to show what's going on. First we moveTo(50, 50), then call bezierCurveTo with the two control points (50, 150) and (250, 150), and the end point (350, 50). The result is a line from the current position to the end point, pulled out of place by the control points.

Once you've got that far, building a randomly-generated snowdrift is trivial - a bit of Math.random(), and instead of context.stroke() use context.fillStyle and context.fill().

To finish the effect, I'm drawing two background snowdrifts on one canvas element, and a foreground snowdrift on a second, with snowflakes falling in between - although I cheated there and used an animated gif which I had rendered earlier. Your CPU thanks me.

TCMI 2.0

Tue, 03 Dec 2013 13:19:05 +0000

I am very proud to announce my finest work to date - the Tacky Christmas Music Interface 2.0!

The old TCMI played midi files, but most browsers seem to struggle with those these days, and even when it did work, modern soundfonts made the experience quite variable. To get around this, the new version of TCMI now uses HTML5 audio to play MP3 or OGG files (depending on browser support).

This also means that if you've got the relevant PPL/PRS license, you can now play non-tacky music to your visitors! Although I'm not quite sure what the point of that would be.

The old version also ran in an iframe, which made front page SEO and deep-linking difficult for 1/12th of the year. This version therefore has the option to use AJAX loads and history.pushState, with the controls in a styleable <div> overlay.

There's also a new cleaner design, but those who want a tacky UI to go with the tacky music can opt to use the classic design.

You can find out more on the TCMI project page, where you can also download some of the old TCMI midis which I've converted into MP3s and OGGs. Improvements welcome on github.

A New Blog

Sun, 01 Dec 2013 01:15:00 +0000

Welcome to my new tech blog, where I'll be writing about programming, sysadmin and anything else related to computers that I think may be of general interest to strangers on the internet.

I've been on the internet for about 16 years now, and I've had this site for almost 14 of them. I want to get back to sharing some of the stuff I'm working on, so this blog will be a mix of articles and tutorials covering various topics related to web development. I also want to learn things through this process too - as a freelance developer, I've spent a lot of the past 10 years working away on my own, and I'm sure I've missed some tricks - so please do send me feedback, either in the comments or through the contact form.

For those who have been visiting my site for years, I'll still keep my personal diary going, but from now on that will be where I ramble on about my cats, and things that annoy me which I can't fit into 140 characters on twitter.

For those who are new, welcome, and I hope that you will find my blog useful.