Versioned Media and Expires Headers in Django

We try to make our sites as responsive as possible, and as part of our testing, we realised that we should do the right thing and add Expires Headers to our static media. Our web servers are configured so when a client requests an image, stylesheet or JavaScript file, it is returned along with a far-future expires header. This tells the client not to ask for that file again, but to cache it for a month or more.

Encouraging Caching with Expires Headers

Without an expires header, the client will request media files each time it loads a page. Using if-modified-since and etag headers, the server usually doesn’t return the media files, but instead returns a 304 Not Modified response. Not resending the data is good. Not having to deal with the request at all is even better – that’s what expires headers offer.

Of course, if you tell clients not to request your stylesheet again for a month, what happens when you change your stylesheet? The client won’t know and won’t get the changes. That’s pretty disastrous. What we need is a way of changing the URL when our media changes so that clients will pick up the new version.

Changing URLs when Content Changes

There are a number of ways to serve your media so you can specify far-future dates in the expires header, but still have the client pick up new versions. We refer to this as versioned media.

One common scheme is to put the modification date of the file in its URL. When the file date changes, the URL changes and clients request the new version. The URL might look something like /media/main.css?200812180930 or /media/main-2008-12-18-0930.css. The former is easier because the querystring is ignored by the web server and the file returned as normal.

Using the date is good if you want more granular per-file versioning, but it seems a little messy. We decided to use a version number in the URL instead, e.g. /media/v123/main.css. To make this work we need to put a version number in the templates and have our web server ignore the version number and just serve the file.

Versioned Media Context Processor

Typically, Django-based sites use the MEDIA_URL context processor to include external resources such as Javascript and images in to their templates. We expanded on this idea by having VERSIONED_MEDIA_URL which puts the version number in as well.

Remembering to update a version number would be error prone, so we wanted to transparently support any updates to media. We use Subversion for version control, and figured out that we could use the versioning metadata of our media directory to help us generate a unique path, which would change as the media was updated. That’s exactly what we needed, and would allow us to specify expires headers on all paths which include a version number but still ensure that users would receive new copies of files if any changed.

from django.utils.version import get_svn_revision
from django.conf import settings

VERSIONED_MEDIA_URL = None

def get_versioned_media_url():
    if hasattr(settings, 'MEDIA_VERSION') and settings.MEDIA_VERSION is not None:
       version = 'v%s' % settings.MEDIA_VERSION
    else:
        revision = get_svn_revision(settings.MEDIA_ROOT)
        version = revision.replace('SVN-', 'v')
    return u'%s%s/' % (settings.MEDIA_URL, version)

def versioned_media(request):
    """Adds versioned media url to the context."""
    global VERSIONED_MEDIA_URL
    if not VERSIONED_MEDIA_URL:
        VERSIONED_MEDIA_URL = get_versioned_media_url()
    return {'VERSIONED_MEDIA_URL': VERSIONED_MEDIA_URL}

You can see from the code that we’ve added a MEDIA_VERSION setting which is either manually set, or can be set by a deployment script. We make use of django’s get_svn_revision method to pick up the version number from our MEDIA_ROOT and we then append our version number to our MEDIA_URL, adding it to context as VERSIONED_MEDIA_URL.

It’s convenient to update code on your server with a simple svn up, but serving from a working copy may have security issues. Instead, we have a deployment script which updates a working copy then copies the files (excluding .svn directories) to the directories used by the web server. It finds the revision number and writes that to the settings file so our version number is still updated automatically.

Configuring Expires Headers in Nginx

Now that we have a version-specifc URL for all our media, we need to configure the webserver to add an Expires Header to any requests which are destined for a versioned URL. We use Nginx, but the theory applies to any webserver.

location /versioned-media {
    internal;
    expires 90d;
    alias   /srv/www/live/thebarbershop/site/media;
}

location /media {
    rewrite /v(?:\d+)/(.*) /versioned-media/$1;
    rewrite /vunknown/(.*) /media/$1;
    root   /srv/www/live/thebarbershop/site;
}

We configure our /media URL so that any request which matches the version string created by our context processor is forwarded to the /versioned-media path, which then applies the expires header and sets the expiry date to 90 days in the future. Any request path without a version number simply gets served without the expires header.

One drawback: committing a change means that all versioned media URLs change, not just the one for the file that changed. However, we feel this is only a small drawback given the advantages this gives for the common case of a high-traffic site with relatively infrequent changes to the base media.

When you couple adding Expires Headers with other techniques like:

• YUI compressing your javascript and css
• GZip compression for your media
• Django’s GZipMiddleware for your templates
• CSS spriting,

you can dramatically reduce both the number and size of requests to your application, and give users a more responsive experience.

Handling Subdomains in Django

This is the first part of our series on some of the more interesting tech we’ve developed for Red Robot Studios. We’re working from webserver up, so we thought that subdomains would be a good place to start.

Subdomains are useful when you want to host multiple sites with the same code and different data. For example, providing websites for clubs where each club has its own subdomain. In Django, you could have a Club model and some associated models holding data. When a user visits alpha.example.com, you want to show the data from the Club model instance for alpha.

DNS Subdomains and Wildcards

You could add individual subdomains using DNS CNAMEs or A records, but if you want to generate objects in your Django app on the fly, have a look at wildcards. To match any subdomain you add “*” as a subdomain. It looks something like:

*.example.com 14400 IN A 208.77.188.166

This will match www.example.com as well as alpha.example.com, beta.example.com, etc.

Webserver Wildcards

The webserver config needs a similar setting so it knows to respond to any subdomain.

In Nginx, it looks something like:

server {
    listen       208.77.188.166:80;
    server_name  example.com *.example.com;
...

In Apache:

<VirtualHost 208.77.188.166>
    ServerName example.com
    ServerAlias *.example.com
...

If you want to use SSL certificates on your subdomains, you need to get a “wildcard subdomain certificate”. They cost more than a regular certificate, but are necessary to provide a valid certificate for any subdomain on your site. We go ours from RapidSSLOnline.

This should be enough config for your Django app to receive requests for any subdomain. Now your Django app needs to respond appropriately for each subdomain.

Getting the Subdomain in Django

There are a few different ways to do it, but we went with a piece of middleware that gets the subdomain from the request and retrieves a matching model.

We have added two additional settings: DOMAIN_MIDDLEWARE_MODEL and DOMAIN_MIDDLEWARE_INSTANCE_NAME to our settings.py so we can specify the model which the middleware queries, and the name which the instance is given when added to our request instance.

DOMAIN_MIDDLEWARE_MODEL = 'core.Club'
DOMAIN_MIDDLEWARE_INSTANCE_NAME = 'club'

The model is assumed to have a field named “slug”, which the middleware uses to match the subdomain against an instance of the model.

Right, so let’s create our middleware:

class DomainMiddleware(object):
    """Gets the correct instance of an application-specific model by matching the
    sub-domain of the request."""

    def __init__(self):
        self.site_domain = Site.objects.get_current().domain
        if self.site_domain.startswith('www.'):
            self.site_domain = self.site_domain[4:]
        self.SUBDOMAIN_RE = re.compile(r'^(?:www\.)?(?P<slug>[\w-]+)\.%s' % re.escape(self.site_domain))
        try:
            app_name, model_name = settings.DOMAIN_MIDDLEWARE_MODEL.split('.', 2)
            self.model = get_model(app_name, model_name)
            self.instance_name = settings.DOMAIN_MIDDLEWARE_INSTANCE_NAME
            assert self.instance_name
        except (AttributeError, AssertionError):
            raise ImproperlyConfigured('DomainMiddleware requires DOMAIN_MIDDLEWARE_MODEL and DOMAIN_MIDDLEWARE_INSTANCE_NAME settings')

In our init method we do some basic setup like creating a regex which will match the subdomain slug, and loading in our model using django.db.models.get_model with the app.model args from DOMAIN_MIDDLEWARE_MODEL.

    def process_view(self, request, view_func, view_args, view_kwargs):
        """If domain is not main site, check for subdomain.

        Get the model from the subdomain slug.
        """
        port = request.META.get('SERVER_PORT')
        domain = request.META.get('HTTP_HOST', '').replace(':%s' % port, '')
        if domain.startswith('www.'):
            domain = domain[4:]
        if domain != self.site_domain:
            match = self.SUBDOMAIN_RE.match(domain)
            if match:
                slug = match.group('slug')
                instance = get_object_or_404(self.model, slug=slug)
            setattr(request, self.instance_name, instance)
        return None

In process_view we grab the subdomain from the HTTP_HOST header of the request, and using get_object_or_404 we load the correct instance of the model and set it as an attribute on our request object with the name given in DOMAIN_MIDDLEWARE_INSTANCE_NAME.

When someone goes to alpha.example.com the middleware picks out alpha and gets the Club instance with slug=alpha and adds it to request to be used in views.

The middleware uses Site.objects.get_current() to get the base URL, so make sure you have Site set up properly or none of your subdomains will match.

The advantage of having the middleware load in the correct instance for you is that your views can simply use the club attribute to access all related data for this club.

def index(request):
    members = request.club.members.approved().order_by('-creation_date')
    return list_detail.object_list(
        request                 = request,
        queryset                = members,
        template_name           = 'club/index.html',
        template_object_name    = 'member'
    )

This is extremely useful when you want to ensure that your data is correctly filtered: you don’t need to have each view filter based on the subdomain, which is pretty error-prone, and if you get it wrong a user’s data would end up on someone else’s page. This way is a lot simpler.

Job Boards now with Submit, Analytics and API

Instead of launching a separate version of Fuselagejobs we decided to tighten our focus and add new features to the existing job boards.

• You can now allow visitors to submit job listings to your job board. If you approve the job listing, it will be displayed on your job board.
• We’ve improved the management interface where you can add and manage jobs and contact details.
• You can now use Google Analytics to track visitors to your job board.
• We added a lightweight JSON API to make it easier to display your jobs on other websites. Widgets coming soon.
• We’re confident these new features make the job boards more useful and better value. We’d love to hear your feedback on these changes or anything you’d like to see.

• Get your own job board now on Fuselagejobs – it’s free to try.

Getting Satisfaction from Red Robot Studios

When we started Red Robot Studios earlier this year, we had a couple of guiding principles we wanted to make our core philosophy:

1. Build great software
2. Keep everything simple until we know what to make complex
3. Let customers and the community guide the direction of the company and its products

With these principles in mind we built our first application Fuselagejobs which is based, in part, on work we did for Djangogigs last year. Fuselagejobs is comfortably lightweight at the moment and performs its core function of managing jobs listings pretty well. However, we’re very much aware that users will want more features and functionality as time goes on. Hence, the need for us to keep in touch with the community’s wants and needs.

I’ll admit it now: we’re not psychic. As we work on products for Red Robot Studios, we can’t know for sure what features people want or expect. The best we can do is to make something we think is good, and ask you for your feedback so we can learn what we did right and what we need to become better at.

So, we’ve started using Get Satisfaction as a way to gather feedback and keep in touch with our customers and the wider community. We hope to get ideas and suggestions as well as address any problems or grievances, and build products which you the community help to direct.

It’s all done “in the open” so we can be transparent and approachable instead of some faceless corporation hiding behind a website.

We will be writing frequently on our blog, and you’re more than welcome to follow us on Twitter too (Andrew and Scott).

So, if you’ve got anything you’d like to discuss, we’d love to hear from you.

Looking for a job board? Try Fuselagejobs.

Why we launched a Job Board app

Our first product is Fuselagejobs which lets you easily start your own job board and list job vacancies online. It might seem like a strange choice for a hosted web application. Job listings have been on the web for a long time and there are established big players as well as open source alternatives. So where do we fit in?

Big Players

The big job databases have lots of traffic, lots of employers and lots of candidates. They list jobs from different industries and rely on search to find what you are looking for. That’s ok, but maybe not the best way for candidates to find interesting jobs and for employers to find the best candidates. It all seems very big and impersonal.

Niche Job Boards

Small job boards that target an industry niche or geographical area are becoming popular. We saw great success with Djangogigs, a job board for web developers who use the Django framework. That’s a good example of a tightly-focussed industry niche. It was launched over a year ago and has had over 500 jobs submitted so far. It quickly became the de facto place for employers to post job ads and candidates to look for work in that niche.

Many of the large blogs and other websites with a lot of users have started their own job boards as a way of helping their community and generating some revenue. The jobs are focussed enough that they are interesting to the website regulars. Employers pay to advertise knowing the people who see their job adverts are the type of candidates they’d like to attract.

DIY for non-geeks

So why not use one of the open source packages to set up your own job board? You could if you have the technical skills or have staff to do that for you. If you don’t want to wait for the IT guys or in-house developers to get round to it, Fuselagejobs is a good alternative. It’s easy to use, requires no technical skills and is fully hosted. Just sign up and start adding jobs within minutes.

Innovation

A basic job board is pretty easy to develop, but we think there’s room for innovation to make it easier for the right candidates to find the right jobs. As always, we’re keen to hear your feedback on what you’d like job boards to do for you.

So that’s why we launched a job board app!

Check out Fuselagejobs and create your own job board for free.

Job boards made easy

We’ve just launched Fuselage which lets you create your own job board.

Each job board has its own website with admin interface and is hosted by Red Robot Studios. It’s very easy to get started and there’s nothing to install – all you need is a modern web browser.

Niche job boards are taking off. A niche job board brings together people with specific skills or located in a particular area. We know from past experience that a focussed community can build around a job board leading to higher quality candidates and more relevant job listings – a win-win situation for employers and candidates.

Fuselage fills the need for people who want to create their own niche job board or for companies or recruiters who need a simple way to list jobs online.

Fuselage is the first app to be released by Red Robot Studios and we have high hopes for it. In the coming weeks we’ll be adding new features and releasing different packages.

If you’re interested in job boards, you can try Fuselage for free.

Better software with less hassle

There’s a lot of software around, but sometimes it causes more problems than it solves. Installing, configuring and maintaining software distracts you from your business. Finding the right software is a significant time investment in itself.

There’s a better way. We’re pleased to announce the launch of Red Robot Studios Ltd. Our mission is to give you better software with less hassle.

• Easy to try
• Quick to get started
• Does what you need without a lot of distractions

Our software is web-based and hosted, so there’s nothing for you to install. You just need a web browser. Try it. If it’s not right for you, adiós. If it does what you need, you can use the software to help run your business and leave us to take care of upgrades, backups and security.

Red Robot Studios Ltd is the brainchild of Andrew Gleave and Scott Barnham, two entrepreneurial software developers from the UK (Isle of Man). With years of experience developing for the web, we have the skills to back up our vision.

Our first product is Fuseleage: an easy way for you to run a niche job board. A niche job board brings together people with specific skills or located in a particular area. Better candidates, more relevant jobs. Use it to advertise positions for your company or organisation, or target a niche and sell job listings to employers for profit.

Thanks for stopping by. We’re hard at work on some better software with less hassle, so don’t be a stranger.