Introducing django-bakery

    A set of helpers for baking out your Django site as flat files

    Los Angeles Times photographic archive, UCLA Library
    1988: Confectionary carrots come off the line at Van de Kamp's bakery in Los Angeles and are placed on cakes by Martha Cofre, left, and fellow workers.

    When Web traffic spikes and your site starts to sag, your first im­pulse might be to ar­chi­tec­ture up, to add more serv­ers, shard the data­base and cache, cache, cache. Provided you have the skill, time and money, that will get the job done.

    Lack­ing any of those three in­gredi­ents, the only guar­an­teed way to avoid a data­base crash is to not have a data­base. That sounds flip­pant, but it’s true. When faced with high traffic de­mands and little time or fund­ing, the Data Desk does ex­actly that. We save every page gen­er­ated by a data­base-backed site as a flat file and then host them all us­ing a stat­ic file ser­vice like Amazon S3.

    We call this pro­cess “bak­ing.” It’s our path to cheap­er, more stable host­ing for simple sites. We use it for pub­lish­ing elec­tion res­ults, timelines, doc­u­ments, in­ter­act­ive tables, spe­cial pro­jects and even this blog.

    The sys­tem comes with some ma­jor ad­vant­ages, like:

    1. No data­base crashes
    2. Zero serv­er con­fig­ur­a­tion and up­keep
    3. No need to op­tim­ize your app code
    4. You don’t pay to host CPUs, only band­width
    5. An off­line ad­min­is­tra­tion pan­el is more se­cure
    6. Less stress (This one can change your life)

    There are draw­backs. For one, you have to build the bakery in­to your code base. More im­port­ant, a flat site can only be so com­plex. No on­line data­base means your site is all read and no write, which means no user-gen­er­ated con­tent and no com­plex searches. Sites we host that could not be baked in­clude Map­ping L.A. and NHTSA Vehicle Com­plaints, each of which al­lows users to in­ter­act with a large and shift­ing data­set.

    So what’s the trick?

    To stream­line the pro­cess, we de­veloped an open-source Django lib­rary called django-bakery. It makes bak­ing out your site easi­er by in­teg­rat­ing the steps in­to Django’s stand­ard pro­ject lay­out.

    To try it out, the first thing you need to do is in­stall the lib­rary from PyPI, like so:

    $ pip install django-bakery
    

    Then edit your settings.py and its INSTALLED_APPS.

    IN­STALLED_APPS = (
        ...
        'bakery',
        ...
    )
    

    Then add a BUILD_DIR dir­ect­ory path where the flattened site will be baked.

    im­port os
    ROOT_PATH = os.path.dir­name(__­file__)
    BUILD_DIR = os.path.join(ROOT_PATH, 'build')
    

    The cru­cial step is to re­fact­or your views to in­her­it our class-based views. They are de­signed to auto­mat­ic­ally flat­ten them­selves. Here is a list view and a de­tail view us­ing our sys­tem.

    from yourapp.mod­els im­port Dummy­Mod­el
    from bakery.views im­port Build­able­De­tailView, Build­ableL­istView
    
    class DummyL­istView(Build­ableL­istView):
        """
        A list of all tables.
        """
        queryset = Dummy­Mod­el.live.all()
    
    class DummyDe­tailView(Build­able­De­tailView):
        """
        All about one table.
        """
        queryset = Dummy­Mod­el.live.all()
    

    After you’ve con­ver­ted your views, add them to a list in settings.py where all build­able views will be stored.

    BAKERY_VIEWS = [
        'yourapp.views.DummyL­istView',
        'yourapp.views.DummyDe­tailView',
    ]
    

    Then run the man­age­ment com­mand that will bake them out.

    $ python manage.py build
    

    That should cre­ate your build dir­ect­ory and flat­ten all the des­ig­nated views in­to it. You can re­view its work by fir­ing up the buildserver, which will loc­ally host your flat files in the same way the Django’s runserver hosts your data­base-driv­en pages.

    $ python manage.py buildserver
    

    To pub­lish the site on Amazon S3, all that’s ne­ces­sary yet is to cre­ate a buck­et. You can go to aws.amazon.com/s3/ to set up an ac­count. If you need some ba­sic in­struc­tions you can find them here. Now set your buck­et name in the settings.py file:

    AWS_BUCK­ET_­NAME = 'my-buck­et'
    

    Next, in­stall s3cmd, a util­ity we’ll use to move files back and forth between your desktop and S3. In Ubuntu, that’s as simple as:

    $ sudo apt-get install s3cmd
    

    If you’re us­ing Mac or Win­dows, you’ll need to down­load this file and fol­low the in­stall­a­tion in­struc­tions you find there.

    Once it’s in­stalled, we need to con­fig­ure s3cmd with your Amazon lo­gin cre­den­tials. Go to Amazon’s se­cur­ity cre­den­tials page and get your ac­cess key and secret ac­cess key. Then, from your ter­min­al, run:

    $ s3cmd --configure
    

    Fi­nally, now that everything is set up, pub­lish­ing your files to S3 is as simple as:

    $ python manage.py publish
    

    The next level

    If your site pub­lishes a large data­base, the build-and-pub­lish routine can take a long time to run. Some­times that’s ac­cept­able, but if you’re peri­od­ic­ally mak­ing small up­dates to the site it can be frus­trat­ing to wait for the en­tire data­base to re­build every time there’s a minor edit.

    We tackle this prob­lem by hook­ing tar­geted build routines to our Django mod­els. When an ob­ject is ed­ited, the mod­el is able to re­build only those pages that ob­ject is con­nec­ted to. We ac­com­plish this with a build meth­od you can in­her­it. All that’s ne­ces­sary is that you define a list of the de­tail views con­nec­ted to an ob­ject.

    from django.db im­port mod­els
    from bakery.mod­els im­port Build­ableMod­el
    
    class Dummy­Mod­el(Build­ableMod­el)
        de­tail_views = ('yourapp.views.DummyDe­tailView',)
        title = mod­els.Char­Field(max_length=100)
        de­scrip­tion = mod­els.Text­Field()
    

    Now, when obj.build() is called, only that ob­ject’s de­tail pages will be re­built. If oth­er pages ought to be up­dated as well, par­tic­u­larly if they come from views that don’t take the ob­ject as an in­put, you should in­clude those in the pre-defined _build_related mod­el meth­od called at the end of build.

    from django.db im­port mod­els
    from bakery.mod­els im­port Build­ableMod­el
    
    class Dummy­Mod­el(Build­ableMod­el)
        de­tail_views = ('yourapp.views.DummyDe­tailView',)
        title = mod­els.Char­Field(max_length=100)
        de­scrip­tion = mod­els.Text­Field()
    
        def _build_re­lated(self):
            """
            Re­build the sitemap and RSS feed as part of the build routine.
            """
            im­port views
            views.Sitem­apView().build_queryset()
            views.DummyRSS­Feed().build_queryset()
    

    With this sys­tem in place, a up­date pos­ted to the data­base by an entrant us­ing the Django ad­min can set in­to mo­tion a small build that is then synced with your live site on Amazon S3. We use that sys­tem to host ap­plic­a­tions with in-house Django ad­min­is­tra­tion pan­els that, for the entrant, walk and talk like a live data­base, but then auto­mat­ic­ally fig­ure out how to serve them­selves on the Web as flat files. That’s how a site like timelines.latimes.com is man­aged.

    Fi­nally, to speed the pro­cess a bit more, we hand off the build from the user’s save re­quest in the ad­min to a job serv­er that does the work in the back­ground. This pre­vents a push-but­ton save in the ad­min from hav­ing to wait for the en­tire build to com­plete be­fore re­turn­ing a re­sponse. Here is the save over­ride on the Timelinemod­el that as­sesses wheth­er the pub­lic­a­tion status of an ob­ject has changed, and then passes off build in­struc­tions to a Cel­ery job serv­er.

    @trans­ac­tion.com­mit_­manu­ally
    def save(self, *args, **kwargs):
        """
        A cus­tom save that bake the page and re­pub­lishes it when ne­ces­sary.
        """
        log­ger.de­bug("Sav­ing %s" % self)
        # if obj.save(build=False) has been passed, we skip everything.
        if not kwargs.pop('build', True):
            su­per(Timeline, self).save(*args, **kwargs)
            trans­ac­tion.com­mit()
        else:
            # First fig­ure out what we're go­ing to have to do after we save.
            # If the timeline has not yet been cre­ated...
            if not self.id:
                if self.is_pub­lished:
                    ac­tion = 'pub­lish'
                else:
                    ac­tion = None
            else:
                cur­rent = Timeline.ob­jects.get(id=self.id)
                # If it's been un­pub­lished...
                if not self.is_pub­lished and cur­rent.is_pub­lished:
                    ac­tion = 'un­pub­lish'
                # If it's be­ing pub­lished...
                elif self.is_pub­lished:
                    ac­tion = 'pub­lish'
                # If it's re­main­ing un­pub­lished...
                else:
                    ac­tion = None
            su­per(Timeline, self).save(*args, **kwargs)
            trans­ac­tion.com­mit()
            log­ger.de­bug("Post-save ac­tion: %s" % ac­tion)
            # Do whatever needs to be done
            if ac­tion:
                if ac­tion == 'pub­lish':
                    tasks.pub­lish.delay(self)
                elif ac­tion == 'un­pub­lish':
                    tasks.un­pub­lish.delay(self)
    

    The tasks don’t have to be com­plic­ated. Ours are as simple as:

    im­port sys
    im­port log­ging
    from django.conf im­port set­tings
    from cel­ery.dec­or­at­ors im­port task
    from django.core im­port man­age­ment
    log­ger = log­ging.get­Log­ger('timelines.tasks')
    
    @task()
    def pub­lish(obj):
        """
        Bake all pages re­lated to a timeline, and then sync with S3.
        """
        try:
            # Here the ob­ject is built
            obj.build()
            # And if the set­tings al­low pub­lic­a­tion from this en­vir­on­ment...
            if set­tings.PUB­LISH:
                # ... the pub­lish com­mand is called to sync with S3.
                man­age­ment.call_­com­mand("pub­lish")
        ex­cept Ex­cep­tion, exc:
            log­ger.er­ror("Task Er­ror: pub­lish",
                ex­c_info=sys.ex­c_info(),
                ex­tra={
                    'status_­code': 500,
                    're­quest': None
                })
    
    @task()
    def un­pub­lish(obj):
        """
        Un­bake all pages re­lated to a timeline, and then sync to S3.
        """
        try:
            obj.un­build()
            if set­tings.PUB­LISH:
                man­age­ment.call_­com­mand("pub­lish")
        ex­cept Ex­cep­tion, exc:
            log­ger.er­ror("Task Er­ror: un­pub­lish",
                ex­c_info=sys.ex­c_info(),
                ex­tra={
                    'status_­code': 500,
                    're­quest': None
                })
    

    And that’s it. These tools have already proven use­ful for us, but have only been sketched out. All of the code is free and open on Git­Hub and any con­tri­bu­tions or ad­vice is wel­come.

    Much re­spect due

    This ap­plic­a­tion was made in close col­lab­or­a­tion with Ken Schwencke, my part­ner here at the Data Desk. Without his ima­gin­a­tion, cri­ti­cism and con­tri­bu­tions, most of our work would be im­possible, in­clud­ing this lib­rary.

    Also, I presen­ted the ideas be­hind django-bakery last month to a group of our peers at the 2012 con­fer­ence of the Na­tion­al In­sti­tute for Com­puter-As­sisted Re­port­ing. The NICAR com­munity is a con­stant source of chal­lenge and in­spir­a­tion. Many of our ideas, here and else­where, have been ad­ap­ted from things the com­munity has taught us.

    Readers: What’s your take? Share it here.

    Advertisement

    Latest work

      About The Data Desk

      This page was created by the Data Desk, a team of reporters and Web developers in downtown L.A.