iOS/Web Developer's Life in Beta

I’ve already posted before on how to set up Hudson to compile and build iPhone applications, but I just had a “OMG I <3 Hudson!” moment just now, and felt I had to share it.

I do most of my mobile development literally while I’m mobile; on the train during my morning commute, from coffee shops on the weekend, or in front of the TV in the evenings when I’m winding down for the night. Because of this, I don’t have any consistent time when I’m making checkins, nor do I have the time to create builds for my beta users.

One of the iPhone apps I’m building is for a client, and because my time is limited, I want to streamline communication as much as possible, especially since I have a day job that I really like and uses 8 of my precious 24 hours per day. I don’t want to have to tell them every time new features are ready to be tested. Additionally, if I have to manually compile and send a beta build to them whenever I complete a new feature, I’ll never have enough time to actually get any work done. Add to that the fact that many mail servers will block iPhone applications for one reason or another (file size limits, misfiring antivirus filters, or any number of reasons).

So the solution I’ve come up with is an extension of my iPhone application build scripts combined with some nice plugins for Hudson. All of the above steps are completely automated away and handled for me just by checking my code into git.

1. The first thing I did was updated my build scripts to generate an “.ipa” archive for Distribution builds containing an iTunesArtwork file, and a “.zip” archive for Release builds.
2. My build script sets the version number of my application to the $BUILD_NUMBER of my Hudson job, so that the version is guaranteed to be unique.
3. I added the SCP plugin to Hudson and configured a location on my web server where Hudson could upload completed builds to the website. This enables my clients to access the page to download new releases.
4. In order to inform my clients that a new build is available, I’ve used the customizable email support in Hudson to send an email to a custom list of email addresses when a successful build is completed. In this email I include a link to the uploaded IPA so they can easily download the beta archive.
5. So that I don’t have to go through the trouble of describing what changes I made, I use the $CHANGES_SINCE_LAST_SUCCESS variable in my custom email template that lists all the commit messages I’d submitted to git that were included in this build.
6. For good measure, I SCP to the website a copy of the “.mobileprovision” certificate file that was used to build the application, so the beta tester can add it to iTunes if the contents have changed.

The result of all this work is that I can work wherever I am, continually committing changes into git. When I’m ready to cut a new build, I submit my changes to github where a webhook script will notify Hudson that changes have been made. Hudson checks out my changes, builds them on my Mac Mini at home, SCP’s the resulting archive up to my web server, and notifies my beta testers that a new archive is available and what changes they should keep an eye out for.

For a copy of the build scripts and configuration I use inside my projects, take a look here:

This assumes there’s a file in the build user’s home directory called “.build_password” that contains the password used to unlock your keychain. Run “chmod 600″ on that file in order to make it hidden to anyone else on your system.  And make sure you check in the “.mobileprovision” files your XCode project file references.  If you change your certificates, you’ll need to copy the updated mobileprovision files from your “Library/MobileDevice/Provisioning Profiles” directory.

So you can see why I’m excited about this.  I hope you are too.

I’ve recently begun drinking the koolaid of Git, and damn it’s tasty! The things I can do with git that I couldn’t have done before (or would have been difficult to do) makes me excited about it. In fact, the one feature that I thought was a drawback — the no-one-true-server nature of it — is actually its strongest selling point.

See, the way I’ve taken to doing my development now is I create two “remote” repositories. First is “origin” which points to a repository managed by Gitosis. Second is a “live” repository that points at a working directory on my production server. That working directory is where my live site actually runs in.

On its own this is handy. As I develop new features, I push my changes to origin. Once my code is ready, and after I run “make test” to verify my site passes all its unit tests, I push to my live repo. At this point I ssh to the server, restart my server processes, and in theory all should be well.

The need for automation

I discovered quickly that in practice this was fraught with error. After a fairly large refactor, I found that the code that worked perfectly well on my development laptop fell over on production. Old or missing libraries, dependancy problems, ownership permissions on files, you name it. My site was down for 2 hours while I tried to resolve these issues. I added more unit tests to my code, but still this wouldn’t have caught these problems.

At this point I decided that a more automated approach was needed. I used a friend’s post-update hook as a template, which simply merged in changes and restarted nginx following a push to live. To this I added a long one-liner, and added the relevant commands to /etc/sudoers with the “nopasswd” option. In the end, the function looks like this:

bounce_fcgi() {
  (cd $GIT_WORK_TREE
  echo $PWD >&2
  [ -f Makefile ] && make clean >&2
  perl Makefile.PL >&2 && make test >&2 &&
    sudo /etc/init.d/nginx test 2&>/dev/null &&
    sudo /etc/init.d/nginx reload &&
    sudo /etc/init.d/webapp restart >&2
 )}

Essentially, after my new changes are merged in, it:

1. Creates and runs my Makefile
2. Runs all my unit tests
3. Tests my new nginx server configuration
4. Reloads my new nginx configuration
5. Restarts my FastCGI web application

If any of those steps fails, the full process halts. All the >&2 arguments ensure that the output of these commands are echoed to my local console from the remote server. So when I type “git push live”, all the test output is displayed to me inline. If an error occurs, I can immediately fix it in my local environment and push out a new change without having to log in to my remote server once.

My web application is written in Catalyst, and I use Test::WWW::Mechanize::Catalyst extensively, so I not only unit test my back-end classes, but I test the URLs users actually interact with. It even goes so far as creating and destroying test accounts within my database, so every feature of the website is tested, right down to validating the contents of my robots.txt and sitemap.xml files.

Next Steps

There are still some holes that need to be filled here.

• If running my tests fail for some reason, I would like to roll-back the remote working directory to the previous version and restart my services, that way the site continues to function under a known-good state.
• I would like to use WWW::Google::SiteMap::Ping to notify Google, and perhaps other search engines that support XML Sitemaps, that the contents of my site have changed and a reindexing is needed.
• My site is localized, so I would like to regenerate my PO translation files, and if any strings have changed or are out of date, automatically send an email to my translators with the new POT template file attached.
• Run my HTML through a spelling checker, to verify I don’t put any typos in any of my pages.
• Since I try to use caching as much as possible, when a web page’s content has changed, I would like to automatically connect to my Memcached servers to purge the relevant pages from its cache so new versions are immediately available.

A fringe benefit is makes it fun and exciting to write unit tests! Whenever I find an area that isn’t covered, I sit down and crank out more tests to validate new features. I never want to be caught with my pants down on a deployment. Each time it catches something I missed, it makes the whole thing worthwhile.

Does anyone have any thoughts on how I can improve this? Is anyone doing something similar that I can learn from? I’m very excited about my new deployment system, and wish I’d had this ages ago. If there’s anything you want to know more about, please leave a comment.