Last minute talk on automated Perl builds using Hudson tonight

My friend Scott McWhirter, who heads up the Vancouver Perl Monger’s group, asked me yesterday to give a last-minute talk on anything in particular at tonight’s meeting. He wasn’t exactly begging, but I know he’s short on speakers this month, and he wanted something interesting to show.

So I decided I’d talk about building and testing Perl-based projects using Hudson. I’ve been planning on writing a blog post on the subject for the past month, but haven’t found the time to finish off the post properly. So if you’re interested in the topic, and you don’t want to wait for me to get around to writing about it online, please feel free to drop by tonight!

Update: The talk went well! Until I have time to put a more comprehensive post up on the topic, you can always read the slides from tonight’s talk.

How to automate your iPhone app builds with Hudson

As any iPhone application developer who’s released at least a single app to the App Store will tell you, releasing your app is a terrible pain in the…well, it’s not a fun experience.  After your second or third app you start to get the hang of things, but there’s still pain and suffering involved.  Managing certificates, getting settings configured properly, and iterating between development, AdHoc beta builds, and the final App Store release builds, all make the process seem tediously manual and prone to human error.

In professional software development shops, you would use a Continuous Integration server to monitor your source control repository, check out changes as they’re submitted, compile, test and package up builds, and notify developers of the build’s health via emails and a web-based “Dashboard”.  I missed having this while developing my PhoneGap-based iPhone applications, so I decided to once and for all bring good development practices to my iPhone work.

Why do I need to configure automated builds anyway?

I get this a lot from people when I’m trying to convince them of the need for automated builds.  I personally find it hard to imagine people getting by without them in a single-developer project, let alone when multiple developers contribute to a project.

Monitoring the health of an application

Lets face it, we’re human, and we make mistakes.  It’s alright to break code from time to time, but what really sucks is when you find out far too late.  Did your recent changes accidentally eliminate your Entitlements.plist file, thus breaking distribution or release builds?  Do you have a file or library you forgot to check in, meaning when you delete the project from your working directory all those changes will just vanish?

Instead of having to remember to check each of those things manually (which, lets face it, you’ll forget at least half of the things you’re supposed to do inevitably), why not have an automated system tell you every time you make a change?  And if you’re in a multi-developer project, you’ll be able to see who broke the build and what change specifically broke it.

Always be ready for distributing your application

Many times in the natural course of development you’ll break code.  You’ve gotta break something in order to improve it.  But sometimes someone (your wife, a client, a beta tester) will want to try out your application before you have an opportunity to finish off your recent changes.  Instead of spending ages back-tracking your work to get your application to compile, why not rely on your automated build system to keep archives of previously successful builds?

Release what you test

Since you want to test an application before you release it to the App Store, you’ll probably create an Ad-Hoc distribution build to give to friends, family, or official beta testers before you bundle your application up to send to the App Store.  Maybe your testers will find bugs, maybe they won’t.  But at the end of the day that compiled app bundle you just created isn’t actually what you submit to Apple.  You need to compile a completely different app bundle with very different files stored in a Zip file, and if you’re not careful you could potentially be releasing something different than what you tested.

Why not have your automated build system create both your Ad-Hoc distribution build as well as an App Store release build every time?  That way you’re not only always ready to release something to the App Store, but you can be guaranteed that you’re submitting to Apple the exact code that your testers evaluated.

More benefits than I can list

If you’re really serious about best practices, you’ll probably want to write unit tests for your code and have those run after your code has been compiled, but before your build is packaged and archived.  Just because your code compiles doesn’t mean that it will behave correctly.  And lets face it, if you have a lot of tests, you’ll never wait for all of them to run throughout the course of your work.  So by running your tests as a prerequisite to a build succeeding, you’re guaranteed that you’ve got a safety net.

There’s plenty of other best practices that having an automated build system can help with, so what I’m discussing here will just cover the tip of the iceberg.  If I’ve convinced you that automating your builds, read on.

Continue reading “How to automate your iPhone app builds with Hudson”

Managing sites with Git and intelligent post-update hooks

I’ve recently begun drinking the koolaid of Git, and damn it’s tasty! The things I can do with git that I couldn’t have done before (or would have been difficult to do) makes me excited about it. In fact, the one feature that I thought was a drawback — the no-one-true-server nature of it — is actually its strongest selling point.

See, the way I’ve taken to doing my development now is I create two “remote” repositories. First is “origin” which points to a repository managed by Gitosis. Second is a “live” repository that points at a working directory on my production server. That working directory is where my live site actually runs in.

On its own this is handy. As I develop new features, I push my changes to origin. Once my code is ready, and after I run “make test” to verify my site passes all its unit tests, I push to my live repo. At this point I ssh to the server, restart my server processes, and in theory all should be well.

The need for automation

I discovered quickly that in practice this was fraught with error. After a fairly large refactor, I found that the code that worked perfectly well on my development laptop fell over on production. Old or missing libraries, dependancy problems, ownership permissions on files, you name it. My site was down for 2 hours while I tried to resolve these issues. I added more unit tests to my code, but still this wouldn’t have caught these problems.

At this point I decided that a more automated approach was needed. I used a friend’s post-update hook as a template, which simply merged in changes and restarted nginx following a push to live. To this I added a long one-liner, and added the relevant commands to /etc/sudoers with the “nopasswd” option. In the end, the function looks like this:

bounce_fcgi() {
  echo $PWD >&2
  [ -f Makefile ] && make clean >&2
  perl Makefile.PL >&2 && make test >&2 &&
    sudo /etc/init.d/nginx test 2&>/dev/null &&
    sudo /etc/init.d/nginx reload &&
    sudo /etc/init.d/webapp restart >&2

Essentially, after my new changes are merged in, it:

  1. Creates and runs my Makefile
  2. Runs all my unit tests
  3. Tests my new nginx server configuration
  4. Reloads my new nginx configuration
  5. Restarts my FastCGI web application

If any of those steps fails, the full process halts. All the >&2 arguments ensure that the output of these commands are echoed to my local console from the remote server. So when I type “git push live”, all the test output is displayed to me inline. If an error occurs, I can immediately fix it in my local environment and push out a new change without having to log in to my remote server once.

My web application is written in Catalyst, and I use Test::WWW::Mechanize::Catalyst extensively, so I not only unit test my back-end classes, but I test the URLs users actually interact with. It even goes so far as creating and destroying test accounts within my database, so every feature of the website is tested, right down to validating the contents of my robots.txt and sitemap.xml files.

Next Steps

There are still some holes that need to be filled here.

  • If running my tests fail for some reason, I would like to roll-back the remote working directory to the previous version and restart my services, that way the site continues to function under a known-good state.
  • I would like to use WWW::Google::SiteMap::Ping to notify Google, and perhaps other search engines that support XML Sitemaps, that the contents of my site have changed and a reindexing is needed.
  • My site is localized, so I would like to regenerate my PO translation files, and if any strings have changed or are out of date, automatically send an email to my translators with the new POT template file attached.
  • Run my HTML through a spelling checker, to verify I don’t put any typos in any of my pages.
  • Since I try to use caching as much as possible, when a web page’s content has changed, I would like to automatically connect to my Memcached servers to purge the relevant pages from its cache so new versions are immediately available.

A fringe benefit is makes it fun and exciting to write unit tests! Whenever I find an area that isn’t covered, I sit down and crank out more tests to validate new features. I never want to be caught with my pants down on a deployment. Each time it catches something I missed, it makes the whole thing worthwhile.

Does anyone have any thoughts on how I can improve this? Is anyone doing something similar that I can learn from? I’m very excited about my new deployment system, and wish I’d had this ages ago. If there’s anything you want to know more about, please leave a comment.

Reblog this post [with Zemanta]