iOS/Web Developer's Life in Beta

I’m an avid “Idea Man”.  I love coming up with new ideas; for iPhone apps, for web apps, and even for real-world inventions.  Most of my ideas only sound great in my head, but when I open my mouth the idea seems to turn sour.  A smaller number of ideas manage to survive the thought-to-word boundary.  An even smaller minority of those ideas manage to make it down onto my “Idea Book” that I use to keep track of all the potential projects I’m going to work on.  Admittedly it’s not so much a single “book” as a collection of binders, scraps of paper, and in some cases 3×5 index cards.

Suffice to say amongst all those ideas, some gems manage to stand out above the rest.  A few I’ve actually finished, such as Boomle and myDrumPad.  The end product is seldom what I’d planned when I came up with the idea, but either due to time constraints, or the project evolving during the course of its development, things change.  Mostly I’m pleased with the end result, but there are some things I wish I had time for.  For instance, I’d like to re-implement Boomle using Cocos2D to make the animations smoother, and maybe add some extras such as obstacles, more challenging levels and Game Center integration.

Unfortunately, while I absolutely love myDrumPad, my original plans for it involved the creation of loops and patterns within the app.  Once I built it and started playing with it in real-time, as opposed to my ideas jotted down on paper, I discovered that the interface — while easy to use for quickly tapping out beats — doesn’t lend itself easily to creating loops.  I could create loops, sure, but my goal was to create an app that could be used in live performances, and sadly the interface just doesn’t lend itself to real-time editing of loops.  That’s not to say that I’m unhappy with myDrumPad.  Sales are going fairly well, and I have a few more updates planned.  I’m still adding additional languages to it using ICanLocalize.com (currently it has native support for English, Korean and French, and Japanese and German are following soon).

One of my constant loves for as long as I have been programming has been music and sound production.  And since I’m a rather unconventional thinker, I’d like to try my hand at an unconventional music interface.  I like the idea that someone without any knowledge of music notation or performance can learn to use my apps to produce the music they hear in their head, and to express themselves in ways they wouldn’t otherwise be able to.

I’m still formulating an idea in my head, but the loop and patterns features I was planning for myDrumPad are going to be built into a new application instead, with a completely different interface.  Like myDrumPad, it will be an experiment to see if producing music in this new UI metaphor will be successful.  It’s hard to explain, so I’m going to wait until I have a more accurate prototype to show.

But this app idea doesn’t exist in isolation.  In my little book of ideas I have several on-the-go prototypes just itching to get started, but since I still maintain a day job (and will for the foreseeable future) my time is limited to a few hours per day, and weekends.  My web-based space game, my photo sharing web application, my sonar-based iPhone utility app…all these ideas are stuck in my head as fantastic ideas I’d love to pursue, if only I had the time.

I’m sure it will be an interesting adventure to see which idea wins out in the end.  Do I build a web-based MMOG space game set in an infinitely-scalable universe?  Or do I build a Core Audio-based iPhone utility?  How do I determine which project has the highest likelihood of succeeding, or at least have the greatest likelihood of being finished in a reasonable time-frame by a single developer?

In an ideal world I’d build all of them.  Unfortunately, as an idea man, I think of ideas of varying awesomeness faster than I can build them.  As long as I require sleep, I suppose I’ll be stuck in this situation.

Things have been busy over the past few days, which is the reason why I haven’t had a chance to post about this until now. But for the PhoneGap community, I have some good news and some bad news.  First, the good news: I got a phone call from the Apple app reviewer that was reviewing my test app.  And before I go any further, I want to say a few things.

When I crafted my original letter to Apple, I was very cautious how I wrote it, because I don’t want to give anyone the wrong impression. I don’t have any problems with Apple or the fact that they have an app review process. I actually think the strict regulations Apple has with its review process is a good thing for users. If left to their own devices, app developers would release anything and everything whether it had merit or not, in the hopes of earning even a tiny bit of a buck.

That’s not to say that I have nothing but a big warm and fuzzy feeling for the review process, just that it’s like going to the dentist. It’s painful, but you’ll thank them later in life when your teeth would otherwise be falling out – or, in the case of the App Store review process, when the app store would otherwise be overrun with endless seas of buggy and non-functional crap.

Most developers out there seem to vilify the app reviewers, making them out to be legions of sadistic bureaucrats who like nothing more than to waste developer’s time. I’ve never been comfortable with that, because people simply don’t work that way. The app reviewers don’t have an easy job.  They’re not developers, they don’t know how to write apps, they just review them to ensure they meet the guidelines Apple sets forth.

So when they reject your app, it’s not because they’re out to get you, they’re just enforcing policy.  If they make some assumption about your app, perhaps you’re not giving enough information to help them along?  There’s a comments field that you can use to help the reviewer of your app do their job, instead of throwing an app over the fence and saying “Here, take that”.  They don’t owe you anything, and are simply making sure that they cover their asses.  Could you imagine how bad it looked on the guy who reviewed the “Shaking Baby” app? I’m not sure if he lost his job, or just got seriously reprimanded, but this is the sort of thing these guys are trying to prevent.  But because of the black-box nature of the whole process, it’s easy to jump to conclusions.

Finally, before I dig into the meat of my conversation with my App Store app reviewer, I want to point out a few things that many developers seem to have forgotten in their haste to get their apps released:

1.    The app store is a voluntary process; you don’t want to play by their rules, you don’t have to.
2.    All developers signed an agreement that states Apple can choose to reject any apps they want based on their own reasons. There’s no human right’s violations committed when Apple decides that your app isn’t fit for the store for some reason.
3.    Apple is a business, whose goals are to advance their stock and market position for their investors. While some of Apple’s decisions may seem confusing from outside the black box, keep in mind that they are prone to keeping secrets until they throw down their big announcements at whatever annual conference they’re attending. So there are sometimes more reasons why they wouldn’t want some feature leveraged by app developers if they plan to do something much the same themselves. Is it fair? No. Is this what they’re doing? I have no idea, this is speculation on my part.

Now that I have that out of the way, I’d like to relate to you my conversation with Steve from the App Store.

(more…)

I’ve been using git for all my new software development and have been converting my old Subversion (and *gasp* CVS) repositories over. Throughout this process I keep learning more and more about git, and my love story with it continues every day. Thanks to my friend Chris, he sent along this blog posting that really describes how you can do hardcore agile software development in teams with git. This really is helping me solidify in my head what it is I want to do, but have just never had the tools or know-how to put into practice.

At Hashrocket we use git both internally and in our Agile mentoring and training. Git gives us the flexibility to design a version control workflow that meets the needs of either a fully Agile team or a team that is transitioning towards an Agile process.

via ReinH | A Git Workflow for Agile Teams.

I’m a big fan of all things Apple, and as you can tell from my past blog posts I’m a big fan of iPhone development. I’ve even dusted off my aging C skills, and have learned to love Objective-C. The one thing I haven’t learned to love, like all other iPhone app developers, is their application release process, and the seemingly arbitrary app store acceptance department.

Don’t get me wrong, I think how Apple fiercely guards the App Store to prevent bad, buggy, or offensive material from getting on there is a great thing. Some of my mother-in-law’s students in the class she works in have iPhones or iPod Touches, and these little 10-year-olds love the little apps I’ve put out. They’re fun, light-hearted, and they get a lot of enjoyment out of these and other apps. It’s reassuring to know that if I install an app, it won’t crash my phone (too badly) or that a child won’t be offended by them (too much).

However, the level of detail they give in their rejections seem both arbitrary and unnecessary. You develop an app and have to throw it over the fence to Apple, after which you wait with no level of detail as to the status, or success, or any sense of progress through whatever queue they have. Finally, you more often than not get a big fat rejection letter that gives you no detail as to why your app didn’t meet their secret criteria.

In my experience so far, this has actually been a virtue. My apps actually did have minor race conditions, or problems on specific platform versions. So, fair enough, I fix my changes and submit them in. But recently, the framework I work with and have been helping develop seems to be under fire from Apple for no apparent reason. And it’s the contradictory nature of their message that is what gets under my skin.

In this long thread on the PhoneGap mailing list, a number of developers writing their applications under PhoneGap have been given rejection letters saying something like:

Upon review of your application, cannot be posted to the
App Store due to the usage of private API. Usage of such non-public
API, as outlined in the iPhone SDK Agreement section 3.3.2 is
prohibited:

” An Application may not itself install or launch other executable
code by any means, including without limitation through use of a plug-
in architecture, calling other frameworks, other APIs or otherwise.
No interpreted code may be downloaded and used in an Application
except for code that is interpreted and run by Apple’s Published APIs
and built-in interpreter(s).

The PhoneGap API implemented in your application is an external
framework.

For those new to PhoneGap, it’s a project that gives you an XCode project directory with Obj-C classes pre-made that allow you to develop iPhone apps in HTML / JavaScript. It wraps up several of the iPhone’s native controls and exposes those capabilities to JavaScript. In this way, an application developer could write their app in a hybrid of native Objective-C code and the iPhone’s own native browser control, the UIWebView.

It seems though that Apple constitues the use of their own controls as a 3rd-party library. Apple’s own developer documentation includes code sample projects, much like PhoneGap’s, intended to get a developer started on using their SDK. Is it incorrect to assume that others can do the same?

Additionally, there are 3rd-party companies such as AdMob, Medialets, and a number of others that provide ads, application tracking, and other resources to iPhone developers. Their code is given to you as pre-compiled libraries, that are most certainly not included on the iPhone when you pull it out of its shrink-wrap. So how is it that all these apps are released to the App Store with these 3rd-party libraries linked in, and an app framework whose source code is freely available and uses officially documented features can’t be?

I decided to put an end to the speculation, and wrote a letter to Apple’s developer support on this matter. I’ll give it a chance to percolate through their support department, and if I don’t hear an answer back via email, I plan to call their support department until I can get an answer.

For future reference, and in the interests of keeping the discussion open, here’s a copy of the letter I sent in to Apple.

From: Michael Nachbaur
Date: May 17, 2009 6:04:28 PM PDT (CA)
To: idp-dts@apple.com
Subject: Library classification clarification

Hello, I’m an iPhone software developer, and one of the core developers of the PhoneGap project. A number of users of PhoneGap – a set of Objective-C classes aimed at leveraging the UIWebView to access iPhone-supported hardware features – have reported that their apps have been rejected from the App Store because they supposedly use a “3rd-Party Library”. I wanted to get some clarification about this, as this is not only untrue, it is completely at odds with the goals of our project.

PhoneGap only uses officially-supported features of the iPhone, as documented within XCode’s iPhone SDK documentation. We even make sure that we don’t even use deprecated features of the iPhone, as we want to ensure 100% compatibility. All the software we use is exposed natively by the iPhone, and is in use in many other apps on the App Store.

So I wanted to get an official clarification from Apple as to why these apps are being rejected, and what, if anything, we as the maintainers and developers of the PhoneGap project can do to rectify the situation. We are trying to empower developers with quality starting-blocks for developing their own applications, much as the Apple sample applications do.

So please, if there’s anything we’re doing wrong, we would more than happily change our code to accommodate Apple’s policies. As far as we can tell, we support Apple’s licenses even better than other apps on the App Store, because we don’t import 3rd-party libraries such as AdMob, Medialets, or any of the other ad and tracking libraries that are obviously featured on almost every app in the App Store. And these are quite plainly 3rd-party libraries, and as such should they not be even more restricted than we are?

As a community, of both software developers and of business-people, we are very anxious to hear back from you, and would like to discuss this with someone at Apple to get an official answer, instead of the conversation being one-sided and filled with speculation.

Thank you for your time, and I look forward to hearing back from someone soon.

What are your thoughts on the matter? Can anyone reading this find some reason why Apple would target PhoneGap?

Update: I got an update on this problem from Steve, one of the app reviewers, over at Apple.

Like many people before me, and I’m sure many more after, I’ve had an ungodly amount of difficulty with Apple’s iPhone certificate provisioning system. I personally think it’s a brilliant solution to a difficult problem, but unfortunately for us they made it a pain in the ass to deal with. I’ve only run up against a few basic issues with them, and thought I’d document my solutions to them here.

And before you say it, yes I know there’s already tons of people complaining about iPhone app certificates, and there’s plenty of pages describing how to fix them on the ‘net to go around. But it was like finding a needle in a haystack for me. There were too many hits on forums and mailing lists that gave bad advice, or weren’t relevant anymore due to updates in XCode and the iTunes Connect website. So I’m summarizing here what works for me, and I’ll leave it up to the reader to figure if it works for you.

Entitlements.plist

These aren’t to be used for Debug builds. This file is necessary for some sort of magic when you deploy an app on the phone for either distribution or release mode. A lot of documentation will explain how you have to create it and add it to your Info.plist file.

This is great, but if you’re trying to figure out why your Debug profile isn’t working anymore, don’t attempt to add the Entitlements file to that build profile. It just makes things come to a screeching halt.

Debug vs Distribution vs Release

This one is annoying, and keeps popping its ugly head up. Say for instance I’ve got my developer certificate that I’ve been using for debug builds. Then I want to test my app on a few phones, so I go about creating a distribution certificate for it. In the project settings pane I change the drop-down to the new certificate, hit build, and *bam* I hit a certificate signing error when it tries to install the app on the phone.

As many people on the ‘net have pointed out, XCode is dumb and doesn’t assign the certificate to the correct profile. Some people manage to play around with it enough to get it to work, usually involving lots of restarts of XCode, removing and re-adding certificates, etc. I find some of that works, but when I get frustrated I just drop to VIM on the console, edit my project.pbxproj file, and change my cert settings manually.

If I’m really in a bind and I can’t get XCode to do what I want, I change my debug certificate to use the one I want for distribution and then drop into VIM. From here I can copy and paste the certificate lines it put in there to represent my distribution cert, and I copy it to my distribution or release build section. Then I change my debug cert back to my developer certificate.

Removing old certs from your system

Sometimes you just need to rebuild a cert. You might have made a mistake, or more often you have another phone you’d like to test your distribution build on. This means going into the iPhone Developer Portal and adding a device to the certificate. This means that the cert will now be different, and you’ll need to download the new provisioning profile.

Ah, but you already have that cert in XCode and, as you might have guessed, XCode isn’t smart enough to replace the old cert with the new one. You’ll just get two certs that will fight it out, and will always lose. So go into the XCode organizer and delete your old certs. You add the new cert, it compiles and signs itself, but when you try to install it on the phone it falls flat on its face.

You can go into the Organizer, select your phone, and remove provisioned apps and provisioning profiles from there. Or you can go onto the phone itself, and go to Settings -> General -> Provisioning, and you can delete your certs manually. Once that’s done, you might want to give XCode a restart for good measure and then add the new certificates from the iTunes Developer Portal.

Wildcard Certificates

I haven’t tried this myself personally, but when I started writing iPhone apps I was under the false impression that a more specific certificate was a good thing, as opposed to using wildcard certificates. But I’ve seen tons of references on the web of people recommending using them, and “You’re just asking for trouble if you create different certs for each of your apps”.

All I can say is…Whoops! I already released two apps to the appstore, and I don’t think I can change my certificates after the fact. Plus, I’m not sure if suddenly switching to wildcard certs for the rest of my apps will be a bad thing. Either way I’ll be doing this for my next app. If I only have 2 apps that will give me certificate pain when I maintain them, then so be it.

I’m now a registered iPhone App Developer! After the usual waiting, emailing, phoning, and faxing that I won’t bore you with because blogs all over the world are full of people ranting about Apple’s registration process, I now have my developer key. (Personally, I don’t see what people are complaining about…it wasn’t all that painful.)

So now that I’ve written two iPhone applications, the first thing I wanted to do was load them on the phone and see how they actually behaved on real hardware. The results were, lets just say, less than thrilling. While they ran and did everything I would have expected of them, there are strange inconsistencies between the rendering of WebKit on the phone vs the desktop.

What went wrong

First off, let me state for the record that these are development versions of my apps, and I hadn’t figured on them working well right away. Lets face it, as developers we spend most of our lives working with broken crap, and try our best to make it less broken. That being said, I’m still an optimist and I had hoped.

First, what worked:

1. The app’s icon, start-up screen, and general layout all behaved. Everything was where it was supposed to be.
2. The app started up fairly fast. The only other app I’ve used that’s based on PhoneGap is the Translink public transit app (since I take a combination of a bus and train to get to work). It takes far longer than most apps to start, so I had expected the worse for my app.
3. Clicking on buttons and other UI elements in my first app worked just fine. My second app, which uses more complicated gestures (pinch, rotate, etc) also worked without a hitch.
4. My JavaScript ran quite well, with only a few minor exceptions.

Now on to what didn’t work. Keep in mind, the list above is a lot longer than I’d expected, so the oddities below are tolerable, for the time being.

1. My CSS transitions are behaving very oddly. Instead of smoothly fading from 0% to 100% opacity, they jump between 0% and 100% immediately, back down to 0%, and then fade gracefully back up to 100%.
2. The toolbar of my app, which is implemented as HTML/CSS, flickers whenever I click on buttons anywhere on the screen. This is especially weird since the events for those buttons, located in the center of the screen, don’t touch or interact with the HTML or CSS attributes of the toolbar in any way.
3. For one of my apps I use an element with CSS sprites. None of these seem to work. When I change CSS classes, instead of changing the background position of the sprite image, it stays in the same skewed position.

Next steps

From here, I think I need to either use CSS classes, or set style properties from JavaScript exclusively. Mixing and matching might be causing problems. What do you think?

However, as I finish this, my wife and my in-laws want to go to the pub. So screw this hacking on frustrating code from home. I’m going for a pint and some appies!

Before I go though, I’m going to appeal to the iPhone and PhoneGap developer community at large: Have any of you encountered these problems before? Do you have any suggestions on how to make things work better?

I’ve recently begun drinking the koolaid of Git, and damn it’s tasty! The things I can do with git that I couldn’t have done before (or would have been difficult to do) makes me excited about it. In fact, the one feature that I thought was a drawback — the no-one-true-server nature of it — is actually its strongest selling point.

See, the way I’ve taken to doing my development now is I create two “remote” repositories. First is “origin” which points to a repository managed by Gitosis. Second is a “live” repository that points at a working directory on my production server. That working directory is where my live site actually runs in.

On its own this is handy. As I develop new features, I push my changes to origin. Once my code is ready, and after I run “make test” to verify my site passes all its unit tests, I push to my live repo. At this point I ssh to the server, restart my server processes, and in theory all should be well.

The need for automation

I discovered quickly that in practice this was fraught with error. After a fairly large refactor, I found that the code that worked perfectly well on my development laptop fell over on production. Old or missing libraries, dependancy problems, ownership permissions on files, you name it. My site was down for 2 hours while I tried to resolve these issues. I added more unit tests to my code, but still this wouldn’t have caught these problems.

At this point I decided that a more automated approach was needed. I used a friend’s post-update hook as a template, which simply merged in changes and restarted nginx following a push to live. To this I added a long one-liner, and added the relevant commands to /etc/sudoers with the “nopasswd” option. In the end, the function looks like this:

bounce_fcgi() {
  (cd $GIT_WORK_TREE
  echo $PWD >&2
  [ -f Makefile ] && make clean >&2
  perl Makefile.PL >&2 && make test >&2 &&
    sudo /etc/init.d/nginx test 2&>/dev/null &&
    sudo /etc/init.d/nginx reload &&
    sudo /etc/init.d/webapp restart >&2
 )}

Essentially, after my new changes are merged in, it:

1. Creates and runs my Makefile
2. Runs all my unit tests
3. Tests my new nginx server configuration
4. Reloads my new nginx configuration
5. Restarts my FastCGI web application

If any of those steps fails, the full process halts. All the >&2 arguments ensure that the output of these commands are echoed to my local console from the remote server. So when I type “git push live”, all the test output is displayed to me inline. If an error occurs, I can immediately fix it in my local environment and push out a new change without having to log in to my remote server once.

My web application is written in Catalyst, and I use Test::WWW::Mechanize::Catalyst extensively, so I not only unit test my back-end classes, but I test the URLs users actually interact with. It even goes so far as creating and destroying test accounts within my database, so every feature of the website is tested, right down to validating the contents of my robots.txt and sitemap.xml files.

Next Steps

There are still some holes that need to be filled here.

• If running my tests fail for some reason, I would like to roll-back the remote working directory to the previous version and restart my services, that way the site continues to function under a known-good state.
• I would like to use WWW::Google::SiteMap::Ping to notify Google, and perhaps other search engines that support XML Sitemaps, that the contents of my site have changed and a reindexing is needed.
• My site is localized, so I would like to regenerate my PO translation files, and if any strings have changed or are out of date, automatically send an email to my translators with the new POT template file attached.
• Run my HTML through a spelling checker, to verify I don’t put any typos in any of my pages.
• Since I try to use caching as much as possible, when a web page’s content has changed, I would like to automatically connect to my Memcached servers to purge the relevant pages from its cache so new versions are immediately available.

A fringe benefit is makes it fun and exciting to write unit tests! Whenever I find an area that isn’t covered, I sit down and crank out more tests to validate new features. I never want to be caught with my pants down on a deployment. Each time it catches something I missed, it makes the whole thing worthwhile.

Does anyone have any thoughts on how I can improve this? Is anyone doing something similar that I can learn from? I’m very excited about my new deployment system, and wish I’d had this ages ago. If there’s anything you want to know more about, please leave a comment.

A few months ago I had the opportunity to meet with the crew developing PhoneGap down at their headquarters (Nitobi, down in Gastown district of Vancouver). It’s only a few blocks away from where I work, but the horrible downpour of rain made me think twice about going (hey, it is Vancouver after all). Next time I’ll bring my laptop with me, but it gave me an introduction to what they’re looking for help with, and where they’re taking PhoneGap.

Due to work and home commitments however, I haven’t been able to develop any applications for it until recently. I was stuck at home for almost a week with a terrible flu, and that gave me the opportunity to tinker, and helped me to avoid daytime television.

First Attempts

At first I tried to develop an app much in the same way I develop web applications on other browsers. I built a MooTools package including the dependancies I needed, stuck it in a directory, and began writing classes to implement the UI logic I needed. After compiling and getting it all onto the iPhone simulator, I screeched to a halt.

None of my logic worked. The static HTML was just that…static. So, I trimmed code, recompiled, rinse, repeat. Try again with jQuery, no joy. Try with XUI, and still things were behaving oddly.

Essentially I discovered that I was just trying to do too much. I pulled everything back, simplified my code, and did away with the external libraries. Importing MooTools and jQuery weren’t the problems exactly, though I figured it wasn’t helping. The thing I discovered is that WebKit, CSS3 and Safari’s JavaScript engine are sophisticated enough that there isn’t much need for complicated JavaScript toolkits.

Second, more successful attempts

After taking a step back, I went back to basics. I built my class using basic prototype class construction (though I miss “new Class({})” from MooTools). Instead of using JavaScript-based animations, I read up and implemented CSS3 transitions. I did my HTML layout debugging in Firefox with Firebug, proofed them in Safari on my desktop, and only then did I compile and test in the iPhone simulator.

My biggest problem was the lack of a debugger or JavaScript console on the iPhone simulator. My only way of accurately testing where my code was falling down was by issuing “alert()” statements, which of course breaks the flow of code.

I now have my app running smoothly, and as soon as my business name and licenses go through, I’m going to register as an Apple iPhone developer so I can test my app on my iPhone. With luck, I’ll be pushing my application through the AppStore release procedure, which I’m sure will be another learning curve as well.

For now, I’m happy with what I have. I’ve created a simple application that works client/server via XHR with fall-back to disconnected mode when there’s no network connection, the animations and transitions are smooth, and the JavaScript is minimal with no external libraries other than gap.js needed.

I’ll post again once I release, so everyone can see what it is I’ve been working with. I have great plans for future applications, and I’m excited about PhoneGap.

PhoneGap Impressions

Before I go though, I want to make a few points about my observations of PhoneGap. I might be misinformed on a few points, but the lack of documentation and its relative alpha/beta state may just mean that these features are coming.

• No JavaScript console available
• XCode can’t figure out when HTML/JavaScript dependancies change, so you have to do a “Clean All” before recompiling and testing your changes
• XUI really isn’t ready for broad use, in my opinion. It seems like a great idea, and I would like to contribute to it, but it’s easier right now for me to write plain JavaScript.
• Use CSS3 whenever possible; avoid issuing animations in JavaScript.
• If you do have to do more complicated animations, use JavaScript only to piece together different sets of CSS3 animations by setting different CSS classes at intervals.

Fortunately, the PhoneGap google group is active, and quick to respond. I’m looking forward to becoming more involved as I build more with PhoneGap.

On the train ride to work today I was listening to “Daft Punk” on my iPhone while reading my blogs, my usual morning activity. And as my mind wandered, the song “Harder, Better, Faster, Stronger” came on; it’s one of my favourites, as I’m an avid electronic music fan. But since I was reading Ajaxian and several other software industry business blogs at the time, the song seemed strangely appropriate. Everyone is trying to find ways of working harder, developing better software, releasing to market faster, and having a much stronger market position (honestly, this is what was running through my head). This made me think more about what I do to fulfill those 4 business goals, so well voiced by Daft Punk.

“Work it harder”

Personally, I like having a life. I have a wife, hobbies, Open Source projects I like to contribute to. And of course I have my day job where I do more than I’m asked to do, always trying to come up with creative side projects to help make more sales, improve the product, or visualize our usage data in a way to help us build a better product.

So given the cliché that we all have 24 hours in a day, and you have sleep, meals, and commute time involved, how do you optimize your daily work habits to maximize that time?

The way we do this at work is through Extreme Programming or Agile Software Development, both are complementary techniques to eliminate waste from the traditional development cycle. Planning documents, requirements analysis, task assignment and many other wasteful planning processes are typically eliminated.

Coming up with a comprehensive specification before the project is even started is dreadfully wasteful. They typically become a hindrance because no developer can possibly foresee what problems they’ll encounter part-way through. These development practices allow a team as a whole to be flexible, to adapt to changes mid-way through the development cycle, and develop the important bits first. If you haven’t heard about or tried Extreme Programming, I highly recommend you go read up on it. Extreme Programming Explained is a great introduction for people used to more conventional development practices.

The important thing I’ve found is that these practices can even be adapted to single-person teams. You have to alter a few things in order to get it to work for a solo team, but adhering to the principles of “Build the simplest thing that works end-to-end first, then refactor later” is perhaps the most important practice I can suggest. For myself, it’s hard to get lost in developing “Teh Bestest Softwares Evar!” Sometimes you have to take a step back, and question yourself as to what is truly important to your project? Go build that, and don’t waste your time with unnecessary frills.

“Make it better”

Cutting out what isn’t necessary, or building the simplest possible thing doesn’t mean eliminating best practices. Quite the contrary, Extreme Programming pushes very heavily on test-driven development. The ultimate goal is to write your tests before you write a single piece of code. It doesn’t have to be a difficult test, but when you first run it, the test should fail. Once you get a failing test, you write the corresponding code for your project to make the test pass. Then you move on to the next task, rinse and repeat. One of the books on our bookshelf here at work is Test Driven Development, and it’s a great book on learning how using tests to drive your development can really help. I actually need to re-read it again, since my test-writing skills are getting a bit rusty.

By stepping through your project one bit at a time, writing tests as you go, you ensure that any regression in your code is caught by your test coverage. If you’re developing applications that a user will interact with, either through a GUI or a web interface, then automated tests are needed to be able to programatically exercise your interface.

We use Selenium within a series of Windows VMWare instances. Our test runner machine spawns VM instances on our VMWare server, as well as installs fresh copies of our software on our test servers. It then runs our tests using a test harness I developed called Test::A8N to be able to execute our tests, remote-controlling the browsers from within Windows.

Along with all these tests, you need visibility into how your tests are performing. There’s no point in writing all those tests, if you never run them. So we have a nightly process that runs all of our tests in parallel across several servers (so that they can complete in a single night), and then renders an HTML “Dashboard” showing which tests passed and failed, summed up as Red, Yellow or Green. A wide-screen LCD display is easily visible by the entire team, and shows us at a glance just how we’re doing. If you see large patches of red, then you know you’ve got some bug fixing to get to today.

There are other philosophies behind both Extreme and Agile programming, but I won’t get into them here. I highly suggest you read one of these books, as they’re really eye-opening into how you can develop better software without the traditional pain and bureaucracy found in more traditional methods.

“Do it faster”

With all this attention to detail and automated tests, it’s important that your development is still faster than the traditional ways of writing code. Sometimes you have to throw out what isn’t working, and this includes your preciously-written code. If something that served your purposes a year, 6 months, or even 2 weeks ago isn’t serving you anymore, then it’s time for it to go!

Refactoring is the process of taking something that worked before, and altering or culling what doesn’t fit anymore, and writing fresh code to replace it. Many times this means throwing entire chunks of code away. It’s important to not be married to your software. It’s just work, you’ve got plenty more where that came from. Just write what is needed, and nothing more. Don’t over-engineer, since that future “Maybe” feature you’re coding for most likely won’t be needed or used anyway.

Case in point: here at work, a former employee had written a UI framework that all our UI pages were built upon. It handled navigation, Ajax code execution, and so forth. It served its purposes very well at its time, but as features were added it became more and more brittle. Finally, shortly before I was hired, they added localization support, and it broke the camel’s back. Developing new pages was tedious, time consuming, and was littered with unused features that no one knew how or even if they worked.

After a short time of dealing with this, I cut it all out and replaced it with something smaller, lighter, and custom-fit to what it was we needed to do. It cleaned up our URIs so they were easy to understand, sped up development significantly, and gave us a stable platform to continue our subsequent refactoring on top of. All of this, and we eliminated approximately 3,000 lines of code. We’ve settled on the MooTools JavaScript framework for our code, and extend it liberally to build our client-side Ajax UI.

The framework is still holding up a year later, though I know it will have to go through some growing pains later. It’s currently Apache2/mod_perl based, and we’ll eventually be ripping this out and replacing it with a lighter FastCGI wrapper. But the important thing to take from this is that we weren’t so tied to our code that we weren’t afraid to throw it away when it started to smell.

“Makes us stronger”

The end result of all of this is to improve everyone’s lives. The developers have more fun at work, the project gets completed faster, more value is delivered to the customer in a timely fashion, the shareholders make more money, and the business improves.

Therefore I like to find areas in which I can help the company. Partly because there are many more interesting problems than just what I work with in my day job. So I write tools to help the salespeople sell more software. I write tools to help the product managers identify what our customers need and use. I’m currently working with our documentation team to find new ways to represent our documentation to make it more visible to our customers when they need help.

Ultimately, if you think outside the box and thing about your project from your customer’s perspective, it opens up a much bigger world for you to find your niche. Forget about what your job title is or what you’re being tasked to do. What problems do your customers have? If you had to use the software you’re developing on a day-to-day basis, would you enjoy yourself? Or would you pick up the phone and try to find a new vendor?

There’s a great rant over on Seth Godin’s blog titled “Just doing my job.” Don’t hide behind your job title as an excuse to justify your company’s actions, and don’t use it as an excuse to be lazy. Expand your skills, help your customers solve real problems, and don’t write bad software.

“More then ever after, our work is never over”

Most importantly than anything, never blindly accept that what you’re doing now is the best there can be. Newer technologies are developed, better strategies for developing or planning software can be invented, and people become complacent. Always read, always learn. Attend conferences, teach others what you know so you can understand your craft better.

So it’s very fitting that the last phrase in Daft Punk’s “Harder, Better, Faster, Stronger” is this: “Our work is never over”.