I just found this morning, through ironically the same customer that was having difficulties yesterday (this must be karma), that a key feature of my software has been disabled in the build on my website for the last 2 weeks.  I had been seeing traffic higher than ever, double the number of confirmed downloads I had in January, and was wondering “Why am I seeing 20-30% of my usual sales?  Well, it turns out the reason was that I was performing an involuntary A-B test: A with the feature,  and B without, based on what day you had first downloaded the software.  A wins by a longshot.

Here’s what happened: I ship Bingo Card Creator with about a dozen preprogrammed word lists.  Since I’m skeptical of my customers’ ability to do the Open File -> Navigate To Folder -> Select File routine, I make this very easy through a Wizards menu.  Click on the Wizards Menu, click your subject, click the menu item which describes what grade level/skill you are working on.  I had known this was a crucial feature when I included it in 1.02 because it alleviates the “empty screen problem” (I essentially can’t sell to someone who hasn’t seen a card printed out, and if you have to type in 25 words before you can print a card then you’re much less likely to invest the time).  I hadn’t known it was quite so crucial though.

Here’s what happened.  The Wizard menu is autogenerated, and unlike the vast majority of my code the logic is pretty smart.  It spiders a particular subdirectory in the installation, doing a breadth-first search of the file system tree, and making every directory a submenu and every properly formatted file an entry in the submenu it appears in.  This lets me add new items to the Wizard menu without tweaking anything in the Java code.  If a directory is empty, it doesn’t get shown.  If no files are found at all, the Wizard menu doesn’t show, because nothing ticks people off like non-functional programs, right?  (Grumble grumble.)

Anyhow, this feature has worked since 1.02 and I test to make sure it works every time I make a build, because its just so critical and because without the feature I’d actually have to type in word lists to type printing and that is slow.

I test in Eclipse and it works fine.

I test after exporting my JAR and it works fine.

I test after wrapping the JAR in the EXE and it works fine.

I test after building my installer and installing and it works fine.

My customer downloads the installer and sees no Wizards menu.

Can you spot what edge case my testing missed?  Oh, its fun — I accidentally introduced a single extra character into my Inno Setup script.  Since my setup script is not in my Subversion I never even noticed that I had made the change.  This resulted in the setup program copying the SampleLists (where the wizards reside) folder into the application directory, not into the application/SampleLists directory.  On my machine, since I was installing without uninstalling (clobbers identically named files but DOES NOT DELETE files not present in the new installer), I still had the old C:\Program Files\Bingo Card Creator\SampleLists directory with the proper structure, and things worked peachy.  Then my old, loyal customers who were doing updates installed and everything worked peachy.  Then new people downloaded the installer and, boom, no Wizards for two weeks.

After work today I’ll release a .01 “upgrade” to bugfix Download.com and all the other places that cache installers.

Lessons learned:

  • Setup script goes into the Subversion repository so that at any time it is either known-good or marked as changed since the last known-good.
  • Uninstall before installing to do final testing.  If I had another machine lying around I’d even do a virtual machine or something so that I was guaranteed of a fresh test environment.
  • Keep the lines of communication open with customers.  They can save you from yourself.