October 11, 2009

Book: Exploratory Software Testing




A few years back, I read How to Break Software by James Whittaker.  I liked it.  It wasn't wonderful, but it had a good batch of practical, useful tips.  Then I read How to Break Software Security and How to Break Web Software.  I liked them as well, but not as much.  Still, I figured I'd read James Whittaker's newest book Exploratory Software Testing.  Sadly, the downward progression of his writing continues.  This book is by far the worst of the bunch.

Chapter 1 - The Case for Software Quality is nothing more than "software is terrific, but it has bugs".   That's it, nothing more here.

Chapter 2 - The Case for Manual Testing talks a bit about testing, and tries to define exploratory testing.  Whittaker's definition has apparently caused some controversy among some well-known practitioners of exploratory testing, so here is his perhaps unique definition:
When the scripts are removed entirely (or as we shall see in later chapters, their rigidness relaxed), the process is called exploratory testing.
Whittaker then divides exploratory testing into two sections.  Exploratory testing in the small is that which guides the tester to make small, distinct decisions while testing.  Exploratory testing in the large guides the tester in how an application is explored more than how a specific feature is tested.

Chapter 3 - Exploratory Testing in the Small was, to me, the only useful chapter in the whole book.  Here Whittaker offers practical advice with examples for thinking about constructing test data, software state, and test environment.

Chapter 4 - Exploratory Testing in the Large is where Whittaker dives into what appears to be the point of the whole book - his Tourist Metaphor.  Apparently this is a big hit at Microsoft, but I found it pointless.  Think about every type of testing you have ever performed.  Now try to torture it into a phrase that ends with the word Tour.  There you go - that's the chapter. 

Just to give you a flavor, here's a list of all these Tours, and their variations:
  • The Guidebook Tour
    • Blogger's Tour
    • Pundit's Tour
    • Competitor's Tour
  • The Money Tour
    • Skeptical Customer Tour
  • The Landmark Tour
  • The Intellectual Tour
    • Arrogant American Tour
  • The FedEx Tour
  • The After-Hours Tour
    • Morning-Commute Tour
  • The Garbage Collector's Tour
  • The Bad-Neighborhood Tour
  • The Museum Tour
  • The Prior Version Tour
  • The Supporting Actor Tour
  • The Back Alley Tour
    • Mixed-Destination Tour
  • The All-Nighter Tour
  • The Collector's Tour
  • The Lonely Businessman's Tour
  • The Supermodel Tour
  • The TOGOF Tour
  • The Scottish Pub Tour
  • The Rained-Out Tour
  • The Couch Potato Tour
  • The Saboteur Tour
  • The Antisocial Tour
    • Opposite Tour
    • Crime Spree Tour
    • Wrong Turn Tour
  • The Obsessive-Compulsive Tour
Perhaps the idea of calling UI Testing a Supermodel Tour appeals to you, and will make for a richer, more productive set of tests.  I don't get it.  I just don't see any value here.  Doesn't testing have enough variation in language and definitions already, without adding this silliness?

Chapter 5 - Hybrid Exploratory Testing Techniques tells us that it's acceptable to combine scenario testing with exploratory testing.  Then it spends time rehashing each of the tours from Chapter 4 and tries to suggest a side trip for each. 

Chapter 6 - Exploratory Testing in Practice presents essays written by several Microsoft testers describing how they each used one or more of the tours in a testing situation.  It appears as if Whittaker instructed his charges to write a "What I did this summer"-style  essay, in the form of "How I used Tours to do my testing". 

Chapter 7 - Touring and Testing's Primary Pain Points tries to tell us (in a few paragraphs) how to avoid five pain points - Aimlessness, Repetiveness, Transiency, Monontony, and Memorylessness.  There's little real instruction here.  For example, we are told that in order to avoid repetitiveness, we must know what testing has already occurred, and understand when to inject variation.  Uhm, ok.

Chapter 8 - The Future of Software Testing has nothing at all to do with the other chapters, or exploratory testing.  It's basically Whittaker's gee-whiz vision of what might be possible (some day) in the future.  Perhaps.  Whittaker has given this talk in several webinars - it's simply rehashed here.

Since these chapters take up only 136 pages, and obviously aren't enough to fill out a real book, three unrelated appendices are bolted on.  A few pages about Testing as a career, and a bunch of pages lifted directly from Whittaker's blogs fill out the book to over 200 pages.

If you really want to learn about Exploratory Testing, this is probably not the place.  Exploratory Software Testing is fluff - stretched and tortured out barely to book-length.  There's not much in the way of learning here.

And if Microsoft testers are really instructed to "Tell me what kind of testing you did today, and make sure it ends with the word Tour", then I feel very sorry for them.

Also see:
http://www.satisfice.com/blog/archives/362
http://adam.goucher.ca/?p=1225
http://blog.utest.com/exploratory-software-testing-a-new-book-from-james-whittaker/2009/09/
http://www.testingreflections.com/node/view/3190
And here's my Amazon review of this book, along with comments about my review:
http://www.amazon.com/review/RKQ4KRQM21Y9F/ref=cm_cr_rdp_perm
Labels: ,