February 24, 2009

The Phoenix Checklist



My friend Rob Lambert enjoyed my post "Ask Great Questions!"  And in an email, he mentioned a blog entry he wrote last year - http://blogs.imeta.co.uk/RLambert/archive/2008/10/17/is-software-testing-like-being-in-the-cia.aspx .  Very nice!

The blog post refers to The Phoenix Checklist, which seems to me like a very useful framework for questioning in general.  I'm going to reread Rob's blog and this list periodically, and hopefully learn how to ask better questions.

From "Thinkertoys" by Michael Michako:

Phoenix is a checklist of questions developed by the Central Intelligence Agency to encourage agents to look at a challenge from several different angles.

The Phoenix Checklist

The Problem
  • Why is it necessary to solve the problem?
  • What benefits will you gain by solving the problem?
  • What is the unknown?
  • What is it you don't yet understand?
  • What is the information you have?
  • What isn't the problem?
  • Is the information sufficient?  Or is it insufficient?  Or redundant?  Or contradictory?
  • Should you draw a diagram of the problem?  A figure?
  • Where are the boundaries of the problem?
  • Can you separate the various parts of the problem?  Can you write them down? 
  • What are the relationships of the parts of the problem?
  • What are the constants (things that can't be changed) of the problem?
  • Have you seen this problem before?
  • Have you seen this problem in a slightly different form?
  • Do you know a related problem?
  • Can you think of a familiar problem having the same or a similar unknown?
  • Suppose you find a problem related to yours that has already been solved.  Can you use it?  Can you use its method?
  • Can you restate your problem?  How many different ways can you restate it?  More general?  More specific?  Can the rules be changed?
  • What are the best, worst, and most probable cases you can imagine?
The Plan
  • Can you solve the whole problem?  Part of the problem?
  • What would you like the resolution to be?  Can you picture it?
  • How much of the unknown can you determine?
  • Can you derive something useful from the information you have?
  • Have you used all the information?
  • Have you taken into account all essential notions in the problem?
  • Can you separate the steps in the problem-solving process? Can you determine the correctness of each step?
  • What creative thinking techniques can you use to generate ideas?  How many different techniques?
  • Can you see the result?  How many different kinds of results can you see?
  • How many different ways have you tried to solve the problem?
  • What have others done?
  • Can you intuit the solution?  Can you check the result?
  • What should be done?  How should it be done?
  • Where should it be done?
  • Who should do it?
  • What do you need to do at this time?
  • Who will be responsible for what?
  • Can you use this problem to solve some other problem?
  • What is the unique set of qualities that makes this problem what it is and none other?
  • What milestones can best mark your progress?
  • How will you know when you are successful?

February 22, 2009

Ask Great Questions!


One of our Product Managers recently responded to an email thread that was going around with this:

"As usual, Joe asks great questions!"

I was flattered.  It really pleases me when I hear that about anyone on my team, as I consider that a very high compliment for a tester.  One of the principal causes of software bugs is miscommunication, and asking great questions is one way (perhaps the best way) to prevent bugs before they make their way into code.

Ask All of the Stakeholders
  • Is this what we need?
  • What is the relative priority of this work versus all other work I may have?
Ask the Customers
  • What problems are you seeing?
  • What is most important to you?
Ask the Requirements or Product Manager
  • What does X mean?
  • What is missing here?
  • What should happen if I do Y?
  • Which of these features are "musts", and which are "nice to have"?
  • What is the expected volume of this system?
  • What are the performance needs?
  • In what environments (OS, Browser, database, etc) must this system execute?
  • Into which locales/languages will this system be released?
  • Are there any standards (corporate, UI, legal, etc) that are relevant?
  • What are the security requirements?
Ask the Schedule
  • What are the milestones?
  • When must this system ship?
  • Is there any room for slippage?
  • How accurate must my schedule estimates be?
  • When do I expect to receive each feature?
  • How quickly must I determine if a build is testable?
Ask the Documentation/User Manual/Getting Started Manual
  • What is claimed in writing about this system?
  • What examples are written?
  • What is suggested for users to try first?
Ask the Developers copyrightjoestrazzere
  • How extensive will your unit testing be?
  • Are you using any new technology?
  • What worries you about this system under test?
  • Are there any other tests you would like to see performed?
  • Are you getting what you need from me?
Ask Yourself
  • How detailed should my Test Plan be for this system?
  • How much of my testing should be written into Test Case?
  • How can I best take advantage of Test Automation here?
  • Where are the boundaries in this system?
  • How do I make this error message appear?
  • Is this a bug, or is my test invalid?
  • What's the simplest, most straightforward way to reproduce this bug?
  • Is this bug report clear enough for the intended audience?
  • What happens if I try this again, in a slightly different way?
  • What would a new user do here?  Whould an experienced user do anything differently?
  • In what order should I execute my tests, so as to give the most important feedback first?
  • What else should I test now?
  • What tests haven't I tried yet on this system?
  • What can I do today that would have the most significant positive impact on quality?
  • What do I need to learn, in order to be effective in testing this system?
  • What do I need in my test lab to be successful?
  • Do I have enough help?
  • Am I proceeding on a path that will complete on time? 
Ask Other Testers
  • Do you have any advice or tips on testing this system?
  • When you tested a prior version of this system (or one like it), how did it go?
Ask the Product Team
  • Who does what?
  • How often will we meet?
  • What do we need to do to communicate effectively during the project?
Ask Your Boss
  • How am I doing?
  • Am I spending my time wisely?

February 21, 2009

WinTask - Test Data Generator



This is a handy collection of WinTask test data generation functions



'
' Test Data Generator
'
' Author: Joe Strazzere

Dim FirstM$(1000)
Dim FirstF$(1000)
Dim FirstA$(1000)
Dim LastA$(1000)
Dim States$(1000)
Dim StateAs$(1000)
Dim Provinces$(1000)
Dim Cities$(1000)
Dim Lipsum$(1000)

' Returns a random string based on the picture supplied.
'
' This can be useful for generating large amounts of psuedo-random data for testing.
'
' picture is a string with special characters interpreted as follows:
'
' 9 = generate a random number (0-9) in its place
' A = generate a random alphabetic character (A-Z, a-z) in its place
' U = generate a random uppercase alphabetic character (A-Z) in its place
' L = generate a random lowercase alphabetic character (a-z) in its place
' N = generate a random alphanumeric character (A-Z, a-z, 0-9) in its place
' X = generate any character in its place
' = an escape character.  Anything following this character is placed in the result string,
'     even if it is a picture character.  Two backslashes () cause a single backslash to be
'     placed into the result string.
' any other character is simply copied into the result string
Function RandStr$(test$)
    Local i
    Local a
    Local new$
    Local work$
    work$=""
    i=1
    While i<= len(test$)
        Select Case Mid$(test$,i,1)
            Case ""
                 i=i+1
                 new$=Mid$(test$,i,1)
            Case "9"
                 new$=str$(Random(9))
            Case "X"
                 new$=Chr$(Random(93)+33)
            Case "U"
                 new$=Chr$(Random(25)+65)
            Case "L"
                 new$=Chr$(Random(25)+97)
            Case "A"
                 Repeat
                     a = Random(57)+65
                 Until a < 91 Or a >96
                 new$=Chr$(a)
            Case "N"
                 Repeat
                     a = Random(74)+48
                 Until a < 58 Or (a >64 And a <91) Or a > 96
                 new$=Chr$(a)
            Case Else
                 new$=Mid$(test$,i,1)
        EndSelect
        work$=work$+new$
        i=i+1
    Wend
    RandStr$=work$
EndFunction

'Zip Code
Function ZipCode$()
    ZipCode$=RandStr$("99999")
EndFunction

'Zip+4 Code
Function ZipPlus4$()
    ZipPlus4$=RandStr$("99999-9999")
EndFunction

'Canadian Postal Code
Function CAPostalCode$()
    CAPostalCode$=RandStr$("U9U 9U9")
EndFunction

'Social Security Number
Function SocSec$()
    SocSec$=RandStr$("999-99-9999")
EndFunction

'Phone Number
Function Phone$()
    Phone$=RandStr$("(999) 999-9999")
EndFunction

'Random Date - between two dates
Function RandomDate$(FromDate$,ToDate$)
    Local days
    Local rand
    days=DateBetween("d",FromDate$,ToDate$)
    rand=Random(Days)
    RandomDate$=DateToDate$("d",rand,FromDate$)
EndFunction

'Male First Name
Function MaleFirst$()
    MaleFirst$=FirstM$(Random(FirstMCount))
EndFunction

'Female First Name
Function FemaleFirst$()
    FemaleFirst$=FirstF$(Random(FirstFCount))
EndFunction

'First Name (Male or Female)
Function FirstName$()
    FirstName$=FirstA$(Random(FirstACount))
EndFunction

'Last Name
Function LastName$()
    LastName$=LastA$(Random(LastACount))
EndFunction

'First then Last name
Function FirstLastName$()
    FirstLastName$=FirstA$(Random(FirstACount))+" "+LastA$(Random(LastACount))
EndFunction

'First, Middle Initial, Last name
Function FirstMLastName$()
    FirstMLastName$=FirstA$(Random(FirstACount))+" "+RandStr$("U")+". "+LastA$(Random(LastACount))
EndFunction

'Last, comma, First name
Function LastFirstName$()
    LastFirstName$=LastA$(Random(LastACount))+", "+FirstA$(Random(FirstACount))
EndFunction

'State Name
Function StateName$()
    StateName$=States$(Random(StateCount))
EndFunction

'State Abbreviation
Function StateAbbr$()
    StateAbbr$=StateAs$(Random(StateACount))
EndFunction

'Canadian Province Name
Function ProvinceName$()
    ProvinceName$=Provinces$(Random(ProvinceCount))
EndFunction

'City Name
Function CityName$()
    CityName$=Cities$(Random(CityCount))
EndFunction

'Lorem Ipsum Word
Function LipsumWord$()
    LipsumWord$=Lipsum$(Random(LipsumCount))
EndFunction

'Lorem Ipsum Sentence - average 14 words in a sentence
Function LipsumSentence$(MaxLen)
    Local sentencelen
    Local sentence$
    Local separator$
    Local i

    sentencelen=Random(MaxLen)+1
    sentence$=""
    i=0

    While i < sentencelen
        If Random(50) < 1 Then
            separator$="; "
        Else
            If Random(10) < 1 Then
                separator$=", "
            Else
                separator$=" "
            EndIf
        EndIf
        sentence$=sentence$+Lipsum$(Random(LipsumCount))+separator$
        i=i+1
    Wend
    ' Capitalize the first letter
    sentence$=Ucase$(Left$(sentence$,1))+Right$(sentence$,(Len(sentence$)-1))
    ' Remove extra whitespace and add a period
    If (Right$(sentence$,2)="; " OR Right$(sentence$,2)=", ")Then
        LipsumSentence$=Left$(sentence$,Len(sentence$)-2)+"."
    Else
        LipsumSentence$=Left$(sentence$,Len(sentence$)-1)+"."   
    EndIf
EndFunction
' You can add or remove names as desired
Male$="Aaron,Barry,Carl,Daniel,Edward,Frank,George,Harold,Ike,Joseph,Kareem,Lawrence,Michael,Nicolas,Oliver,Patrick,Quinton,Richard,Salvatore,Thomas,Umesh,Vincent,William,Xerxes,Yves,Zeus" Female$="Abby,Barbara,Caitlin,Donna,Ellen,Faye,Gail,Hannah,Ilene,Jenna,Kate,Laura,Maggie,Nadine,Odella,Paige,Queenie,Rachael,Samantha,Tamara,Uma,Valerie,Wendy,Xera,Yvonne,Zoe"
Last$="Al-Hussein,Bacon,Carter,Dennis,Edwards,Fowler,Grant,Hadley,Irving,Josephson,Keith,Landon,Michaels,Ng,O'Brien,Paulson,Quigley,Richardson,St. Pierre,Thompson,Unger,Van Kempen,Williamson,Xeronson,Young,Zachary"
' You can add or remove territories as required
State$="Alabama,Alaska,Arkansas,Arizona,California,Colorado,Connecticut,Delaware,District of Columbia,Florida,Georgia,Hawaii,Idaho,Illinois,Indiana,Iowa,Kansas,Kentucky,Louisisana,Maine,Maryland,Massachusetts,Michigan,Minnesota,Mississippi,Missouri,Montana,Nebraska,Nevada,New Hampshire,New Jersey,New Mexico,New York,North Carolina,North Dakota,Ohio,Oklahoma,Oregon,Pennsylvania,Puerto Rico,Rhode Island,South Carolina,South Dakota,Tennessee,Texas,Utah,Vermont,Virginia,Washington,West Virginia,Wisconsin,Wyoming"
StateA$="AK,AL,AR,AZ,CA,CO,CT,DC,DE,FL,GA,HI,IA,ID,IL,IN,KS,KY,LA,MA,MD,ME,MI,MN,MO,MS,MT,NC,ND,NE,NH,NJ,NM,NV,NY,OH,OK,OR,PA,PR,RI,SC,SD,TN,TX,UT,VA,VT,WA,WI,WV,WY"
Province$="Alberta,British Columbia,Manitoba,New Brunswick,Newfoundland,Nova Scotia,Northwest Territories,Ontario,Prince Edward Island,Quebec,Saskatchewan,Yukon Territory"
City$="Albany,Boston,Carson City,Denver,East Point,Fairfax,Georgetown,Harts Landing,Indianapolis,Johnstown,Kansas City,Lower Mills,Middletown,North Hills,Oakland,Philadelphia,Quincy,Ridgeview,South Orange,Teaneck,Upsilanti,vienna,West Haven,Xavier,Youngstown,Zanzibar"
LoremIpsum$="lorem,ipsum,dolor,sit,amet,consectetur,adipisicing,elit,sed,do,eiusmod,tempor,incididunt,ut,labore,et,dolore,magna,aliqua,ut,enim,ad,minim,veniam,quis,nostrud,exercitation,ullamco,laboris,nisi,ut,aliquip,ex,ea,commodo,consequat,duis,aute,irure,dolor,in,reprehenderit,in,voluptate,velit,esse,cillum,dolore,eu,fugiat,nulla,pariatur,excepteur,sint,occaecat,cupidatat,non,proident,sunt,in,culpa,qui,officia,deserunt,mollit,anim,id,est,laborum"

FirstMCount=SplitIntoArray(Male$,FirstM$(),",") 'Count of Male first names
FirstFCount=SplitIntoArray(Female$,FirstF$(),",") ' Count of Female first names
FirstACount=SplitIntoArray(Male$+","+Female$,FirstA$(),",")' Count of All first names
LastACount=SplitIntoArray(Last$,LastA$(),",") ' Count of All last names
StateCount=SplitIntoArray(State$,States$(),",") ' Count of All States
StateACount=SplitIntoArray(StateA$,StateAs$(),",") ' Count of All State Abbreviations
ProvinceCount=SplitIntoArray(Province$,Provinces$(),",") ' Count of All Canadian Provinces
CityCount=SplitIntoArray(City$,Cities$(),",") ' Count of All Cities
LipsumCount=SplitIntoArray(LoremIpsum$,Lipsum$(),",") ' Count of All Lorem Ipsum words

'MsgBox(ZipCode$())
'MsgBox(ZipPlus4$())
'MsgBox(CAPostalCode$())
'MsgBox(SocSec$())
'MsgBox(Phone$())
'MsgBox(RandomDate$("01/01/2000", Date$()))
'MsgBox(MaleFirst$())
'MsgBox(FemaleFirst$())
'MsgBox(FirstName$())
'MsgBox(FirstLastName$())
'MsgBox(FirstMLastName$())
'MsgBox(LastFirstName$())
'MsgBox(StateName$())
'MsgBox(StateAbbr$())
'MsgBox(ProvinceName$())
'MsgBox(CityName$())
'MsgBox(LipsumWord$())
'MsgBox(LipsumSentence$(20))

February 16, 2009

A Winter Walk Around Lake Quannapowitt in Wakefield

 (Lake  Quannapowitt, Wakefield MA)

My wife and I like to get out of the house and walk as often as we can.  In bad weather, the elliptical machine helps provide the exercise we need, but it's clearly not the same.

While it's still very much winter in New England, yesterday was a bright, sunny 39 degrees with no wind, so we decided to take a nice walk.  We drove to nearby Wakefield, and took a brisk stroll around Lake Quannapowitt.

In warmer weather, the 3.25-mile walkway around Lake Quannapowitt is filled with walkers, joggers, dogs, and baby carriages.  But yesterday, it was fairly empty.

A few folks like us were bundled up and walking or jogging.  A few had dogs.  We did see two ice boats, and even one ice surfer.  The walkway was dry, as long as we stayed on the paved sidewalk.  We did venture onto one of the unpaved paths, and slipped around a bit, then quickly got back on the sidewalk.

It was very nice, good exercise, and makes me very eager for real Spring days, so we can do more. It's almost March, isn't it?

WinTask - The Hello World Test

In Chapter 8 of the terrific book "How We Test Software At Microsoft" by Alan Page, Ken Johnston and Bj Rollison, the authors talk about Model-Based Testing.
As part of this discussion, Alan Page introduces a simple application for modeling.  This program is reproduced here as a WinTask script.

If you compile this script to a .rob file, and give the new or prospective tester a copy of WinTask and the .rob file, he can try testing the program for himself, using test cases he has devised.

You may even choose to intentionally create a buggy version of this script, to see if the new tester can find all of the bugs.




'
' Hello World Sample Application - Model Based Testing
'
' Author: Joe Strazzere
'
' Adapted from "How We Test Software at Microsoft"
' - Alan Page, Ken Johnston, Bj Rollison
' Chapter 8
'
BEGINDIALOG Dialog 345, 355, 260, 260
CAPTION "Hello World Application"
    PUSHBUTTON "Say Hello", Button1, 20, 20, 90, 30
    EDITTEXT Edit1$, 140, 20, 90, 30
    PUSHBUTTON "Say World", Button2, 20, 70, 90, 30
    EDITTEXT Edit2$, 140, 70, 90, 30
    PUSHBUTTON "Clear Text", Button3, 20, 120, 90, 30
    DEFPUSHBUTTON "&Cancel", Button4, 93, 170, 69, 31
ENDDIALOG

'
'Instructions
'
Instr$=""
Instr$=Instr$+"Hello World is a simple application with four buttons."+CRLF
Instr$=Instr$+""+CRLF
Instr$=Instr$+"The first button prints 'Hello' into the first text box."+CRLF
Instr$=Instr$+"The second button prints 'World' into the second text box."+CRLF
Instr$=Instr$+"The third button clears the text boxes."+CRLF
Instr$=Instr$+""+CRLF
Instr$=Instr$+"The Cancel button stops the application."

MsgBox(Instr$,64,"Hello World")

CallDialog Dialog,"Hello World Application"

While Button4 <> 1
    If Button1 = 1 Then
        Edit1$="Hello"
    EndIf
    If Button2 = 1 Then
        Edit2$="World"
    EndIf
    If Button3 = 1 Then
        Edit1$=""
        Edit2$=""
    EndIf
    CallDialog Dialog,"Hello World Application"
Wend



also see: http://blogs.msdn.com/microsoft_press/archive/2009/02/13/new-book-how-we-test-software-at-microsoft.aspx

February 14, 2009

WinTask - WeekDayDate (undocumented function)

'
' WeekDayDate - Returns the day of the week as a number
' using the date given in the argument as the basis.
' (undocumented)
'
' Author: Joe Strazzere
'
' see: WeekDay
'
' Note: Don't use Date$() as the argument to WeekDayDate()
' It can cause WinTask to crash
'

'
' example - display the day number
' of the date that was two days ago
'
var$=DateToDate$("d",-2,Date$())
var=WeekDayDate(var$)
MsgBox(var)

February 12, 2009

WinTask - SelectHTMLItemByValue (undocumented function)

'
' SelectHTMLItemByValue - selects an item in a Combobox/Listbox within a Web page.
'      This undocumented function is like SelectHTMLItem(), except
'      that it selects by value rather than item text.
'
' Author: Joe Strazzere
'
'
' ret=SelectHTMLItemByValue(<html_descriptor>,item$[,shift|ctrl])
'

' go to http://www.w3schools.com/TAGS/tryit.asp?filename=tryhtml_option
'
' Paste in the following, and click "Edit and Click Me >>":
'  &lt;html&gt;
'  &lt;body&gt;
'  &lt;select name="MySelect" size=4 multiple="multiple"&gt;
'    &lt;option value="a"&gt;Volvo&lt;/option&gt;
'    &lt;option value="b"&gt;Saab&lt;/option&gt;
'    &lt;option value="c"&gt;Mercedes&lt;/option&gt;
'    &lt;option value="d"&gt;Audi&lt;/option&gt;
'  &lt;/select&gt;
'  &lt;/body&gt;
'  &lt;/html&gt;

'
' Then run this script
'

UsePage("Tryit Editor v1.4")
    SelectHTMLItemByValue("SELECT[NAME= 'MySelect']", "a")
    SelectHTMLItemByValue("SELECT[NAME= 'MySelect']", "b", shift)
    SelectHTMLItemByValue("SELECT[NAME= 'MySelect']", "d", ctrl)

February 7, 2009

Dear FIRST_NAME LAST_NAME - Information? Weak!

Here's an email I received, apparently as part of a marketing campaign managed by folks who don't really care.


I'm tempted to write back.
Dear %%LAME_MARKETING_MANAGER%%,

While I always enjoy a good cup of %%BEVERAGE_BRAND%%, your email campaign has left me somwhat less than enthusiastic. 

Are you sure %%FEATURED_SPEAKER_ONE%% and %%FEATURED_SPEAKER_TWO%% will actually show up?  After all, they may be using a better %%MAIL_MERGE_PROGRAM%% than you (it could hardly be worse), and may have a different %%EVENT_DATE%% in mind.

Anyway, thanks for the %%GIFT_CARD_TYPE%% offer, and good luck to the first %%WINNER_COUNT%% people.  I hope they actually get some beneift.

Sincerely,
%%FIRST_NAME%% %%LAST_NAME%%

February 1, 2009

Book: Perfect Software and other illusions about testing

Another addition to my QA Bookshelf.



Perfect Software is a high-level look at some of the "big picture questions" about testing, such as: Why do we have to bother testing?  Do we have to test everything?  What makes testing so hard?  Is perfect software possible?  Why can't we just accept a few bugs?

It's a small book (under 200 pages), and a quick read.  From the preface:
"I envision this book in the hands of professional testers, developers, consumers, analysts, designers, programmers, all of their managers, and all of their coworkers.

Most professional testers will know most of what's in this book, but I hope that by reading on, they will see new ways to communicate what they know - to their managers, developers, coworkers, and customers.

I'd like to help both developers and testers understand what their managers face when they confront software testing issues."
Contents:
  1. Why Do We Bother Testing?
  2. What Testing Cannot Do
  3. Why Not Just Test Everything?
  4. What's the Difference Between Testing and Debugging?
  5. Meta-Testing
  6. Information Immunity
  7. How to Deal With Defensive Reactions
  8. What Makes a Good Test?
  9. Major Fallacies About Testing
  10. Testing Is More Than Banging Keys
  11. Information Intake
  12. Making Meaning
  13. Determining Significance
  14. Making a Response
  15. Preventing Software Testing from Growing More Difficult
  16. Testing Without Machinery
  17. Testing Scams
  18. Oblivious Scams
Here's an interesting example that will give you a flavor of the book. 

In chapter 3 "Why Not Just Test Everything?", Weinberg has a section called "There are an infinite number of possible tests."  He talks about a backdoor placed into a highly secure program whereby the ordinary password protection could be bypassed by typing W followed by three spaces, then M followed by three spaces, then J followed by exactly 168 more keystrokes without once using the letter L.  Then he writes:
"Do you get the point by now?  If you didn't guess that the number of tests required to exhaustively test software is infinite, or at least "a number greater than I could run in my lifetime", you didn't understand the point of this chapter.  Now you do."
If you are looking for a "how to" book, you should look elsewhere.  If you are looking for a "why" (and sometimes "why not") book, this might be for you.