One of the first books on testing that I ever read was "The Art of Software Testing" by Glenford J. Myers
The first paragraph of the book asks the reader to take a short self-test to help demonstrate the difficulty in developing adequate test cases for even a very simple program. The program is widely known as The Triangle Problem (from Gerald Weinberg) or The Triangle Test, and is reproduced here as a WinTask script.
If you compile this script to a .rob file, and give the new or prospective tester a copy of WinTask and the .rob file, she can try testing the program for herself, using test cases she has devised.
You may even choose to intentionally create a buggy version of this script, to see if the new tester can find all of the bugs.
This implementation of The Triangle Test asks the tester for the Expected Results, and keeps track of the correctness of each result.
'
' Triangle Test 2 - Demonstrates testing a simple application
'
' Author: Joe Strazzere
'
' Adapted from Gerald Weinberg and Glenford Myers
'
' This version asks for your expected results
' and indicates correctness in a MsgBox and the Log
Dim Answers$(1000)
Dim List1$(3)
'
' The function which actually calculates the triangle type
' It is simple to introduce errors into this code if desired
'
Function TriangleType$(side1$,side2$,side3$)
s1=Val(side1$)
s2=Val(side2$)
s3=Val(side3$)
If (Str$(s1) <> Ltrim$(Rtrim$(side1$))) or (Str$(s2) <> Ltrim$(Rtrim$(side2$))) or (Str$(s2) <> Ltrim$(Rtrim$(side2$))) Then
TriangleType$="INVALID"
Else
If (s1 <= 0) or (s2 <= 0) or (s3 <= 0) Then
TriangleType$="INVALID"
Else
If (s1 > s2) and (s1 > s3) and (s1 >= s2 + s3)Then
TriangleType$="INVALID"
Else
If (s2 > s1) and (s2 > s3) and (s2 >= s1 + s3)Then
TriangleType$="INVALID"
Else
If (s3 > s1) and (s3 > s2) and (s3 >= s1 + s2)Then
TriangleType$="INVALID"
Else
If (s1 = s2) and (s2 = s3)Then
TriangleType$="EQUILATERAL"
Else
If (s1 = s2) or (s2 = s3) or (s1 = s3) Then
TriangleType$="ISOSCELES"
Else
TriangleType$="SCALENE"
EndIf
EndIf
EndIf
EndIf
EndIf
EndIf
EndIf
EndFunction
BEGINDIALOG Dialog 347, 362, 550, 317
CAPTION "Triangle Test 2"
ICON "Question", 22, 27, 32, 32
TEXT "Enter the three sides and your expected results.", 125, 30
TEXT "Then click OK to analyze, or Cancel to quit.", 125, 50
EDITTEXT Edit1$, 137, 88, 70, 20
EDITTEXT Edit2$, 245, 89, 70, 20
EDITTEXT Edit3$, 346, 88, 70, 20
TEXT "Expected Results:", 137, 134
LISTBOX List1$(), 241, 129, 177, 75, SelList1$
TEXT Type1$, 137, 196
TEXT Type2$, 243, 196
DEFPUSHBUTTON "OK", btnOK, 355, 238, 75, 23
PUSHBUTTON "Cancel", btnCancel, 444, 238, 75, 23
ENDDIALOG
'
' Instructions
'
Instr$=""
Instr$=Instr$+"The program reads three integer values."+CRLF
Instr$=Instr$+CRLF
Instr$=Instr$+"The three values are interpreted as representing the sides of a triangle."+CRLF
Instr$=Instr$+CRLF
Instr$=Instr$+"The program displays a message that states whether the triangle is "+CRLF
Instr$=Instr$+"SCALENE, EQUILATERAL, ISOSCELES or INVALID"
MsgBox(Instr$,64,"Triangle Test 2")
'
' Initial Display
'
List1$(0)="INVALID"
List1$(1)="SCALENE"
List1$(2)="EQUILATERAL"
List1$(3)="ISOSCELES"
Type$=""
In$=""
CallDialog Dialog,"Triangle Test 2"
'
' Subsequent Display
'
Type1$="Type of Triangle: "
i=0
Correct = 0
Incorrect = 0
While btnOK = 1
i=i+1
If i > 1000 Then
Goto Answers
EndIf
In$="Input: ["+Edit1$+"] , ["+Edit2$+"] , ["+Edit3$+"]"
Type2$=TriangleType$(Edit1$,Edit2$,Edit3$)
If SelList1$ = Type2$ Then
Correct = Correct + 1
Answers$(i)="CORRECT - "+In$+" Expected: "+SelList1$+" Actual: "+Type2$
Else
Incorrect = Incorrect + 1
Answers$(i)="INCORRECT - "+In$+" Expected: "+SelList1$+" Actual: "+Type2$
Endif
CallDialog Dialog,"Triangle Test 2"
Wend
'
' Assessment of Answers
'
Answers:
Ans$=""
Answers$(0) = "Correct = "+Str$(Correct)+", Incorrect = "+Str$(Incorrect)+CRLF+"--------------------------------"
j=0
While Answers$(j) <> ""
Ans$=Ans$+Answers$(j)+CRLF
Comment(Answers$(j))
j=j+1
Wend
MsgBox(Ans$,64,"Triangle Test 2 - Your Answers")
Your gathering place for information and ideas about Quality Assurance, Testing, and other topics of interest.
January 25, 2009
WinTask - The Triangle Test
One of the first books on testing that I ever read was "The Art of Software Testing" by Glenford J. Myers
The first paragraph of the book asks the reader to take a short self-test to help demonstrate the difficulty in developing adequate test cases for even a very simple program. The program is widely known as The Triangle Problem (from Gerald Weinberg) or The Triangle Test, and is reproduced here as a WinTask script.
If you compile this script to a .rob file, and give the new or prospective tester a copy of WinTask and the .rob file, she can try testing the program for herself, using test cases she has devised.
You may even choose to intentionally create a buggy version of this script, to see if the new tester can find all of the bugs.
This is a basic implementation of The Triangle Test.
'
' Triangle Test - Demonstrates testing a simple application
'
' Author: Joe Strazzere
'
' Adapted from Gerald Weinberg and Glenford Myers
'
Dim Answers$(1000)
'
' The function which actually calculates the triangle type
' It is simple to introduce errors into this code if desired
'
Function TriangleType$(side1$,side2$,side3$)
s1=Val(side1$)
s2=Val(side2$)
s3=Val(side3$)
If (Str$(s1) <> Ltrim$(Rtrim$(side1$))) or (Str$(s2) <> Ltrim$(Rtrim$(side2$))) or (Str$(s2) <> Ltrim$(Rtrim$(side2$))) Then
TriangleType$="INVALID"
Else
If (s1 <= 0) or (s2 <= 0) or (s3 <= 0) Then
TriangleType$="INVALID"
Else
If (s1 > s2) and (s1 > s3) and (s1 >= s2 + s3)Then
TriangleType$="INVALID"
Else
If (s2 > s1) and (s2 > s3) and (s2 >= s1 + s3)Then
TriangleType$="INVALID"
Else
If (s3 > s1) and (s3 > s2) and (s3 >= s1 + s2)Then
TriangleType$="INVALID"
Else
If (s1 = s2) and (s2 = s3)Then
TriangleType$="EQUILATERAL"
Else
If (s1 = s2) or (s2 = s3) or (s1 = s3) Then
TriangleType$="ISOSCELES"
Else
TriangleType$="SCALENE"
EndIf
EndIf
EndIf
EndIf
EndIf
EndIf
EndIf
EndFunction
BEGINDIALOG Dialog 347, 362, 550, 317
CAPTION "Triangle Test"
ICON "Question", 22, 27, 32, 32
TEXT "Enter the three sides.", 125, 30
TEXT "Then click OK to analyze, or Cancel to quit.", 125, 50
EDITTEXT Edit1$, 137, 88, 70, 20
EDITTEXT Edit2$, 245, 89, 70, 20
EDITTEXT Edit3$, 346, 88, 70, 20
TEXT Type1$, 137, 196
TEXT Type2$, 243, 196
DEFPUSHBUTTON "OK", btnOK, 355, 238, 75, 23
PUSHBUTTON "Cancel", btnCancel, 444, 238, 75, 23
ENDDIALOG
'
' Instructions
'
Instr$=""
Instr$=Instr$+"The program reads three integer values."+CRLF
Instr$=Instr$+CRLF
Instr$=Instr$+"The three values are interpreted as representing the sides of a triangle."+CRLF
Instr$=Instr$+CRLF
Instr$=Instr$+"The program displays a message that states whether the triangle is "+CRLF
Instr$=Instr$+"SCALENE, EQUILATERAL, ISOSCELES or INVALID"
MsgBox(Instr$,64,"Triangle Test")
'
' Initial Display
'
Type$=""
In$=""
CallDialog Dialog,"Triangle Test"
'
' Subsequent Display
'
Type1$="Type of Triangle: "
i=0
Correct = 0
Incorrect = 0
While btnOK = 1
i=i+1
If i > 1000 Then
End
EndIf
In$="Input: ["+Edit1$+"] , ["+Edit2$+"] , ["+Edit3$+"]"
Type2$=TriangleType$(Edit1$,Edit2$,Edit3$)
CallDialog Dialog,"Triangle Test"
Wend
The first paragraph of the book asks the reader to take a short self-test to help demonstrate the difficulty in developing adequate test cases for even a very simple program. The program is widely known as The Triangle Problem (from Gerald Weinberg) or The Triangle Test, and is reproduced here as a WinTask script.
If you compile this script to a .rob file, and give the new or prospective tester a copy of WinTask and the .rob file, she can try testing the program for herself, using test cases she has devised.
You may even choose to intentionally create a buggy version of this script, to see if the new tester can find all of the bugs.
This is a basic implementation of The Triangle Test.
'
' Triangle Test - Demonstrates testing a simple application
'
' Author: Joe Strazzere
'
' Adapted from Gerald Weinberg and Glenford Myers
'
Dim Answers$(1000)
'
' The function which actually calculates the triangle type
' It is simple to introduce errors into this code if desired
'
Function TriangleType$(side1$,side2$,side3$)
s1=Val(side1$)
s2=Val(side2$)
s3=Val(side3$)
If (Str$(s1) <> Ltrim$(Rtrim$(side1$))) or (Str$(s2) <> Ltrim$(Rtrim$(side2$))) or (Str$(s2) <> Ltrim$(Rtrim$(side2$))) Then
TriangleType$="INVALID"
Else
If (s1 <= 0) or (s2 <= 0) or (s3 <= 0) Then
TriangleType$="INVALID"
Else
If (s1 > s2) and (s1 > s3) and (s1 >= s2 + s3)Then
TriangleType$="INVALID"
Else
If (s2 > s1) and (s2 > s3) and (s2 >= s1 + s3)Then
TriangleType$="INVALID"
Else
If (s3 > s1) and (s3 > s2) and (s3 >= s1 + s2)Then
TriangleType$="INVALID"
Else
If (s1 = s2) and (s2 = s3)Then
TriangleType$="EQUILATERAL"
Else
If (s1 = s2) or (s2 = s3) or (s1 = s3) Then
TriangleType$="ISOSCELES"
Else
TriangleType$="SCALENE"
EndIf
EndIf
EndIf
EndIf
EndIf
EndIf
EndIf
EndFunction
BEGINDIALOG Dialog 347, 362, 550, 317
CAPTION "Triangle Test"
ICON "Question", 22, 27, 32, 32
TEXT "Enter the three sides.", 125, 30
TEXT "Then click OK to analyze, or Cancel to quit.", 125, 50
EDITTEXT Edit1$, 137, 88, 70, 20
EDITTEXT Edit2$, 245, 89, 70, 20
EDITTEXT Edit3$, 346, 88, 70, 20
TEXT Type1$, 137, 196
TEXT Type2$, 243, 196
DEFPUSHBUTTON "OK", btnOK, 355, 238, 75, 23
PUSHBUTTON "Cancel", btnCancel, 444, 238, 75, 23
ENDDIALOG
'
' Instructions
'
Instr$=""
Instr$=Instr$+"The program reads three integer values."+CRLF
Instr$=Instr$+CRLF
Instr$=Instr$+"The three values are interpreted as representing the sides of a triangle."+CRLF
Instr$=Instr$+CRLF
Instr$=Instr$+"The program displays a message that states whether the triangle is "+CRLF
Instr$=Instr$+"SCALENE, EQUILATERAL, ISOSCELES or INVALID"
MsgBox(Instr$,64,"Triangle Test")
'
' Initial Display
'
Type$=""
In$=""
CallDialog Dialog,"Triangle Test"
'
' Subsequent Display
'
Type1$="Type of Triangle: "
i=0
Correct = 0
Incorrect = 0
While btnOK = 1
i=i+1
If i > 1000 Then
End
EndIf
In$="Input: ["+Edit1$+"] , ["+Edit2$+"] , ["+Edit3$+"]"
Type2$=TriangleType$(Edit1$,Edit2$,Edit3$)
CallDialog Dialog,"Triangle Test"
Wend
WinTask - The Next Date Test
In Chapter 5 of the terrific book "How We Test Software At Microsoft" by Alan Page, Ken Johnston and Bj Rollison, is one of the best discussions about Equivalence Class Partitioning and Boundary Value Analysis I have ever read.
As part of this discussion, Bj Rollison introduces an example application he calls The Next Date program. (Bj tells me that the concept for the Next Date problem is from Paul C. Jorgensen’s book, “Software Testing: A Craftsman’s Approach” which is another fine book on software testing.)
This program is reproduced here as a WinTask script.
If you compile this script to a .rob file, and give the new or prospective tester a copy of WinTask and the .rob file, he can try testing the program for himself, using test cases he has devised.
You may even choose to intentionally create a buggy version of this script, to see if the new tester can find all of the bugs.
'
' NextDate - Demonstrates testing a simple application
' using Equivalence Class Partitioning
' and Boundary Value Analysis
'
' Author: Joe Strazzere
'
' Adapted from "How We Test Software at Microsoft"
' - Alan Page, Ken Johnston, Bj Rollison
'
'
' Here is where the NextDate function is implemented
' It would be simple to introduce intentional errors here
'
Function NextDate$(mm$,dd$,yyyy$)
mm=Val(mm$)
dd=Val(dd$)
yyyy=Val(yyyy$)
If (Len(mm$) > 2) or (Len(dd$) > 2) or (Len(yyyy$) > 4) Then
NextDate$="Invalid Date: ["+mm$+"/"+dd$+"/"+yyyy$+"]"
Else
If (yyyy < 1582) or (yyyy > 3000) Then
NextDate$="Invalid Date: ["+mm$+"/"+dd$+"/"+yyyy$+"]"
Else
If (dd > 31) Then
NextDate$="Invalid Date: ["+mm$+"/"+dd$+"/"+yyyy$+"]"
Else
If (mm > 12) Then
NextDate$="Invalid Date: ["+mm$+"/"+dd$+"/"+yyyy$+"]"
Else
' Special Date range
' the dates 10/5/1582 through 10/14/1582 are invalid
' (see page 85 in HWTSAM)
If (mm = 10) and (dd >= 5) and (dd <= 14) and (yyyy = 1582) Then
NextDate$="Invalid Date: ["+mm$+"/"+dd$+"/"+yyyy$+"]"
Else
' Special Date range
' the day after 10/4/1582 is 10/15/1582
' (see page 85 in HWTSAM)
If (mm = 10) and (dd = 4) and (yyyy = 1582) Then
NextDate$="The next date is: 10/15/1582"
Else
NextDate$="The next date is: "+DateToDate$("d",1,mm$+"/"+dd$+"/"+yyyy$)
EndIf
EndIf
EndIf
EndIf
EndIf
EndIf
EndFunction
'
' Handle unexpected errors and DateToDate errors here
'
Sub error_proc()
Text2$="Invalid Date: ["+Edit1$+"/"+Edit2$+"/"+Edit3$+"]"
EndSub
OnAction Error
DoSub error_proc
EndAction
BEGINDIALOG Dialog 347, 362, 350, 220
CAPTION "Next Date"
TEXT "Enter a date, then click OK to analyze, or Cancel to quit.", 31, 18
TEXT "Month", 43,62
TEXT "Day", 133,62
TEXT "Year", 223,62
EDITTEXT Edit1$, 40, 82, 50, 20
TEXT "/", 105, 85
EDITTEXT Edit2$, 130, 82, 50, 20
TEXT "/", 195, 85
EDITTEXT Edit3$, 220, 82, 50, 20
TEXT Text1$, 40, 115
DEFPUSHBUTTON "OK", btnOK, 137, 157, 75, 23
PUSHBUTTON "Cancel", btnCancel, 217, 157, 75, 23
ENDDIALOG
'
' Instructions
'
Instr$=""
Instr$=Instr$+"The NextDate program takes three integer inputs that represent a specific"+CRLF
Instr$=Instr$+"month, day, and year in the Gregorian calendar and returns the next"+CRLF
Instr$=Instr$+"Gregorian calendar date in the month/day/year format."+CRLF
Instr$=Instr$+CRLF
Instr$=Instr$+"The algorithm to determine the next date is based on the Gregorian calendar,"+CRLF
Instr$=Instr$+"which is commonly used throughout the world today"+CRLF
Instr$=Instr$+CRLF
Instr$=Instr$+"The output is always in the m/d/yyyy format regardless of the user's locale"+CRLF
Instr$=Instr$+"setting (meaning the output is not culture sensitive)."+CRLF
Instr$=Instr$+CRLF
Instr$=Instr$+"The valid range for input is 1/1/1582 through 12/31/3000"
MsgBox(Instr$,64,"NextDate")
'
' Initial Display
'
Text1$=""
CallDialog Dialog,"Next Date"
'
' Subsequent Display
'
While btnOK = 1
Text2$=""
Text1$=NextDate$(Edit1$,Edit2$,Edit3$)
If Text2$ <> "" Then
Text1$=Text2$
EndIf
CallDialog Dialog,"Next Date"
Wend
As part of this discussion, Bj Rollison introduces an example application he calls The Next Date program. (Bj tells me that the concept for the Next Date problem is from Paul C. Jorgensen’s book, “Software Testing: A Craftsman’s Approach” which is another fine book on software testing.)
This program is reproduced here as a WinTask script.
If you compile this script to a .rob file, and give the new or prospective tester a copy of WinTask and the .rob file, he can try testing the program for himself, using test cases he has devised.
You may even choose to intentionally create a buggy version of this script, to see if the new tester can find all of the bugs.
'
' NextDate - Demonstrates testing a simple application
' using Equivalence Class Partitioning
' and Boundary Value Analysis
'
' Author: Joe Strazzere
'
' Adapted from "How We Test Software at Microsoft"
' - Alan Page, Ken Johnston, Bj Rollison
'
'
' Here is where the NextDate function is implemented
' It would be simple to introduce intentional errors here
'
Function NextDate$(mm$,dd$,yyyy$)
mm=Val(mm$)
dd=Val(dd$)
yyyy=Val(yyyy$)
If (Len(mm$) > 2) or (Len(dd$) > 2) or (Len(yyyy$) > 4) Then
NextDate$="Invalid Date: ["+mm$+"/"+dd$+"/"+yyyy$+"]"
Else
If (yyyy < 1582) or (yyyy > 3000) Then
NextDate$="Invalid Date: ["+mm$+"/"+dd$+"/"+yyyy$+"]"
Else
If (dd > 31) Then
NextDate$="Invalid Date: ["+mm$+"/"+dd$+"/"+yyyy$+"]"
Else
If (mm > 12) Then
NextDate$="Invalid Date: ["+mm$+"/"+dd$+"/"+yyyy$+"]"
Else
' Special Date range
' the dates 10/5/1582 through 10/14/1582 are invalid
' (see page 85 in HWTSAM)
If (mm = 10) and (dd >= 5) and (dd <= 14) and (yyyy = 1582) Then
NextDate$="Invalid Date: ["+mm$+"/"+dd$+"/"+yyyy$+"]"
Else
' Special Date range
' the day after 10/4/1582 is 10/15/1582
' (see page 85 in HWTSAM)
If (mm = 10) and (dd = 4) and (yyyy = 1582) Then
NextDate$="The next date is: 10/15/1582"
Else
NextDate$="The next date is: "+DateToDate$("d",1,mm$+"/"+dd$+"/"+yyyy$)
EndIf
EndIf
EndIf
EndIf
EndIf
EndIf
EndFunction
'
' Handle unexpected errors and DateToDate errors here
'
Sub error_proc()
Text2$="Invalid Date: ["+Edit1$+"/"+Edit2$+"/"+Edit3$+"]"
EndSub
OnAction Error
DoSub error_proc
EndAction
BEGINDIALOG Dialog 347, 362, 350, 220
CAPTION "Next Date"
TEXT "Enter a date, then click OK to analyze, or Cancel to quit.", 31, 18
TEXT "Month", 43,62
TEXT "Day", 133,62
TEXT "Year", 223,62
EDITTEXT Edit1$, 40, 82, 50, 20
TEXT "/", 105, 85
EDITTEXT Edit2$, 130, 82, 50, 20
TEXT "/", 195, 85
EDITTEXT Edit3$, 220, 82, 50, 20
TEXT Text1$, 40, 115
DEFPUSHBUTTON "OK", btnOK, 137, 157, 75, 23
PUSHBUTTON "Cancel", btnCancel, 217, 157, 75, 23
ENDDIALOG
'
' Instructions
'
Instr$=""
Instr$=Instr$+"The NextDate program takes three integer inputs that represent a specific"+CRLF
Instr$=Instr$+"month, day, and year in the Gregorian calendar and returns the next"+CRLF
Instr$=Instr$+"Gregorian calendar date in the month/day/year format."+CRLF
Instr$=Instr$+CRLF
Instr$=Instr$+"The algorithm to determine the next date is based on the Gregorian calendar,"+CRLF
Instr$=Instr$+"which is commonly used throughout the world today"+CRLF
Instr$=Instr$+CRLF
Instr$=Instr$+"The output is always in the m/d/yyyy format regardless of the user's locale"+CRLF
Instr$=Instr$+"setting (meaning the output is not culture sensitive)."+CRLF
Instr$=Instr$+CRLF
Instr$=Instr$+"The valid range for input is 1/1/1582 through 12/31/3000"
MsgBox(Instr$,64,"NextDate")
'
' Initial Display
'
Text1$=""
CallDialog Dialog,"Next Date"
'
' Subsequent Display
'
While btnOK = 1
Text2$=""
Text1$=NextDate$(Edit1$,Edit2$,Edit3$)
If Text2$ <> "" Then
Text1$=Text2$
EndIf
CallDialog Dialog,"Next Date"
Wend
January 11, 2009
More Differences Between Developers And Testers
Arik at Testuff writes the following about Ido Schacham (SQAForums member idosius). To me, this is a terrific explanation of some of the differences between testers and developers:
"During development, there is always the dilemma of whether to let developers test the product or to hire testers to do the job. I always believed that it was better to hire a tester, but if you needed some more testers, you could use developers to temporarily extent the testing team. I thought that testing was an easier job then was programming, and it made sense to me that if the developer has some free time on his hands, the company can use him as a tester as well.
After a few months with Ido I realized that I was way off. Asking a developer do a tester’s job is like asking a hen to fly – he can do it, but usually rather badly. Developers have a particular state of mind that helps them write new code, solve problems in old code and improvise solutions when the direct approach doesn’t seem to work. They subconsciously trod lightly when using the software and cautiously walk around problematic areas and places they know might now work correctly. A tester’s state of mind is altogether different – he’s always looking for new and exciting ways to bring the program to its knees and is constantly on the lookout for the smallest inconsistencies, controls that are off by even one pixel and ways the application can be improved. I’ve never thought of it this way until I had a chance to work alongside a world-class tester."
I think that this is very well stated.
At SQAForums, I disagreed with much of what Ido wrote about Testuff's features, cutesy icons, fonts, and the use of video recordings as bug reports.
But if this change of attitude came about due to Ido, then he performed his job at Testuff well.
Labels:
QA
January 10, 2009
Not Increasing Customer Trust or Site Transactions
Recently, I received an offer in the mail from Thawte about Extended Validation SSL Certificates.
In part, the marketing materials said:
Increase customer trust and you increase site transactions.
Learn how the green address bar can help.
Find out in our FREE EV Info Pack.
Visit www.thawte.com/dm/9052/
The URL was repeated two other times on the page.
So I went to my computer and typed in www.thawte.com/dm/9052/ and press Enter. I got redirected to a generic error page:
Puzzled (and since this lovely error page didn't bother to tell me what I had actually typed), I tried again. Again, I got the error page.
Fortunately, I decided to remove the trailing slash and try again. This time, I got the page that the marketing material intended:
A few lessons here:
- It's quite simple to set your site up such that the trailing slash is optional. Do it.
- Always make sure your marketing materials match the experience you want for your customers. Test them! If you don't, you may end up wondering why your campaign was such a failure.
- If you must send your potential customers to a generic error landing page, show them what they typed. Often, they will spot their error right away and correct it. If not, and they decide to contact your support folks, the relevant information will be there.
Labels:
QA
January 4, 2009
QA Q and A - Smoke Test
QA
Q&A
Question: What is a Smoke Test?
Answer: A subset of all defined/planned test cases that cover the main functionality of a component or system, to ascertain that the most crucial functions of a program work, but not bothering with finer details. A daily build and smoke test is among industry best practices. [ISTQB]
When you want to quickly assess the state of a software build, you often turn to a Smoke Test (sometimes called a Build Verification Test). Essentially, you are trying to do something that is relatively quick and inexpensive, that will give you a general feeling for the software. You aren't looking for all the bugs. You aren't trying all the scenarios. You just want a general idea if the software works or not.
Often, the goal of this Smoke Test is to decide if more in-depth, more expensive testing is worthwhile, or if the software is too broken to be bothered.
Usually, you exercise only the basic paths of the software, and avoid all the unusual conditions. And usually, you don't test deeply at all - you just skim the surface.
In her book Effective Software Testing: 50 Specific Ways To Improve Your Testing, Elfriede Dustin says that a Smoke Test is "a condensed version of a regression test suite". And many companies do just that - extract a portion of their overall regression test suite, and use it repeatedly as a Smoke Test.
Manual Smoke Testing
Sometimes, a manual Smoke Test just involves "playing with the software for a while". For an experienced user/tester of the software this may make sense. Such a tester may have a good sense of what to try, and just how deep to go.
But often, a written script tells the manual Smoke Tester what to test and how.
Either way, the manual Smoke Test should be brief and to the point. If the software is in generally good shape, the manual Smoke Test should pass. If not, the Smoke Test should fail.
Automated Smoke Testing
I see this as one of the basic uses for my Test Automation Suite. I want to be able to quickly run a test after the build, so I can assess whether or not my team should bother to dig in and spend their time testing it. If the system-under-test passes the Smoke Test, we can proceed. If not, we reject the build until it is fixed.
If the builds occur overnight, I like to be able to schedule this Smoke Test so that it runs after the build and so that the results are ready and waiting for me when I get in the next morning. Sometimes, this allows me to run a larger overnight test and still have the results ready for the morning.
When I start to automate the testing of a system, the Smoke Test is usually the first automation I create.
In their excellent book How We Test Software At Microsoft, Alan Page, Ken Johnston, and Bj Rollison list these attributes of an automated Smoke Test (Build Verification Test):
- Automate Everything - run it on every build, the same every time.
- Test a Little - verify basic functionality.
- Test Fast - should execute quickly
- Fail Perfectly - fail only when the build isn't suitable for further testing
- Test Broadly, Not Deeply - focus on primary usage scenarios
- Debuggable and Maintainable - easy to fix, and keep up to date
- Trustworthy - you must be able to trust these tests
- Critical - requires time and careful thought to get it right
Labels:
QA
Subscribe to:
Posts (Atom)