The Barbara Test

(not actually Barbara)

About 10 years ago, I was lucky enough to be Employee #9 at a startup software company. 

It was a terrific experience.  I got to do a bunch of interesting things.  I learned a lot.

Like many startups, we did a lot of things wrong.  Although none of the original members work there any longer, the company is still around, so I suspect we did at least a few things right as well.

One of the few things we did right was something I like to call "The Barbara Test".

As we developed the first version of our first product, we had a very strong sense of the target user.  Our CEO was a domain expert in our product area.  She knew the problems faced by the target user.  In fact she knew the target user - her name was Barbara.

Early on in the development cycle, a bunch of us (most of the company, actually) flew out to visit Barbara on her home turf in the midwest.  We were able to hear and see what Barbara knew, what she did on a daily basis, how she did it, and the tools she used to do it.  We could see what was easy for Barbara, what was hard, and what was nearly impossible.  We asked Barbara what would make her job easier, what she could live with, and what she couldn't live without.

This was terrific insight!  It was a living, breathing set of User Scenarios.

Armed with this knowledge, we had a great way to judge our design, to assess our decisions, to triage our bugs, to set our priorities, to focus our efforts.  Whenever we had a question, we could always ask ourselves:

• How would Barbara do it?
• Is this easy enough for Barbara?
• Can Barbara get her work done in time, using this feature?
• Does this report have enough detail for Barbara?
• Would Barbara prefer this, or that?
• What kind of Help system does Barbara need?
• Could Barbara live with this bug, or must we fix it now?

In the end Barbara was happy, and so were we (her company purchased our product).

As you test, it's nice to have an understanding of the target user for your product.  If you can't actually meet her, at least be able to imagine her.  Ask your Project Manager: "Tell me about our Barbara."

Think First, Then Test

(I have this statue on my desk)

You've just been given a project to test.  It's a big deal - important to the company, and important to you.  So you are ready to jump in and get going.  Should you just play with the first build?  Should you do some exploratory testing?  Should you check for that first bug you know is almost certainly waiting for you?  Should you start creating some automation?  Should you write a test plan first?

While people seem to naturally have a preference for action, I believe it's most important to think first, then test.

As Cem Kaner, James Bach and Bret Pettichord say in their excellent book Lessons Learned in Software Testing:

The difference between excellent testing and mediocre testing is how you think: your test design choices, your ability to interpret what you observe, and your ability to tell a compelling story about it.  The rest of testing is ordinary office work, for the most part.

They indicate four main categories of thinking that figure into the practice of testing:

Technical thinking.  The ability to model technology and understand causes and effects.  This includes things like knowledge of relevant technical facts and the ability to use tools and predict the behavior of systems.

Creative thinking.  The ability to generate ideas and see possibilities.  You will test only in ways that you can imagine testing.  You will look only for problems that you imagine can exist.

Critical thinking.  The ability to evaluate ideas and make inferences.  This includes the ability to detect and eliminate errors from our thinking, to relate product observations to quality criteria, and to build a compelling case for a particular belief or suggested course of action.

Practical thinking.  The ability to put ideas into practice.  This ability includes such skills as applying test tools and making test techniques and effort fit within the scope of the project.

Here are a few things to think about.

Before you test...

• Who are the relevant stakeholders for this project?
• What do they need from me most?
• How high should the "quality bar" be for this project?
• What are the risks?
• What could go wrong?
• What is the most important thing to test?
• In what order should things be tested?
• What tools do I need?
• What skills do I need to learn before testing this?
• Who could help?
• How much time do I have?

While you are testing...

• What have I learned during testing that I didn't know before?
• What seems riskier?
• What kind of overall quality am I seeing?
• What should I tell the stakeholders about it?
• Should I change my planned strategy?
• Do I need additional tools?
• Do I need more help?
• Do I need more time?
• What is most important for me to do right now?

After you test...

• What worked well?
• What didn't work as well?
• How could I have done that more efficiently?
• What didn't I catch, and why not?
• What should I do differently next time?
• What notes/documentation should I leave for next time?

Messages from Me

In my Spam filter, I've been seeing more and more messages from "me".

For example, right now, I see these:

From: me
Subject: What is your floor?

From: me
Subject: About your results

From: me
Subject: 2 u

From: me
Subject: Let's dine together tonight

From: me
Subject: Our group dismissed

From: me
Subject: Hilarious party, come

From: me
Subject: Your card found

From: me
Subject: Read through this

From: me
Subject: Order #462803

From: me
Subject: Drive me home

From: me
Subject: Re: Your Hyrdoc0done Order #009642

I have to admit I was intrigued by this one:

From: me
Subject: Work is cancelled today

After all, who would know better if work is really cancelled than me?  So I clicked it.

Thoughtfully, Gmail provides these warnings when I actually decide to peek at one of these messages from me, and even invites me to learn more:

Warning: This message may not be from whom it claims to be. Beware of following any links in it or of providing the sender with any personal information.  Learn more
This message was likely forged and did not originate from your account. Learn More

That's quite kind of Gmail.  Even so, I'm guessing that I would know if it actually originated from me.

Sadly, it didn't originate from me.  Work isn't actually cancelled today - but if you still want to skip work on a Friday before a long holiday weekend, you didn't hear it from me.

Now, I know me. 
I've known me for more than 50 years. 
In fact, I know me almost as well as I know myself.
And I like me, but I guess I just don't like so many messages from me. 

If I tell myself not to send so many silly emails, will that actually stop me?

WinTask versus WinTask Lite

P>If you are considering WinTask as a test automation tool, is it necessary to purchase the full WinTask, or can you get by with the lower-cost WinTask lite?

The simple answer - if you want to do any real test automation, then Yes - you should purchase WinTask.  WinTask Lite is missing several functions that are very, very handy for the tester.

The logging functions are useful for writing a results log
The response time functions are useful for checking the duration of tasks
Some of the system functions are necessary for creating extensions to WinTask (to call into system DLLs, for example)
The web functions are essential for testing web-based applications
In some cases, the OCR functions can get text that is otherwise not accessible

Here's what you will NOT get in WinTask Lite (based on version 3.5a):

Window management functions

• CheckedW - Retrieves the state of a checkbox/radiobutton
• ClickOnTextOCR - Clicks mouse button on a text recognized by the OCR engine inside a window
• SelectedItem$ - Returns the selected item in a listbox/combobox
• SendKeysEncrypted - Sends encrypted keyboard input to the last specified Window
• UseWindowRegEx - Specifies, using Regular Expression, the window where the script will now send its keystrokes
• WriteEditEncrypted - Writes an encrypted string into the specified edit field  

Capture functions

• CaptureAreaOCR$ - Captures the text that appears in the specified area of a window using OCR
• HardCopy - Saves a screenshot  

Synchronization functions

• #ErrorFunction$ - Gives the name of the function which causes the error
• #ErrorMsg$ - Gives the error message sent by the function which causes the error
• #ErrorScript$ - Gives the name of the script where the error occurred.
• Disable - Disables the management of a specified event
• Enable - Reactivates the management of a specified event
• OnAction ... EndAction - Manages events
• OnAction Error ... EndAction - Manages error-handling
• Sleep - Makes the current script sleep whereas the events are still managed  

User dialog

• BeginDialog...EndDialog - Defines a dialog box with its controls
• CallDialog - Displays a dialog box defined previously
• InputboxSecret$ - Displays a simple dialog box where the user can type a hidden value
• SelectDir - Returns the name of the directory selected by the user in the standard Browse for Folder dialog
• SelectFile - Returns the name of the file selected by the user in the standard dialog box File Open
• SelectMultipleFile - Returns the name of the files selected in the standard dialog box File Open 

File management functions

• AppendXMLNode - Adds a node in the specified XML file
• EnumXMLAttributes - Retrieves the attribute names and attribute values for the specified XML node
• EnumXMLChildren - Enumerates the child node descriptors for the specified XML node
• GetXMLAttribute - Retrieves the content of an attribute in the specified XML file
• ReadIni$ - Reads a parameter in the specified INI file
• SetXMLAttribute - Modifies or adds an attribute in the specified XML file
• WriteIni - Writes to the specified INI file

Flow control functions

• #ErrorCode - Gives the error code sent by the function which causes the error
• #ExecTimeout - After this timeout, execution of the script is stopped
• #LastErrorLine$ - Gives the script line number where the error occurred
• #ScriptAfterTimeout$ - Specifies the script to run after timeout  

String Management functions

• Encrypt - function encrypts the specified string

System functions

• Allocate - Reserves a memory area for data used by external DLL
• DeleteRegKey - Deletes the specified key in Registry
• DeleteRegValue - Deletes a value in Registry
• External - Calls an external DLL
• External$ - Calls a Windows DLL
• GetCpuLoad - Returns the CPU load percentage
• GetMemUsage - Returns the memory used percentage
• GetProcessCpuLoad - Returns the CPU percentage used by a process
• GetProcessList - Gives the list of active process and their attributes
• GetWindowsList - Gives the list of parent window names present on desktop
• ImpersonateUser - Allows WinTask to acquire additional rights
• KillApp - Terminates the specified application
• KillAppChildren - Terminates the specified application and its child processes
• KillProcess - Kills the specified process
• PeekInteger - Reads one or several bytes in memory and returns an integer
• PeekString$ - Reads a string in memory
• PokeInteger - Writes in memory a value of type integer or Unsigned
• PokeString - Writes in memory a string
• ReadIni$ - Reads a parameter in the specified .INI file
• ReadReg - Reads an integer or a string from Registry
• RevertToSelf - Cancels an impersonation made previously by ImpersonateUser
• SendEmail - Sends an email using the defined SMTP server
• WriteIni - Writes in the specified .INI file
• WriteReg - Creates or modifies a string or numeric value in Registry

Clipboard and log functions

• Comment - Writes a comment in the log file
• LogFile - Forces the script to log its actions in the specified logfile
• StopLog - Stops recording in the logfile 

Com port management functions

• CloseCom - Closes the specified com port
• OpenCom - Opens the specified com port
• ReadCom - Reads the data in com port buffer
• WriteCom - Writes data in the com port buffer

Services management

• IsServiceStarted - Tells if the specified service is started or not
• StartService - Starts the specified service
• StopService - Stops the specified service

Web functions

• CopyLink - Copies the link associated with the specified HTML element into the specified string
• ExistHTMLElement - Checks if the specified HTML element exists or not
• ExtractLink - Returns all the links of the child elements of the specificed HTML element
• GetFrameSource - Retrieves the source of the specified frame
• GetPageSource$ - Returns the source of the current web page
• SelectedHTMLItem$ - Returns the selected item in an HTML list/combo
• WriteHTMLEncrypted - Writes encrypted text in a Web form 

Response time functions

• ResetTimer - Resets the specified clock
• StartTimer - Starts the specified clock
• StopTimer - Stops the specified clock
• Timer - Returns the value of the specified clock

ODBC functions

• #DbDateFormat - Controls the date fields format
• DbBof - Tells if recordset contains no records
• DbClose - Closes the recordset
• DbConnect - Establishes the data source connection through the odbc driver
• DbDisconnect - Closes the data source connection
• DbEof - Tells if pointer is at the end of the recordset
• DbExecute - Executes an SQL command on the opened data source
• DbGetFieldNumeric - Retrieves the value of a numeric field in a recordset
• DbGetfieldString - Retrieves the value of a string field in a recordset
• DbMove - Moves the current record pointer within the recordset at specified position
• DbMoveFirst - Positions the current record on the first record in the recordset
• DbMoveLast - Positions the current record on the last record in the recordset
• DbMoveNext - Positions the current record on the next record in the recordset
• DbMovePrev - Positions the current record on the previous record in the recordset
• DbRecordCount - Returns the number or records in the recordset
• DbSelect - Fills the recordset by retrieving in the table the records matching the sql query

OCR functions

• CaptureOCR$ - captures what is seen in the specified window, OCRizes the content and returns the text as analysed by the OCR engine
• CaptureAreaOCR$ - captures what is seen in the specified area of the specified window, OCRizes the content and returns the text as analysed by the OCR engine
• UseOCREngine - specifies which OCR engine will be used by subsequent OCR function calls

Additionally, there are a few differences in internal limits between WinTask and WinTask Lite.

For WinTask:

• Maximum number of procedures/functions: 500.
• A script cannot contain more than 32 local variables.
• A script cannot contain more than 1000 variables.
• A script cannot contain more than 128 arrays.
• Maximum size for an array is 65535 elements (so from 0 to 65534).
• Maximum number of parameters that can be passed to a procedure/function is 16.
• A script cannot contain more than 64 OnAction statements.
• A maximum of 5 Include levels is allowed.

For WinTask Lite:

• Maximum number of procedures/functions: 100.
• A script cannot contain more than 32 local variables.
• A script cannot contain more than 200 variables.
• A script cannot contain more than 64 arrays.
• Maximum size for an array is 65535 elements (so from 0 to 65534).
• Maximum number of parameters that can be passed to a procedure/function is 16.
• A maximum of 5 Include levels is allowed.

Crowdsourcer's Crowdsourcing Leaves a Crowd of Bugs

According to their website, uTest is "the world's largest marketplace for software testing services."

uTest operates on what they used to call a Pay-Per-Bug model, but now call Pay-For-Performance.  Perhaps that's just a nicer marketing phrase, or perhaps it's because many of their recent projects have been of the "fill out this survey and we'll give you a few bucks" type?  Either way, the point is to reduce the cost of testing to as close to $0 as possible.  And in exchange, the testers are faceless, unknown, unfiltered.  They are part of the "crowd", and the testing has been "crowdsourced".

I've kept an eye on them since they hit the news, and I just don't see where I'd use them for testing any of my company's products, but others are far more enamored.

James Whittaker says that crowdsourcing is the next logical step after outsourcing:

The next logical step in the evolution of testing is for vendors to provide testers and this is exactly the era we’ve entered with crowdsourcing. Yesterday’s announcement by Utest marks the beginning of this era and it is going to be very interesting to see it unfold. Will crowdsourcers outperform outsourcers and win this market for the future? Clearly market economics and the crowds’ ability to execute will determine that but my personal view is that the odds are stacked in favor of the crowd. This is not really an either-or situation but the evolution of the field. The older model will, over time, make way for the newer model. This will be a case Darwinian natural selection played out in the matter of only a few short years. The fittest will survive with the timeframe determined by economics and quality of execution. Crowdsourcing has much going for it including the sheer number of tests and test environments that can be brought to bear by the size and expertise of the crowd. 
(James Whittaker - August 20, 2008  http://blogs.msdn.com/james_whittaker/archive/2008/08/20/the-future-of-software-testing-part-1.aspx)

(Shrug)  I don't know.  I'm really trying to keep an open mind.  Maybe they'll be huge.  Maybe I'll eventually find a need for their services.  Maybe they'll drive the cost of testing way down.  Maybe they'll replace the need for professional testing.  Maybe not.

Recently the premier crowdsourcer that James hails decided to turn the crowd loose on the redesign of their own website.

Of course, we utilized our QA community to test the new site, blog and forums, as well as to conduct usability testing.  This provided us a with great deal of valuable testing before we flipped the switch on this new version. 
(Matt Johnston - May 12, 2009  http://blog.utest.com/kicking-off-a-new-era-utest/2009/05/)

Now, my friend Jim Hazen (a fellow software test professional) and I read uTest's redesign announcement and checked out their site.  Jim noticed a broken link on one of their new pages that starts off "Why pay an outsourcing firm for QA services you don't need? " and told me about it.  It seemed rather ironic.
(http://www.sqablogs.com/jimhazen/2274/But+who+is+going+to+Test+the+Testers.html)

So of course I ran a link check scan of their site using Xenu Link Sleuth (because that's what testers do).  This took about 45 seconds.  I found 14 broken links, involving some very prominent pages.  Go figure.

• Maybe the crowd provided a great deal of valuable (and inexpensive) testing.
• Maybe the crowd found these bugs, but they weren't fixed.
• Maybe the crowd's size and expertise was brought to bear on this site, and it's much better than it would have been otherwise (I shudder to think).

Or maybe the next logical step should have been professional (yet rather smaller than a crowd) testing.

American Airlines Email Error

My friend Kevin recently received this lovely email from American Airlines.

---

---

Now, Kevin manages a team of folks testing a Test Automation Tool Suite for a global software vendor.  He knows his way around testing.  And he knows a bug when it hits him in the inbox.

Imagine receiving an email with a Subject line that starts off with <ERROR>.

Would you think:

• Gee, why do they have me confused with this ERROR guy?
• That's just American Airlines doing what they do best.
• It would be an Error to look at this mail.
• Perhaps they should have tested more.
• In what language does "Error" mean "Hello"?
• Must be a bug in Vista.
• Did they crowd-source the testing of this email campaign?
• Hmm, maybe I should Google the <ERROR> HTML tag.
• If I read this email using Chrome, will it still say Error?
• I gotta get me some of those low-cost ERRORs while I still can.
• Those wacky, wacky marketers are drunk again!

Thanks, Kevin!  Did you take them up on their Error (er Offer)?

WinTask - RandomizeArray

'
' RandomizeArray - Randomize the elements in an array
'
' Author: Joe Strazzere
'
   
DIM MyArray$(10)    ' This is the array to be randomized
   
' Populate the Array
MyArray$(0) = "a"
MyArray$(1) = "b"
MyArray$(2) = "c"
MyArray$(3) = "d"
MyArray$(4) = "e"
MyArray$(5) = "f"
MyArray$(6) = "g"
MyArray$(7) = "h"
MyArray$(8) = "i"
MyArray$(9) = "j"
   
ArrayCount = 10    ' The number of elements in the array
   
' This is the part that randomizes the array
i=0
While i < ArrayCount
    j=Random(ArrayCount)
    tmp$ = MyArray$(i)
    MyArray$(i) = MyArray$(j)
    MyArray$(j) = tmp$
    i=i+1
Wend
   
' Display a few elements to show that they are randomized
MsgBox(MyArray$(0))
MsgBox(MyArray$(1))
MsgBox(MyArray$(2))

WinTask - FFStartBrowser

'
' FFStartBrowser - Starts FireFox  (like StartBrowser)
'
' Author: Joe Strazzere

 

Function FFStartBrowser(URL$)
    Local URL$
    Shell("Firefox.exe "+URL$)
    Pause 5 secs
    If ExistW("FIREFOX.EXE|MozillaWindowClass|Firefox Add-on Updates|1",1) Then
        CloseWindow("FIREFOX.EXE|MozillaWindowClass|Firefox Add-on Updates|1",1)
    EndIf
    If ExistW("FIREFOX.EXE|MozillaDialogClass|Firefox - Restore Previous Session",1) Then
        CloseWindow("FIREFOX.EXE|MozillaDialogClass|Firefox - Restore Previous Session",1)
    EndIf
EndFunction

 

FFStartBrowser("http://www.wintask.com")

WinTask - FFCloseBrowser

'
' FFCloseBrowser - Stops FireFox  (like CloseBrowser)
'
' Author: Joe Strazzere

 

Function FFCloseBrowser()
    KillApp("firefox.exe",1)
EndFunction

WinTask - FFNavigate

'
' FFNavigate - Navigates to a new URL using Firefox (like Navigate)
'
' Author: Joe Strazzere

Function FFNavigate(URL$)
    Local URL$
    UseWindow("FIREFOX.EXE|MozillaUIWindowClass||1",1)
        SendKeys("< Alt>fl")
        SendKeys(URL$)
        SendKeys("< Enter>")
EndFunction

FFNavigate("http://www.google.com/")

WinTask - FFScrape

'
' FFScrape - Return the contents of the Firefox page
'
' Author: Joe Strazzere

Function FFScrape$()
    UseWindow("FIREFOX.EXE|MozillaUIWindowClass||1",1)
        SendKeys("< Alt>ea")
        Pause 5 secs
        SendKeys("< Alt>ec")
        Pause 5 secs
        FFScrape$=GetClipboard$()
EndFunction

var$=FFScrape$()
msgbox(var$)

Perhaps They Should Have Tested More - Tesco

Giant UK supermarket chain Tesco has been unable to take money at a number of its stores following technical problems affecting its tills.

• Some stores were forced to close for several hours
• Long queues and shopping mayhem
• Waiting lines of up to half an hour for self-service checkouts in some stores
• Checkouts at 67 of Tesco's 2,814 UK stores were not working
• Likely to have been caused by routine maintenance of IT systems
• The problem was probably caused by software updates that had been sent out on Sunday night
• The computer malfunction came as the supermarket giant relaunched its Clubcard loyalty scheme

Perhaps they should have tested more.

See Also:

http://news.zdnet.co.uk/itmanagement/0,1000000308,39650939,00.htm
http://www.guardian.co.uk/business/2009/may/11/tillscrash-tesco-closures
http://news.bbc.co.uk/2/hi/business/8043937.stm
http://www.google.com/hostednews/ukpress/article/ALeqM5g2j7187TWfJylYhs6Nopix_n44Rw
http://www.belfasttelegraph.co.uk/business/business-news/tesco-mayhem-as-tills-crash-14298449.html
http://www.theregister.co.uk/2009/05/11/tesco_tills/
http://www.thisisnottingham.co.uk/homenews/Delays-Victoria-Centre-Tesco-glitch/article-981511-detail/article.html

Trial Versions of Commercial Load Test Tools

Many vendors of commercial load test tools offer trial versions.

Here are some:

e-Load

http://www.scl.com/oracle/application-quality-management/howtoeval

LoadRunner

https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-126-17^8_4000_100

LoadTest

http://www.appperfect.com/download/index.html

NeoLoad

http://www.neotys.com/evaluation/download.html

OpenLoad

http://www.opendemand.com/openload/trial.shtml

PureLoad

http://www.minq.se/products/pureload/download/

QEngine for Load Testing

http://www.adventnet.com/products/qengine/download.html

Qtest

http://www.quotium.com/form/product_form.php?product=qtest&type=demo

Rational Performance Tester

http://www.ibm.com/developerworks/downloads/r/rpt/index.html?S_TACT=105AGX23&S_CMP=DWNL

SilkPerformer

http://www.microfocus.com/products/SilkPerformer/SilkPerformerProductTrials.asp

SmarteLoad

http://www.smartesoft.com/downloads.php

StressTester

http://www.stresstester.com/stresstester_download.php

TestComplete

http://www.automatedqa.com/downloads/testcomplete/

WAPT

http://www.loadtestingtool.com/download.shtml

WebLOAD

http://www.radview.com/DownloadCenter/ProductEvaluation.aspx

Web Performance Load Tester 

http://webperformance.com/sales/signup.html

Webserver Stress Tool

http://www.paessler.com/webstress/download

(last updated December 31, 2010)