July 30, 2006

Perhaps They Should Have Tested More - MBTA

MBTA glitch has riders feeling robbed

The MBTA's Charlie is ripping you off, sort of.
Mike of Winthrop explains about a glitch he found in the T's new automated fare collection system:

``Since I took a ride on the Silver Line (90 cents) I had an extra 35 cents on my CharlieTicket. No problem! I simply went to the machine and was relieved to see that there is an `add value' function which would let me add 90 cents so that I could take the T for $1.25," he wrote.

You know what's coming next.

``Every time I tried to add 90 cents, the machine rejected my card with the notice `Card Value does not have minimum of $1.25,' which means that `add value' does not work on any left over change below $1.25.

``Please have the MBTA write me a check for the money owed -- since it is lost to me -- 35 cents down the drain. I never would have used my card on the Silver Line!"

We thought this sounded fishy so we did our own little expensive experiment. Down to JFK/UMass we walked, up to a CharlieTicket vending machine, where we used the ``other amount" button to purchase a ticket for $1.60 to cover bus and subway fare.

We walked through the turnstile, turned around, walked back out the turnstile, stuck our ticket back into the vending machine and checked the amount on it, which was 35 cents.
And lo and behold, when we tried to add 90 cents in value to boost it back to $1.25, Mike was right. The machine would accept nothing lower than $1.25.

And there it sits in our wallet, 35 unusable cents.

(By the way, since there were no change machines around, we're also walking around with a hernia-inducing heap of dollar coins in our pocket.)

We talked to MBTA General Manager Daniel Grabauskas late Friday, who pledged that if the problem is a software glitch, it will be fixed.

``If it's an additional software programming change, and that's what our customers want, then that's what they'll get."

July 29, 2006

Patriots Training Camp 2006

Went to Patriots Training Camp in Foxboro today - lots of fun.


During the evening session we had a brief rainstorm. Afterwards this:



I consider this a good omen!

July 22, 2006

Thinking Like a Professional Tester

A recent incident at work got me thinking about characteristics that distinguish a casual tester from a professional.

Until recently, this company had no professional QAers. Any testing had been done by Product Management folks, and by customers during User Acceptance Testing. These people did the best they could, but they clearly had a lot on their plate - finding bugs wasn't their highest priority.

This time, one person was asked to go and delete a bunch of test accounts that had been added to a system. She deleted most of them but reported that some couldn't be deleted. She wasn't sure why, but they just "wouldn't delete". She didn't seem concerned.

I asked her to put together a list of the accounts which could be deleted and those which could not. When I saw the list, it appeared that all of those which couldn't be deleted had apostrophes in the name. I gave her a few suggestions and asked her to investigate.

It turned out that accounts containing special characters in the name could not be deleted. This bug had been there for quite a while. People testing the system in the past had noticed the problem, but hadn't understood why it happened. Even the customer had noticed the problem, but had just asked the developers to go into the database and delete the accounts.

To me, this points out a few differences between casual testers and professionals:
  • The ability to recognize that a bug is occurring
  • The desire to dig in and find out the characteristics of the bug
  • The skill to generalize the problem
  • The willingness to initiate a bug report

July 8, 2006

Perhaps They Should Have Tested More - Churchill Race Track

Saturday, July 8, 2006

Computers caused Pick 3 glitch

Surface change created confusion

By Jennie Rees
jrees@courier-journal.com
The Courier-Journal



Tuesday's Pick 3 snafu involving the seventh race was caused by a United Tote software problem apparently triggered when a new rule came into play for the first time, Churchill general manager Jim Gates said yesterday.



Churchill received permission last fall to change Pick 3 rules, specifically addressing the problem for when a race comes off the turf after wagering on a multi-race bet has closed.



But the situation had never occurred before the jockeys, after the fifth race, requested the seventh and ninth races come off the grass following a mid-afternoon downpour. By then, betting had closed on the Pick 3 involving races five through seven.



Under the new rule, every horse in the leg involving the surface change is considered a winner. Complicating matters were a scratch after the sixth race of a horse running in the seventh race and a scratch of a horse in the post parade for the eighth race. Those scratches resulted in consolation payoffs in the Pick 3s ending with the eighth and ninth races.



Gates said United Tote had tested the system to accommodate the rules changes and hadn't had any problems.



"…When they inputted the surface change into the system, it wanted to refund all of the wagers, which was not supposed to happen," he said. "In order to get that fixed, they had to shut down that system to make that change. That was the reason for prices not being posted until that evening after the races were over."



The ninth race was not a problem because the surface change was announced before betting closed on the Pick 3 beginning with the seventh race, he said.



Many fans were angered by having to wait until yesterday to cash, because Churchill was closed Wednesday and Thursday. Gates said those who bet at tracks or simulcast outlets that were open Wednesday could cash that day.

July 6, 2006

Generic Software QA Engineer Job Descriptions and Levels

Many companies use descriptions like these when determining pay levels for QAers.

Software QA Engineer Job Descriptions

Primary Responsibilities:
  • Debugs software products through the use of systematic tests to develop, apply, and maintain quality standards for company products. 
  • Develops, maintains, and executes software test plans.
  • Analyzes and writes test standards and procedures.
  • Maintains documentation of test results to assist in debugging and modification of software.
  • Analyzes test results to ensure existing functionality and recommends corrective action.
  • Consults with development engineers in resolution of problems.
  • Provides feedback in preparation of technical appraisals of programming languages, systems, and computation software.
  • Ensures quality computer integration into the overall functions of scientific computation, data acquisition, and processing.
QA Engineer Level 1
KNOWLEDGE:  Learns to use professional concepts.  Applies company policies and procedures to resolve routine issues.
JOB COMPLEXITY:  Works on problems of limited scope.  Follows standard practices and procedures in analyzing situations or data from which answers can be readily obtained.  Contact with others is primarily internal.
SUPERVISION:  Normally receives detailed instructions on all work.
EXPERIENCE:  Typically requires no previous professional experience.

QA Engineer Level 2
KNOWLEDGE:  Uses professional concepts; applies company policies and procedures to resolve a variety of issues.
JOB COMPLEXITY:  Works on problems of moderate scope where analysis of situations or data requires a review of a variety of factors.  Exercises judgment within defined procedures and practices to determine appropriate action.  Has internal and some external contacts.
SUPERVISION:  Normally receives general instructions on routine work, detailed instructions on new projects or assignments.
EXPERIENCE:  Typically requires a minimum of 2 years of related experience.  In some companies, the requirement will be less.

QA Engineer Level 3
KNOWLEDGE:  Uses skills as a seasoned, experienced professional with a full understanding of industry practices and company policies and procedures; resolves a wide range of issues in imaginative as well as practical ways.  This job is the full qualified, career-oriented, journey-level position.
JOB COMPLEXITY:  Works on problems of diverse scope where analysis of data requires evaluation of identifiable factors.  Demonstrates good judgment in selecting methods and techniques for obtaining solutions.  Interacts with senior internal and external personnel.
SUPERVISION:  Normally receives little instruction on day-to-day work, general instructions on new assignments.
EXPERIENCE:  Typically requires a minimum of 5 years of related experience.  In some companies, the requirement will be less.

QA Engineer Level 4
KNOWLEDGE:  Having wide-ranging experience, uses professional concepts and company objectives to resolve complex issues in creative and effective ways.  Some barriers to entry exist at this level (ie, dept/peer review).
JOB COMPLEXITY:  Works on complex issues where analysis of situations or data requires an in-depth evaluation of variable factors.  Exercises judgment in selecting methods, techniques, and evaluation criteria for obtaining results.  Internal and external contacts often pertain to company plans and objectives.
SUPERVISION:  Determines methods and procedures on new assignments, and may provide guidance to other personnel.
EXPERIENCE:  Typically requires a minimum of 8 years of related experience.  In some companies, the requirement will be less.  At this level, graduate coursework may be desirable.
QA Engineer Level 5
KNOWLEDGE:  Having broad knowledge or unique knowledge, uses skills to contribute to development of company objectives and principles and to achieve goals in creative and effective ways.  Barriers to entry such as technical committee review exist at this point.
JOB COMPLEXITY:  Works on significant and unique issues where analysis of situations or data requires an evaluation of intangibles.  Exercises independent judgment in methods, techniques and evaluation criteria for obtaining results.  Contacts pertain to significant matters often involving coordination among groups.
SUPERVISION:  Acts independently to determine methods and procedures on new or special assignments.  May supervise the activities of others.
EXPERIENCE:  Typically requires a minimum of 12+ years of related experience.  In some companies, the requirement will be less.  At this level, graduate coursework may be expected.

QA Engineer Level 6

KNOWLEDGE:  As an expert in the field, uses professional concepts in developing resolution to critical issues and broad design matters.  Significant barriers to entry (ie, top management review approval) exist at this level.
JOB COMPLEXITY:  Works on issues that impact design/selling success or address future concepts, products or technologies.  Often serves as a consultant to management and external spokesperson for the organization.
SUPERVISION:  Exercises wide latitude in determining objectives and approaches to critical assignments.
EXPERIENCE:  Typically requires a minimum of 15+ years of related experience.  In some companies, the requirement will be less.  At this level, a graduate degree may be expected.

July 4, 2006

You Shouldn't Jump to Conclusions

Here is a sanitized version of an interesting email conversation.

Everyone involved worked for the same company, but for different product lines (Product A and Product B).

Product B was used to track bugs, and included a new snap facility for attaching screenshots to Issue Reports. QAers involved with Product A used Product B internally.

A member of the Product A team concluded that Product B was buggy and shouldn't be used. He even recommended using a competitor's product! As you will read, he was a bit hasty in his conclusions.

--------------------------------------------------------------------------------
From: [Product A Architect]
Sent: Tuesday, June 27, 2006 1:17 PM
To: [QA]
Cc: [Product B Developer]

Subject: Images in [Product B] are very large

Most of the new issue I am seeing require long waits to show images that should be quite small but are being delivered as 1000x1000 (approx) BMP files.

I suspect this is the new screen snap facility in [Product B].

PLEASE don’t use this facility as it is buggy, seems to product HUGE files and probably takes up large amounts of db space. Perhaps I am wrong and this is simply a rendering technique in [Product B], but it does not look like it. If this is not due to the Product B and you are just pasting huge image files PLEASE stop.

I would suggest you start using [Competitor’s Product] for screen shots. It is fast, stable, cheap, allows time taken banners and very impressive in cabability.

[Product A Architect]
Development Manager / Software Architect

--------------------------------------------------------------------------------
From: [Product B Development Manager]
Sent: Tuesday, June 27, 2006 11:52 PM
To: [Product A Architect]
Cc: [QA]; [Product A Development Manager]; [Product B Developer]

Subject: FW: Images in [Product B] are very large

Hi [Product A Architect],

I understand your concern about image sizing and we are looking at this now. Thank you for raising this issue.

Could you please clarify what particular are of the most of your concern when you wrote, “PLEASE don’t use this facility as it is buggy?”

Thanks,

[Product B Development Manager]

--------------------------------------------------------------------------------
From: [Product A Architect]
Sent: Wednesday, June 28, 2006 1:15 PM
To: [Product B Development Manager]
Cc: [QA]; [Product A Development Manager]; [Product B Developer]

Subject: RE: Images in [Product B] are very large

:o)

Wouldn’t you call the creation of full screen images in BMP format (the most inefficient format available) when snapping small sections of image resulting in download waits long enough to prevent common use a bug?

Until that very significant problem is addressed, I would FAR prefer that our QA staff (for [Product A] anyway) not use the facility at all rather than produce unusable and space consuming images. A good screen capture facility is very inexpensive.

I have no specific complaints beyond that.

--------------------------------------------------------------------------------
From: Strazzere, Joe
Sent: Wednesday, June 28, 2006 6:36 AM
To: [Product A Architect]; [Product B Development Manager]
Cc: [QA]; [Product A Development Manager]; [Product B Developer]l; [Product Manager]

Subject: RE: Images in [Product B] are very large

The snap screen facility can indeed produce BMP files, but produces JPG files by default.

I haven’t seen that they are any larger than JPGs produced by other tools. Nor have I seen where it “takes up large amounts of db space”.

If there are bugs, someone should write Issue Reports, rather than just telling people not to use it because “it is buggy”.

The new snap screen feature is part of [Product B], and is now a shipping product.

-joe

--------------------------------------------------------------------------------
From: [Product A Architect]
Sent: Wednesday, June 28, 2006 1:27 PM
To: Strazzere, Joe; [Product B Development Manager]
Cc: [QA]; [Product A Development Manager]; [Product B Developer]l; [Product Manager]

Subject: RE: Images in [Product B] are very large

Perhaps this is just [An individual QAer’s] use of the tool? There is quite certainly a problem somewhere even if you are unaware of it’s existence. If I use a tool and find specific problems, I will forward the specifics to the folks who seem to be using it.

[Product B] is not a tool I work on and I am not involved in the QA or development of it. Like ANY other tool in use for development, if there is a problem in deployment, I will halt use of particular features of the product if it is causing problems. I quite frankly don’t care in the least who supplied the tool: if it does not work, I will ask those using it to stop the process that does not work. We will NOT continue to use it pending an issue resolution just because it is internal product. This is about effective management of a development process for [Product A] and not about the development process of the tool in use.

I do understand the concerns and needs of the [Product B] team, but since this thread exists, they do know about the problem I sensed within the product. If that perceived problem is resolved quickly or is based on a faulty understanding or the use of a separate tool by particular individuals, then we can resume use of that feature. Until the problem is isolated, please refrain from using the snap screen facility for [Product A] bugs since at this time it appears to have some problems.

--------------------------------------------------------------------------------
From: Strazzere, Joe
Sent: Wednesday, June 28, 2006 1:33 PM
To: [Product A Architect]; [Product B Development Manager]
Cc: [QA]; [Product A Development Manager]; [Product B Developer]l; [Product Manager]

Subject: RE: Images in [Product B] are very large

You are correct in your conclusion that some images attached to some Issues are overly-large.

You are incorrect in your conclusion that the snap screen facility is buggy.

[The individual QAer] was using MS-Paint, not snap screen.

-joe

Perhaps They Should Have Tested More - Manitoba Lotteries

Casino blames computer glitch for jackpot

WINNIPEG, Manitoba, July 4 (UPI) -- Two Canadian men are demanding a Winnipeg casino pay out the jackpot promised in error by a nickel slot machine.

The Manitoba Lotteries Corp. says the message that the men had won almost $210,000 ($190,000 U.S.) was a software error because the nickel machines usually do not have payouts above $3,000.

But attorney Josh Weinstein told the Canadian Broadcasting Corp. there was no sign on the machine giving a maximum payout. He says the men were promised 4 million nickels for successfully matching five numbers on the Keno machine.

"It's our position that it's not a mistake that my clients should be paying for, if it was a mistake," he said. "We don't have results of independent testing."