April 20, 2007

Perhaps They Should Have Tested More - Research In Motion

Some of the headlines:
  • RIM: 'Series' of Errors Caused Outage
  • Software glitch caused BlackBerry outage 
  • RIM traces BlackBerry outage to poorly tested software update
  • RIM chalks up blackout to "insufficient" testing
  • Software glitch KOd BlackBerry 
  • Software update brought down BlackBerry e-mail: RIM
  • Better optimisation caused RIM's Crackberries to crack
  • Blackberry outage exposes weakness for RIM
  • BlackBerry Shutdown Sends Danger Signal for RIM
  • BlackBerry Outage Caused By Untested New Feature


Last Tuesday night, millions of Blackberry users in the Western Hemisphere experienced a twelve hour service outage leaving them without wireless e-mail access.

In a statement issues on Thursday, April 19th, Research In Motion wrote:
"RIM has determined that the incident was triggered by the introduction of a new, non-critical system routine that was designed to provide better optimization of the system's cache. The system routine was expected to be non-impacting with respect to the real-time operation of the BlackBerry infrastructure, but the pre-testing of the system routine proved to be insufficient.

The new system routine produced an unexpected impact and triggered a compounding series of interaction errors between the system's operational database and cache."
(http://www.nytimes.com/2007/04/20/technology/20blackberrytext.html?_r=1&oref=slogin)

Some believe that poor testing procedures were used.
"Jack Gold, an analyst at J. Gold Associates LLC, said RIM "sure could have" kept customers better informed this week. "Nothing's worse than not knowing what's happening when a problem occurs."

Gold said he thought that the final explanation was a "bit confusing," adding, "It sounds like they tried to run something on their production servers without it being totally ready.... This is something they should be better at. Playing with software on a production environment is always risky. This time it bit them." "
(http://www.computerworld.com/action/article.do?command=printArticleBasic&articleId=9017270)

The failure may lead some to consider alternatives.  Already, some enterprise users are questioning the centralized network architecture of the Blackberry:
"BlackBerry is essentially an outsourced environment. Whenever there's an outage, this determination has to go on whether it's us or them. Once it leaves our site, we have no insight as to what goes on in their network."
(http://www.informationweek.com/news/showArticle.jhtml?articleID=199100624)

Many companies and individuals have become dependent on the wireless e-mail service, to the point that the devices are often called "Crackberrys".
"The BlackBerry blackout was grueling to many — and revealed just how professionally and emotionally dependent so many people had become on their pocket-size electronic lifelines."
(http://www.nytimes.com/2007/04/19/technology/19blackberry.html?em&ex=1177214400&en=388e6b0cc870b114&ei=5087%0A)

"After a massive outage of BlackBerry mobile devices across the globe Wednesday prevented millions from checking the e-mail on the go, workers and technology junkies faced an unsettling truth.

"This outage was more than just a little inconvenient. This was on par with (saying) my computer was down or my phone was down," said Michael Gartenberg, an analyst with Jupiter Research. "People couldn't work the way they were accustomed to working.""
(http://www.govtech.net/digitalcommunities/story.php?id=105093)
[Update 09/08/07]

Another "software glitch".  Another outage.
http://www.baltimoresun.com/technology/ats-ap_technology10sep08,0,3351825.story
Labels:

April 7, 2007

What Am I Worth?




In general, you are worth whatever someone is willing to pay you.

What you ultimately end up receiving for a salary depends on many factors:
  • The specific details of the job
  • What you have done, and how long you have done it
  • What the employer thinks you can do
  • Where you work
  • The industry in which you work
  • The size of the hiring company
  • The size of the hiring department
  • Your education
  • How much the employer values the specific position
  • How well you negotiate
  • How many other people are vying for the same position
  • The job market in general
  • etc, etc
Remember that, within reason, everything about a job is negotiable, including salary.
If the hiring company wants you badly enough, they can sometimes increase their offer.

Remember also, that there is more to a job than simply the salary.  Consider the whole package:
  • Bonus
  • 401(k) contributions
  • Stock options
  • Other benfits
  • Opportunity for advancement
  • Company culture
  • Commuting distance
  • Telecommuting options
  • etc, etc.
Here are a few web sites that can help you calculate what particular jobs might be worth in your area:
Labels: ,

Perhaps They Should Have Tested More - Yahoo! Japan

So we weren't supposed to delete the e-mail messages that were not junk mail? 
Oops!  Perhaps they should have tested more.




Yahoo Japan mistakenly deletes 4.5 mil. e-mails
The Yomiuri Shimbun

Yahoo Japan Corp. accidentally deleted about 4.5 million e-mails sent to 275,600 users due to a mistake in its e-mail service system, the company said.

Yahoo allows users to exchange e-mail messages for free without using an e-mail software.  According to the company, most of the deleted e-mails were received between Dec. 26 and Feb. 25.

The system usually removes e-mails deemed "junk mail" from its server about 40 days after the e-mails are received.  However, it wrongly deleted e-mails that should have been kept on the server.

The company discovered the mistake after receiving complaints from users that they could not open certain e-mails they had received.
(Apr. 8, 2007)

(http://www.yomiuri.co.jp/dy/national/20070408TDY02004.htm)
Labels:

April 6, 2007

Book: Surviving the Top Ten Challenges of Software Testing


William E. Perry and Randall W. Rice

While looking through the Common Body of Knowledge for the CSTE Certification of QAI, I saw that this book was part of the suggested reading list.

It was written by William Perry and Randall Rice (both of whom I had read before, and both of whom work for QAI), so I thought I'd buy a copy and give it a shot.

It's not bad as far as listing testers' challenges, but for me, it doesn't do enough to indicate how to actually solve those challenges.  I don't necessarily agree that these are the top ten challenges, nor that they are ordered correctly.  But if I did, I'd want more practical help in solving them.  Still, it's a good high-level overview of these solutions.

The book has a rather unusual format - each of the ten challenges is discussed in a chapter, using the same format for each chapter.

Here's an outline:

Challenge #10: Getting Trained in Testing
  • Overview
  • State of the Practice
  • Impact on Testing
  • Solutions to the Challenge
    • Raise Management Awareness of Testing
    • Make Time for Training
    • Develop Your Own Skills
    • Certify Your Testing Skills
  • Solution Impediments
    • What if my management will not support my training needs?
    • What if I do not have enough money to attend a testing seminar or conference?
    • What if I do not have enough time for training?
  • Guidelines for Success
    • Set personal goals
    • Seek management's assistance in building your testing skills, but don't wait for it
    • Invest in yourself
    • Strive to add value to the testing effort
  • Plan of Action
    • Get management on your side
    • Develop your own skill-building goals and objectives
    • Never stop learning!
Challenge #9: Building Relationships with Developers
  • Overview
  • State of the Practice
  • Impact on Testing
    • low morale
    • slow or stalled projects
    • inefficient testing
    • poor physical and mental health
    • destroyed relationships
    • termination of employment
    • project failure
  • Solutions to the Challenge
    • Adopt a Win-Win Approach
    • Widen Your View of Testing
    • Move from "Us versus Them" to "Us and Them"
  • Solution Impediments
    • How do we get everyone on the project involved in testing?
    • What if people don't want to be involved in testing?
    • How do we criticize constructively?
    • How do I get management support for testing?
  • Guidelines for Success
    • Build the test team so that participants from a variety of disciplines are involved
    • Make win-win the only way you operate
    • Keep the lines of communication open
    • Keep working on yourself
  • Plan of Action
    • Work on yourself first
    • Study the works on human interactions and potential
    • Keep a personal journal
    • Make a conscious effort to show others in the organization the value of testing and what it can mean to them
    • Involve others in the testing effort
Challenge #8: Testing Without Tools
  • Overview
  • State of the Practice
  • Impact on Testing
    • Manual testing is imprecise and unreliable
    • Manual testing is boring
    • Manual testing is labor intensive
  • Solutions to the Challenge
    • Educate Management on the Use of Test Tools
    • Perform a Tool Survey
    • Define Your Requirements
    • Perform a Cost/Benefit Analysis
    • Investigate Tools Available
    • Integrate Test Tools with an Effective Testing Process
  • Solution Impediments
    • What if my management does not see the need for test tools?
    • What if I can't get funding for test tools?
    • What if not tools exist for my environment?
    • What is other people in the organization do not want to use test tools?
  • Guidelines for Success
    • Secure management support for acquiring test tools
    • Use a team approach when selecting a tool
    • Make good use of what you have
    • Maximize tool use with people and processes
    • Use the right tools
    • Make tool usage mandatory
  • Plan of Action
    • Identify stakeholders
    • Lay the groundwork by raising test tool awareness among the stakeholders
    • Identify or build the testing process
    • Get approval and initiate a project to build a test toolbox
    • Form a team to identify which tools are already in place and to specify which tools are needed
    • Use the tool selection team to identify the tool requirements
    • Identify the selection criteria
    • Perform a tool search using all available information sources and tool directories
    • Contact the qualified tool vendors for detailed information
    • Evaluate the finalists
    • Select the tool
    • Develop the implementation plan, including integration of the tool and the testing process
    • Train the staff in the new process and in how to use the tool to support the process
    • Continue improving the processes
    • Continue training people as they are hired and as refresher training is needed
Challenge #7: Explaining Testing To Managers
  • Overview
  • State of the Practice
  • Impact on Testing
  • Solutions to the Challenge
    • Identify the Stakeholders at the Management Level
    • Raise Awareness of the Testing Function
    • Network with Other Organizations to Learn How They Deal with Management
    • Establish a Testing Charter to Define the Purpose of testing in Your Organization
    • Define Measurable Testing Objectives
    • Dedicate a Manager of Testing Who Understands the Issues and Challenges
    • Make a Testing Process
  • Solution Impediments
    • What if management is satisfied with the status qui, since we've done well in the past with ad hoc testing?
    • What if we can't find a champion for testing?
    • How do I find other organizations with which to network and benchmark?
  • Guidelines for Success
    • Be patient.  Raising management awareness of testing can take a long time
    • Quality is ultimately management's responsibility
    • Focus your message to management on cost and time
    • Meeting the schedule is not the only measure of success
    • Measure and publicize your efforts
    • Marketing is part of your job.  Learn how to market well
  • Plan of Action
    • Develop a testing charter
    • Identify the stakeholders
    • Identify your target market
    • Define your strategy and objectives
    • Assess your current condition
    • Start making the message
    • Keep making the message
Challenge #6: Communicating with Customers - And Users
  • Overview
  • State of the Practice
  • Impact on Testing
  • Solutions to the Challenge
    • Teamwork
    • Communication
    • Continuous Involvement
    • User Acceptance Testing
  • Solution Impediments
    • What if the end-users just don't want to be involved in building or testing the system?
    • What of end-users have an adversarial attitude toward the developers?
    • What if the project management rejects the idea of customer and end-user involvement?
    • What if the end-users do not understand technical issues or know how to perform acceptance testing?
  • Guidelines for Success
    • Communicate tactfully
    • Lighten up a little. Life will still go on after the project is over
    • You can't please everyone
  • Plan of Action
    • Start educating the development management and end-user management on the importance of identifying the customer and end-users
    • Include customer and end-user tasks in your system development and testing methodologies
    • Pilot the process
    • Training the users in performing their tasks
    • Measure the results
Challenge #5: Making Time for Testing
  • Overview
  • State of the Practice
  • Impact on Testing
    • Reduced Test Coverage
    • Increased Risk of Regression Defects
    • Fatigue, Burnout, and Low Morale
  • Solutions to the Challenge
    • Control the Scope of Testing
    • Control Management Expectations
    • Base Test Cases on an Independent Set of Criteria
    • Perform Risk Assessments
    • Reuse Your Testware
    • Estimate the Testing Effort Based on Measurable Criteria
    • Use Automation
  • Solution Impediments
    • What if my management refuses to acknowledge the need for better estimates, test tools, or any of the other solutions you suggest?
    • What if I do not have measurable criteria, such as requirements, to serve as the basis of my tests?
    • How do I control the scope of a test if the scope is constantly changing?
    • What is the common ratio of testers to developers?
  • Guidelines for Success
    • Remember: Work expands to fill available time
    • Have a plan - and plan for the unexpected
    • Control your scope
    • Test early and often
    • Use techniques that leverage your time and resources
  • Plan of Action
    • Define a test process that includes risk analysis
    • Develop test standards and templates that can be reused
    • base test estimates on measurable criteria, such as cases or requirements to be tested
    • Build test cases to validate requirements or business processes, not random combinations of software functions
    • Continually educate management that testing should be measured and estimated by fact, not guesswork or deadlines
    • Keep a history of test measurements to help estimate future projects
    • Continually streamline the testing process to make it more efficient and less time-consuming
    • Consider using an automated test tool for repetitive tests once the test process has been defined and you know what you need to test
Challenge #4: Testing What's Thrown Over the Wall
  • Overview
  • State of the Practice
  • Impact on Testing
  • Solutions to the Challenge
    • Get Management Support to Define Roles and Responsibilities
    • Establish Standards and Processes for Testing
    • Establish Ownership and Accountability at the Developer Level
    • Train Developers to Be Excellent Testers
    • Improve Communication Between Developers and Testers
    • Measure and Refine the Processes Continually
    • Establish Ground Rules
  • Solution Impediments
    • How do I overcome the skepticism over new process initiatives when past efforts have failed?
    • How do I convince management to take the lead in deploying processes?
    • What if the developers don't want to become better testers?
  • Guidelines for Success
    • People must be accountable for the products they produce
    • Management must build a quality culture
    • Test processes must be supported by test tools and test standards
    • Don't let your failures deter you from trying again
    • Process deployment is a function of organizational maturity
  • Plan of Action
    • Get management support for establishing clearly defined roles and responsibilities
    • Establish test standards and test processes
    • Build ownership and accountability into the test processes and standards
    • Train people in how to apply the test processes and standards
    • Continually measure and refine the processes
Challenge #3: Hitting a Moving Target
  • Overview
  • State of the Practice
  • Impact on Testing
  • Solutions to the Challenge
    • Rework of Testware
    • Regression Testing of Previously Tested Software
    • Backlog Created by Rapid Change
  • Solution Impediments
    • How do I test when my organization is changing at a chaotic rate?
    • How do I find the time to perform impact analysis where changes occur so fast in my organization?
    • What if users become impatient with waiting for releases where they otherwise could have more frequent revisions? 
  • Guidelines for Success
    • Change will always be a part of our lives, so be prepared for it
    • Change is occurring at an ever-increasing rate, so be aware of the risks
    • One small change can cause major problems
    • Where change is concerned, control is everything
  • Plan of Action
    • Analyze your situation and find out where changes originate
    • Establish a change control process that will manage changes
    • Make your testing processes change-tolerant by incorporating reuse and modularity
    • Investigate how automated tools can help in your environment
    • Explore the concept of scheduling changes in releases to control the process
    • Add regression testing to your testing process
    • Measure the number and impact of defective changes
    • Use the information from the measurements to improve the maintenance and testing processes
Challenge #2: Fighting a Lose-Lose Situation
  • Overview
  • State of the Practice
  • Impact on Testing
    • Keeping an Organization at a Low Level of Process Maturity
    • Trivializing and Undermining the Testing Process
    • Demoralizing the Testers
    • Fostering a False View of Testing
  • Solutions to the Challenge
    • Communicate the Role of Testing to the Rest of the Organization
    • Identify What Testers Can Reasonably Accomplish
    • Set and Manage Customer Expectations of Production Software
  • Solution Impediments
    • What if communication of the testing role falls on deaf ears in our organization?
    • What of management's expectations remain inordinately high, even though we've tried to explain the limitations to testing?
    • How do I set and manage expectations when our customers expect zero-defect software?
  • Guidelines for Success
    • Understand and communicate the role of testing and how it fits into the entire organization
    • Testing is more than one person's or one team's responsibility
    • It is incumbent on management to build a culture of trust and appreciation, not fear and blame
    • Our customers are our partners.  They have a part in determining the quality of software they receive
  • Plan of Action
    • Develop a testing charter that defines the role of testing on your organization
    • Publish the testing charter so that everyone in the organization understands what testing can or cannot accomplish
    • Define testing processes and standards that facilitate the objectives outline in the testing charter
    • Deploy testing processes and standards
    • Conduct training for the organization, including end-users and customers, on the details of the testing charter, testing standards, and testing processes
Challenge #1: Having to Say No
  • Overview
  • State of the Practice
  • Impact on Testing
  • Solutions to the Challenge
    • Standardize Test Reports
    • Make Test Reporting Part of the Testing Process
    • Manage Your Audience's Expectations
    • Use creative Reporting Techniques
    • Focus on the Facts
    • Be Truthful
    • Document Your Tests
    • Build a Mature Culture
  • Solution Impediments
    • What if my management only wants to hear good news?
    • How do we change the perception of testers as being negative?
    • How do I find the courage to say the system is not ready for production, even if my job is at stake?
    • How do developers decide whether or not to report defects found in a work in progress?
  • Guidelines for Success
    • Tell the truth, even if there are negative consequences
    • Stick to the facts
    • Deliver a solution or solutions for each problem reported
    • Make test reporting part of the testing process
  • Plan of Action
    • Assess your current method of reporting test results and see if improvement is needed
    • Develop test reporting standards
    • Make test reporting part of the testing process
    • Survey management, developers, customers, and end-users as to what information they can expect from testing they would find useful
    • Set expectations of management, developers, customers, and end-users as to what information they can expect from test reports
    • Explore creative ways to present test results using graphics, such as pie charts, bar charts, and the like
    • Improve the test reporting process by continually surveying management, developers, customers, and end-users to measure your effectiveness
Labels: ,

April 4, 2007

Constant Lack of Focus, or Continuous Partial Attention?



Some things I don't like to see happen during meetings:
  • Using laptops to check up on e-mails, send instant messages, etc
  • Constantly checking a cell phone - and even answering during a meeting
  • People talking to each other while you are talking
  • Reading or writing non-meeting-related materials
  • Knitting (yes, one developer used to knit during meetings!)
Until recently, I didn't know that there was even a term for such behavior.

Apparently, it's called "Continuous Partial Attention" and some people consider it not only appropriate, but an effective way to increase productivity. 

Silly me, I just called it rude.

See:
http://www.scottberkun.com/essays/essay51.htm
http://radar.oreilly.com/archives/2005/06/supernova_2005_2.html
http://joi.ito.com/archives/2004/03/29/continuous_partial_attention.html
http://www.inc.com/magazine/20020101/23805.html
http://www.well.com/~neal/
http://www.smartmobs.com/archive/2003/01/03/multitasking_be.html
http://www.nytimes.com/2001/01/30/opinion/30FRIE.html?ex=1143262800&en=787e61f240894d61&ei=5070
http://gilbane.com/blog/archives/2006/03/post.html
http://getreal.corante.com/archives/2005/07/01/linda_stone_at_supernova_continuous_partial_attention.php
http://www.workingforchange.com/article.cfm?ItemID=19475
http://www.roughtype.com/archives/2005/06/continuous_part.php
http://www.stickyminds.com/news.asp?ObjectId=14495
Labels:

April 3, 2007

I Don't Believe in "Best Practices"

Best?

Would you ask your friends:
  • What's the best color?
  • What's the best food?
  • What's the best car?
  • What's the best movie? 
  • What's the best sport?
  • What's the best computer?
  • What's the best Operating System?
  • What's the best programming language?
In the company where I used to work, they liked to talk about Best Practices.  I truly dislike the term "Best Practices".  I don't believe in them.

I believe in practices that have worked out well in the past.
I believe in practices that are good enough that we will likely continue using them.

The dictionary defines the adjective "best" as "better than all others".
So, can you truly recommend a practice that could be better than all others?  All of them?

I don't like it when people ask "What's the best test automation tool?"

Now, you can ask about favorites.  But how can you ask about "best" and expect a reasonable answer?

You can ask about choices that worked out well.  But would someone else's choice necessarily work out well for you?  I know some choices I have made that worked out well on one occasion and poorly on others.

So ask me about good practices.
Ask me about which practices I'd use again, and which practices I'd avoid next time.

But please don't ask me about Best Practices.

see:
    http://www.satisfice.com/articles/good_practice_hunting.pdf
    http://www.satisfice.com/articles/practice.pdf

I've seen this slide before, and have always felt that it reflects some basic ideas that I believe to be true.

The Seven Basic Principles of the Context-Driven School
  • The value of any practice depends on its context.
  • There are no best practices.
  • People, working together, are the most important part of any project's context.
  • Projects unfold over time in ways that are often not predictable.
  • The product is a solution. If the problem isn't solved, the product doesn't work.
  • Good software testing is a challenging intellectual process.
  • Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.
From Cem Kane's web site - http://www.kaner.com/pdfs/ContextTesting.pdf
Labels:

April 1, 2007

Information about Software Quality Assurance Certifications



For those interested in certification, here's a collection of information about some of the more popular Software QA Certifications.


Certified Software Tester - CSTE
  • Administered by the Quality Assurance Institute (QAI)
  • Prerequisites:


    • Must already be working in the QA field
    • Must have significant experience
    • Must have reviewed the appropriate Common Body of Knowledge
    • Must have one of the following


      • A  4 year degree from an accredited college-level institution and 2 years experience in the Information Technology field, or
      • A  3 year degree from an accredited college-level institution and 3 years experience in the Information Technology field, or  
      • A  2 year degree from an accredited college-level institution and 4 years experience in the Information Technology field
    • AND 


      • Must have worked in the information services field within the prior 18 months
  • Must pass a 4 1/2 hour, four-part written examination
  • Examination fee - $350
  • Recertification required every three years
  • For more information see http://www.softwarecertifications.org/

Certified Software Quality Analyst - CSQA Administered by the Quality Assurance Institute (QAI)
  • Prerequisites:


    • Must already be working in the QA field
    • Must have significant experience
    • Must have reviewed the appropriate Common Body of Knowledge
    • Must have one of the following


      • A  4 year degree from an accredited college-level institution and 2 years experience in the Information Technology field, or
      • A  3 year degree from an accredited college-level institution and 3 years experience in the Information Technology field, or  
      • A  2 year degree from an accredited college-level institution and 4 years experience in the Information Technology field
    • AND 


      • Must have worked in the information services field within the prior 18 months
  • Must pass a 4 1/2 hour, four-part written examination
  • Examination fee - $350
  • Recertification required every three years
  • For more information see http://www.softwarecertifications.org/
Certified Software Test Professional - CSTP
  • Administered by the International Institute for Software Testing (IIST)
  • Prerequisites:


    • Must complete at least 10 days of formal study in each of the seven areas of the CSTP Body of Knowledge
    • Must already have been working in the testing field for at least one year
    • Must have had the opportunity to apply formal CSTP training to their job
  • Must pass a written examination for each course taken
  • Course fee - $495 (online) to $695 per course day
  • Recertification required every three years
  • For more information see http://www.testinginstitute.com/cstp.php
Certified Test Manager - CTM
  • Administered by the International Institute for Software Testing (IIST)
  • Prerequisites:


    • Must complete at least 10 days of formal study in each of the seven areas of the CTM Body of Knowledge
    • Must already have been working in the testing field for at least three years
    • Must have been in a lead or management position for at least one year
  • Must pass a written examination for each course taken
  • Course fee - $495 (online) to $695 per course day
  • Recertification required every three years
  • For more information see http://www.testinginstitute.com/ctm.php
ISTQB Certified Tester, Foundation Level 
  • Administered by the International Software Testing Qualifications Board (ISTQB)
  • Prerequisites:


    • None
  • Must pass a 1-hour, 40-question written examination
  • Examination fee - $250
  • Recertification requirements not yet developed
  • For more information see http://www.astqb.org/

ISTQB Certified Tester, Advanced Level 
  • Administered by the International Software Testing Qualifications Board (ISTQB)
  • Prerequisites:


    • Must have five years of full-time work experience in a related field
    • A bachelor's degree or higher from an accredited institution in Computer Science or a related field may substitute for up to two years of work experience
    • Must already hold an ISTQB Certified Tester - Foundation Level certificate
  • Must pass three 90-minute, 40-question written examinations
  • Examination fee - $100 for qualification, $200 for each exam
  • Recertification requirements not yet developed
  • For more information see http://www.astqb.org/

Brainbench  Software Quality Analyst Certification
Brainbench  Software Tester Certification
IBM Rational Certifications
HP Mercury Certifications
Borland Certifications


A comparison of several of the certifications (from the IIST web site):
http://www.testinginstitute.com/CSTPCertificationCompare.pdf
Labels:
Subscribe to: Posts (Atom)