Book: Surviving the Top Ten Challenges of Software Testing

"Surviving the Top Ten Challenges of Software Testing: A People-Oriented Approach"

William E. Perry and Randall W. Rice

While looking through the Common Body of Knowledge for the CSTE Certification of QAI, I saw that this book was part of the suggested reading list.

It was written by William Perry and Randall Rice (both of whom I had read before, and both of whom work for QAI), so I thought I'd buy a copy and give it a shot.

It's not bad as far as listing testers' challenges, but for me, it doesn't do enough to indicate how to actually solve those challenges.  I don't necessarily agree that these are the top ten challenges, nor that they are ordered correctly.  But if I did, I'd want more practical help in solving them.  Still, it's a good high-level overview of these solutions.

The book has a rather unusual format - each of the ten challenges is discussed in a chapter, using the same format for each chapter.

Here's an outline:

Challenge #10: Getting Trained in Testing

• Overview
• State of the Practice
• Impact on Testing
• Solutions to the Challenge
  
  • Raise Management Awareness of Testing
  • Make Time for Training
  • Develop Your Own Skills
  • Certify Your Testing Skills
• Solution Impediments
  
  • What if my management will not support my training needs?
  • What if I do not have enough money to attend a testing seminar or conference?
  • What if I do not have enough time for training?
• Guidelines for Success
  
  • Set personal goals
  • Seek management's assistance in building your testing skills, but don't wait for it
  • Invest in yourself
  • Strive to add value to the testing effort
• Plan of Action
  
  • Get management on your side
  • Develop your own skill-building goals and objectives
  • Never stop learning!

Challenge #9: Building Relationships with Developers

• Overview
• State of the Practice
• Impact on Testing
  
  • low morale
  • slow or stalled projects
  • inefficient testing
  • poor physical and mental health
  • destroyed relationships
  • termination of employment
  • project failure
• Solutions to the Challenge
  
  • Adopt a Win-Win Approach
  • Widen Your View of Testing
  • Move from "Us versus Them" to "Us and Them"
• Solution Impediments
  
  • How do we get everyone on the project involved in testing?
  • What if people don't want to be involved in testing?
  • How do we criticize constructively?
  • How do I get management support for testing?
• Guidelines for Success
  
  • Build the test team so that participants from a variety of disciplines are involved
  • Make win-win the only way you operate
  • Keep the lines of communication open
  • Keep working on yourself
• Plan of Action
  
  • Work on yourself first
  • Study the works on human interactions and potential
  • Keep a personal journal
  • Make a conscious effort to show others in the organization the value of testing and what it can mean to them
  • Involve others in the testing effort

Challenge #8: Testing Without Tools

• Overview
• State of the Practice
• Impact on Testing
  
  • Manual testing is imprecise and unreliable
  • Manual testing is boring
  • Manual testing is labor intensive
• Solutions to the Challenge
  
  • Educate Management on the Use of Test Tools
  • Perform a Tool Survey
  • Define Your Requirements
  • Perform a Cost/Benefit Analysis
  • Investigate Tools Available
  • Integrate Test Tools with an Effective Testing Process
• Solution Impediments
  
  • What if my management does not see the need for test tools?
  • What if I can't get funding for test tools?
  • What if not tools exist for my environment?
  • What is other people in the organization do not want to use test tools?
• Guidelines for Success
  
  • Secure management support for acquiring test tools
  • Use a team approach when selecting a tool
  • Make good use of what you have
  • Maximize tool use with people and processes
  • Use the right tools
  • Make tool usage mandatory
• Plan of Action
  
  • Identify stakeholders
  • Lay the groundwork by raising test tool awareness among the stakeholders
  • Identify or build the testing process
  • Get approval and initiate a project to build a test toolbox
  • Form a team to identify which tools are already in place and to specify which tools are needed
  • Use the tool selection team to identify the tool requirements
  • Identify the selection criteria
  • Perform a tool search using all available information sources and tool directories
  • Contact the qualified tool vendors for detailed information
  • Evaluate the finalists
  • Select the tool
  • Develop the implementation plan, including integration of the tool and the testing process
  • Train the staff in the new process and in how to use the tool to support the process
  • Continue improving the processes
  • Continue training people as they are hired and as refresher training is needed

Challenge #7: Explaining Testing To Managers

• Overview
• State of the Practice
• Impact on Testing
• Solutions to the Challenge
  
  • Identify the Stakeholders at the Management Level
  • Raise Awareness of the Testing Function
  • Network with Other Organizations to Learn How They Deal with Management
  • Establish a Testing Charter to Define the Purpose of testing in Your Organization
  • Define Measurable Testing Objectives
  • Dedicate a Manager of Testing Who Understands the Issues and Challenges
  • Make a Testing Process
• Solution Impediments
  
  • What if management is satisfied with the status qui, since we've done well in the past with ad hoc testing?
  • What if we can't find a champion for testing?
  • How do I find other organizations with which to network and benchmark?
• Guidelines for Success
  
  • Be patient.  Raising management awareness of testing can take a long time
  • Quality is ultimately management's responsibility
  • Focus your message to management on cost and time
  • Meeting the schedule is not the only measure of success
  • Measure and publicize your efforts
  • Marketing is part of your job.  Learn how to market well
• Plan of Action
  
  • Develop a testing charter
  • Identify the stakeholders
  • Identify your target market
  • Define your strategy and objectives
  • Assess your current condition
  • Start making the message
  • Keep making the message

Challenge #6: Communicating with Customers - And Users

• Overview
• State of the Practice
• Impact on Testing
• Solutions to the Challenge
  
  • Teamwork
  • Communication
  • Continuous Involvement
  • User Acceptance Testing
• Solution Impediments
  
  • What if the end-users just don't want to be involved in building or testing the system?
  • What of end-users have an adversarial attitude toward the developers?
  • What if the project management rejects the idea of customer and end-user involvement?
  • What if the end-users do not understand technical issues or know how to perform acceptance testing?
• Guidelines for Success
  
  • Communicate tactfully
  • Lighten up a little. Life will still go on after the project is over
  • You can't please everyone
• Plan of Action
  
  • Start educating the development management and end-user management on the importance of identifying the customer and end-users
  • Include customer and end-user tasks in your system development and testing methodologies
  • Pilot the process
  • Training the users in performing their tasks
  • Measure the results

Challenge #5: Making Time for Testing

• Overview
• State of the Practice
• Impact on Testing
  
  • Reduced Test Coverage
  • Increased Risk of Regression Defects
  • Fatigue, Burnout, and Low Morale
• Solutions to the Challenge
  
  • Control the Scope of Testing
  • Control Management Expectations
  • Base Test Cases on an Independent Set of Criteria
  • Perform Risk Assessments
  • Reuse Your Testware
  • Estimate the Testing Effort Based on Measurable Criteria
  • Use Automation
• Solution Impediments
  
  • What if my management refuses to acknowledge the need for better estimates, test tools, or any of the other solutions you suggest?
  • What if I do not have measurable criteria, such as requirements, to serve as the basis of my tests?
  • How do I control the scope of a test if the scope is constantly changing?
  • What is the common ratio of testers to developers?
• Guidelines for Success
  
  • Remember: Work expands to fill available time
  • Have a plan - and plan for the unexpected
  • Control your scope
  • Test early and often
  • Use techniques that leverage your time and resources
• Plan of Action
  
  • Define a test process that includes risk analysis
  • Develop test standards and templates that can be reused
  • base test estimates on measurable criteria, such as cases or requirements to be tested
  • Build test cases to validate requirements or business processes, not random combinations of software functions
  • Continually educate management that testing should be measured and estimated by fact, not guesswork or deadlines
  • Keep a history of test measurements to help estimate future projects
  • Continually streamline the testing process to make it more efficient and less time-consuming
  • Consider using an automated test tool for repetitive tests once the test process has been defined and you know what you need to test

Challenge #4: Testing What's Thrown Over the Wall

• Overview
• State of the Practice
• Impact on Testing
• Solutions to the Challenge
  
  • Get Management Support to Define Roles and Responsibilities
  • Establish Standards and Processes for Testing
  • Establish Ownership and Accountability at the Developer Level
  • Train Developers to Be Excellent Testers
  • Improve Communication Between Developers and Testers
  • Measure and Refine the Processes Continually
  • Establish Ground Rules
• Solution Impediments
  
  • How do I overcome the skepticism over new process initiatives when past efforts have failed?
  • How do I convince management to take the lead in deploying processes?
  • What if the developers don't want to become better testers?
• Guidelines for Success
  
  • People must be accountable for the products they produce
  • Management must build a quality culture
  • Test processes must be supported by test tools and test standards
  • Don't let your failures deter you from trying again
  • Process deployment is a function of organizational maturity
• Plan of Action
  
  • Get management support for establishing clearly defined roles and responsibilities
  • Establish test standards and test processes
  • Build ownership and accountability into the test processes and standards
  • Train people in how to apply the test processes and standards
  • Continually measure and refine the processes

Challenge #3: Hitting a Moving Target

• Overview
• State of the Practice
• Impact on Testing
• Solutions to the Challenge
  
  • Rework of Testware
  • Regression Testing of Previously Tested Software
  • Backlog Created by Rapid Change
• Solution Impediments
  
  • How do I test when my organization is changing at a chaotic rate?
  • How do I find the time to perform impact analysis where changes occur so fast in my organization?
  • What if users become impatient with waiting for releases where they otherwise could have more frequent revisions? 
• Guidelines for Success
  
  • Change will always be a part of our lives, so be prepared for it
  • Change is occurring at an ever-increasing rate, so be aware of the risks
  • One small change can cause major problems
  • Where change is concerned, control is everything
• Plan of Action
  
  • Analyze your situation and find out where changes originate
  • Establish a change control process that will manage changes
  • Make your testing processes change-tolerant by incorporating reuse and modularity
  • Investigate how automated tools can help in your environment
  • Explore the concept of scheduling changes in releases to control the process
  • Add regression testing to your testing process
  • Measure the number and impact of defective changes
  • Use the information from the measurements to improve the maintenance and testing processes

Challenge #2: Fighting a Lose-Lose Situation

• Overview
• State of the Practice
• Impact on Testing
  
  • Keeping an Organization at a Low Level of Process Maturity
  • Trivializing and Undermining the Testing Process
  • Demoralizing the Testers
  • Fostering a False View of Testing
• Solutions to the Challenge
  
  • Communicate the Role of Testing to the Rest of the Organization
  • Identify What Testers Can Reasonably Accomplish
  • Set and Manage Customer Expectations of Production Software
• Solution Impediments
  
  • What if communication of the testing role falls on deaf ears in our organization?
  • What of management's expectations remain inordinately high, even though we've tried to explain the limitations to testing?
  • How do I set and manage expectations when our customers expect zero-defect software?
• Guidelines for Success
  
  • Understand and communicate the role of testing and how it fits into the entire organization
  • Testing is more than one person's or one team's responsibility
  • It is incumbent on management to build a culture of trust and appreciation, not fear and blame
  • Our customers are our partners.  They have a part in determining the quality of software they receive
• Plan of Action
  
  • Develop a testing charter that defines the role of testing on your organization
  • Publish the testing charter so that everyone in the organization understands what testing can or cannot accomplish
  • Define testing processes and standards that facilitate the objectives outline in the testing charter
  • Deploy testing processes and standards
  • Conduct training for the organization, including end-users and customers, on the details of the testing charter, testing standards, and testing processes

Challenge #1: Having to Say No

• Overview
• State of the Practice
• Impact on Testing
• Solutions to the Challenge
  
  • Standardize Test Reports
  • Make Test Reporting Part of the Testing Process
  • Manage Your Audience's Expectations
  • Use creative Reporting Techniques
  • Focus on the Facts
  • Be Truthful
  • Document Your Tests
  • Build a Mature Culture
• Solution Impediments
  
  • What if my management only wants to hear good news?
  • How do we change the perception of testers as being negative?
  • How do I find the courage to say the system is not ready for production, even if my job is at stake?
  • How do developers decide whether or not to report defects found in a work in progress?
• Guidelines for Success
  
  • Tell the truth, even if there are negative consequences
  • Stick to the facts
  • Deliver a solution or solutions for each problem reported
  • Make test reporting part of the testing process
• Plan of Action
• Assess your current method of reporting test results and see if improvement is needed
• Develop test reporting standards
• Make test reporting part of the testing process
• Survey management, developers, customers, and end-users as to what information they can expect from testing they would find useful
• Set expectations of management, developers, customers, and end-users as to what information they can expect from test reports
• Explore creative ways to present test results using graphics, such as pie charts, bar charts, and the like
• Improve the test reporting process by continually surveying management, developers, customers, and end-users to measure your effectiveness