The State Of Testing 2015

The State of Testing 2015

Back in January, I asked you to help my friend Joel Montvelisky and complete this year's survey he and his company PractiTest ran in association with the magazine Tea Time with Testers. 

Many of you responded, and the results are in!

It's an interesting read. You'll get location, background, experience, and salary information from respondents around the world.

You will find some results that probably aren't surprising ("Testers learn on their own") and some that could be surprising ("Testers are documenting less than in previous years").

You'll hear what Managers look for when hiring testers, and lots more.

Check it out - it's worth your time!

http://qablog.practitest.com/2015/07/the-state-of-testing-report-for-2015-is-out/

---

This article originally appeared in my blog: All Things Quality

	My name is Joe Strazzere and I'm an experienced Quality Assurance professional. I like to lead, to test, and occasionally to write about leading and testing. Find me at http://AllThingsQuality.com/.

Bruce Asks All Things

Ask All Things

Bruce asked a great question:

We have a large number of tests that we would like to automate.  However, it takes time to automate each test, and we have several thousand tests that could be automated. 

Is there an average or maximum number of tests that QA people automate using tools like Selenium or Ranorex?  Thanks for your help.

Sincerely, 

Bruce 

Nice question, Bruce! And one that many folks contemplate when they encounter a large system that needs a lot of test automation.

At one place I worked, we automated over 8,000 tests.

In that particular shop the UI and much of the underlying code was very, very stable. New functionality was added, but it was virtually always added in new instances, and seldom involved changes to existing instances. Thus automated regression tests were able to pay back the investment, and maintenance wasn't overly burdensome.

I know of no industry averages or maximums. Even if there were such averages, I don't see how it would be of much help for your individual case.

Instead, you should consider the full cost of automation in your shop (including initial setup and ongoing maintenance) against the benefits you expect to receive.

Some factors to consider: copyrightjoestrazzere

• Do you have sufficient expertise already on staff to create this automation, or will you need to go outside your organization to get it?
• Do you have time in your schedule to plan, design, develop, test (yes - your automation needs some testing!), document, and deploy your automation?
• How much ongoing maintenance will this automation need?
• How much is the current lack of automation costing your company - both in terms of bugs escaping to Production, and the cost of manual testing you must perform due to the lack of automation?

Many companies choose to start small. Automate only the parts that matter most (perhaps the riskiest or most critical parts of your system), weigh the costs versus benefits, and then decide what to automate next (if anything).

Some companies choose to plan automation for new systems, but only tactically go back and automate legacy systems as time permits.

Other companies choose to outsource the initial automation effort, then bring the maintenance portion in-house.

Let's throw this question to the readers - Do you know of an average or maximum number of tests? Do you know of other factors to consider as you think about automating a large system? Have you faced a large automation effort like this? If so, how did you handle it?

-joe




Do you have questions? Use the new "ASK ALL THINGS" widget over on the right-hand panel. Send me questions about anything:

• about the testing profession
• about test automation
• about bug tracking
• about being a Manager
• about testing and QA jobs
• about quality
• anything!

I'll read through the questions, pick some that not only interest you and me, but questions that I think will interest others. Together we can not only get you the answers you need, but we can provide others with some useful information as well.

Ask All Things!

---

This article originally appeared in my blog: All Things Quality

	My name is Joe Strazzere and I'm an experienced Quality Assurance professional. I like to lead, to test, and occasionally to write about leading and testing. Find me at http://AllThingsQuality.com/.

Perfection Is

What is Perfection?

Perfection

According to Google (and Google knows everything), perfection is..

• Perfection is the enemy of good
• Perfection is a disease of a nation
• Perfection is the enemy of progress
• Perfection is not attainable
• Perfection is in the eye of the beholder
• Perfection is achieved not when there is nothing 
• Perfection is an illusion
• Perfection is my enemy
• Perfection is attainable
• Perfection is overrated
• Perfection is not possible
• Perfection is low standard
• Perfection is boring
• Perfection is subjective
• Perfection is not enough
• Perfection is a perspective
• Perfection is a mistake
• Perfection is a myth
• Perfection is a state of mind
• Perfection is opinion
• Perfection is like chasing the horizon
• Perfection is the enemy of profitability

Can you add to the list?

---

This article originally appeared in my blog: All Things Quality

	My name is Joe Strazzere and I'm currently a Director of Quality Assurance. I like to lead, to test, and occasionally to write about leading and testing. Find me at http://AllThingsQuality.com/.

Like To Find Bugs? Like To Travel? Here's How You Can Do Both!

If you are good at hunting down bugs, you could earn travel miles from United Airlines. In fact, if you are the first to find a particular remote code execution bug, you could earn 1,000,000 miles! copyrightjoestrazzere

The program was implemented in May, and so far two testers have each found 1,000,000-mile bugs.

Jordon Wiens, a software security researcher in Florida was one. The bug he found would have allowed an attacker to execute code remotely on one of United’s systems. In addition to the remote code execution bug, Jordan found another that earned him 250,000 additional travel miles. Should he choose to do so, he can now travel from the United States to Europe about forty-one times, courtesy of United and his bug-hunting skills.

United won't reward you for finding bugs in their onboard Wi-Fi, entertainment systems or avionics. But they do offer miles to testers who find a variety of bugs on United-operated, customer-facing websites such as united.com, beta.united.com, mobile.united.com, mystatus.united.com, smartphone.continental.com as well as bugs on the United app, and other United properties.

The severity of the bug determines the reward:

Bug Bounty payout structure

Severity	Examples	Maximum payout in award miles
High	Remote code execution	1,000,000
Medium	Authentication bypass Brute-force attacks Potential for personally identifiable information (PII) disclosure Timing attacks	250,000
Low	Cross-site scripting Cross-site request forgery Third-party security bugs that affect United	50,000

So if you like to travel, read and follow the United Airlines Bug Bounty instructions, roll up your sleeves, and find some bugs. If you are skilled enough and quick enough, you could be "flying the friendly skies" soon.


Also see:

http://www.united.com/web/en-US/content/Contact/bugbounty.aspx

https://www.washingtonpost.com/blogs/the-switch/wp/2015/07/16/why-united-airlines-is-rewarding-hackers-with-millions-of-free-miles/

http://www.bbc.com/news/technology-33552195

http://www.csmonitor.com/Business/2015/0717/United-Airlines-awards-bug-bounty-Is-it-getting-cybersecurity-savvy

http://consumerist.com/2015/07/14/first-united-airlines-bug-bounty-payout-is-one-million-miles/

http://www.thehoopsnews.com/2015/07/19/6896/united-airlines-hackers-given-million-free-flight-miles-under-bug-bounty-program/

http://triblive.com/usworld/nation/8766543-74/united-wiens-million

---

This article originally appeared in my blog: All Things Quality

	My name is Joe Strazzere and I'm an experienced Quality Assurance professional. I like to lead, to test, and occasionally to write about leading and testing. Find me at http://AllThingsQuality.com/.

Nine Years - And That's All

Nine Years - And That's All Folks!

I recently marked my nine-year anniversary at my current company. Actually, it's now my former company. After nine years, I left.

It was a difficult decision, but for me, it was the right one. It was time to move on to the next phase of my life.

It's hard for me to believe that I was there for nine years. Yet when I look back at all we accomplished, it sometimes seems like far more:

• When I started, there was no real Quality Assurance Team. Whatever small bit of testing occurred was being performed by Product Management folks in their spare time. Since then, we created a terrific team in the US, augmented by some good contractors, and a small team in India as well. copyrightjoestrazzere
• Bugs were not being tracked in any central system. There were a few emails floating around, and an occasional spreadsheet, but no place where people could go to find the status of bugs. Most recently, we used Bugzilla, and people grew tired of me asking "Do we have a bug report for that?"
• Lots of people came and went over the past eight years. Initially, the biggest change was the prior CTO being replaced by my boss.  Since then, many other folks left.re
• We changed a significant portion of the infrastructure behind most of our applications. It became far more scalable and sustainable, although we continued to make changes.
• We formalized many of our development and testing processes, and created the necessary processes where none existed before.
• We went from fighting fires every day, to a much more stable, dependable set of systems. Where before many of our systems needed manual, hands-on attention every day, most ran in a much more automated fashion.
• The product lines changed over time. We weeded out some products that were single-customer, poorly funded products. We created some new products, and retired others.
• A few years ago, we were purchased by a much larger corporation. It wasn't all bad, and it wasn't all good. The volume of big-company administrivia started out small, but increased, and it continued to increase.
• My QA Team was reorged a few years ago. At that time, my current boss reported to someone in the corporate office, rather than the local General Manager. She also had responsibility for more than just our local division. That meant we had even less contact with my boss than before.
• As part of reporting up into corporate, we were required to use all of the formal corporate time-reporting, project management and metrics systems. For me, it was a lot of time spent on administrivia, rather than more productive work. I tried hard to minimize the impact on my team, but I couldn't eliminate all the overhead.
• We recently completed a massive project to move our production infrastructure into the corporate facility. We purchased new hardware, new software, database upgrades, etc, etc. We embraced new processes for security, administration, installation, and support. And of course we often "improved the applications" as we migrated them. With almost all the variables being changed at the same time, this was a big task for everyone involved, and a very big testing task. It was "interesting", and a huge drag on our time for building and testing revenue-producing applications.
• We experimented with a few Agile projects. They took many wrong turns, took longer than anticipated, and didn't end up in viable products. Still, we learned a lot.
• Last year, my QA Team was reorged yet again. This time, I reported directly to someone in the corporate offices in New York, rather than into the local Development group. The new boss had no real background in QA at all - he was primarily concerned with Governance. That was rather different than any other place I had ever worked. No time for any testing - for me it was basically all project management, all the time.
• Eventually, it became clear that the role of a QA Director in this company no longer fit with what I believed in or what I wanted to be doing.

Lots of work, lots of changes. All in all, it was a pretty good nine years. But now that company is behind me.

I'll really miss all of the local people I worked with over the years - they are a great bunch of smart, hardworking people who care about quality. I'm sure they will all continue to be successful.

But I won't miss the administrivia, the many processes, the long string of estimates that were required, the ever-growing number of metrics, all the purported "Best Practices", and the seemingly-endless series of reorgs.

I'm going to take a step back for a while, to relax a bit, to evaluate things, and to spend more time with my grandchildren. I have a lot of projects I want to try, a lot of reading to do, and a lot of writing ahead of me (I hope). Stay tuned!

---

This article originally appeared in my blog: All Things Quality

	My name is Joe Strazzere and I'm an experienced Quality Assurance professional. I like to lead, to test, and occasionally to write about leading and testing. Find me at http://AllThingsQuality.com/.