QA
Q&A
Question: What are Release Tests?
Answer: Tests designed to ensure that the code (which has already been thoroughly tested and approved for release) is correctly installed and configured in Production. This is often a fairly quick test, but may involve some migration tests as well.
In my company, we develop applications that we host on the web. When the development and testing cycle is complete, we must release the new version to Production. Part of that release involves a different kind of testing that we call "Release Testing".
Because of our Service Level Agreements with our customers, we often have to perform these releases after hours - when most users would not be using the system. For some of our systems, that means any time outside of weekday business hours. For other systems, that means Sunday evenings. For still other systems, they are required to stay up essentially 24 x 7. Those systems require special architectures that let us make most changes while the system stays running.
Our developers create a "build package" containing all the code, scripts, etc that must be moved to Production, along with the instructions for doing so. The IT team actually make the changes. And QA performs the Release Tests to make sure everything went as planned.
It wouldn't make sense to test all the features of these applications in Production - they have already been tested extensively in the QA Environment (and sometimes in a Staging Environment), and approved for release.
Instead, we test with a specific goal in mind - to check that the new release arrives safely in Production, and that the application is up and running correctly again.
When designing a Release Test, the QAer must consider how she/he will know that the new code is now running in Production, and hasn't been missed. Often, this means quickly checking whatever changed in this release - perhaps new features, perhaps modifications to existing features. In addition, we want to ensure that things that shouldn't have changed remain running as before.
When problems are found during a Release Test, it sometimes means the abandonment of the release, and a roll-back to the previous state.
Some of the attributes of our Release Tests:
- planned
- quick
- often after-hours
- often under time pressure
- sometimes coordinated with customers
- designed solely to check that the code we have been testing all along is safely in Production, without adverse side-effects
- we don't re-run all our prior tests
- sometimes, must test each server in a load-balanced group individually
- check that configurations are correct
- sometimes, check migrations from the prior version
- sometimes, must check that all active users are first drained from the system
- sometimes, must check that the "Under Maintenance" page and process are working correctly first
- pretty much any new bug is a showstopper bug, requiring roll-back
