The practice of continuous delivery has been with us for many years, but it has started to see real adoption in the last couple of years. Continuous delivery is about automating and simplifying the process of delivering new features to the market at any given time, accelerating delivery and helping bring Dev and Ops together to make them jointly accountable for the release process.
As continuous delivery is helping to break down barriers, culture and bottlenecks in the delivery pipeline, a new practice has evolved to help teams solve this most daunting challenge of the delivery pipeline: Quality.
The practice is called continuous testing, which can be defined as:
- Testing your change in an efficient way (so your tests are completing faster),
- In an optimized way (so that you find issues as early as possible in the cycle),
- Based on risk and priority (so you test what is important to your users and the business).
Some may think of this as equivalent to test automation, but that premise would be wrong.
Automated testing tools can execute tests repeatedly, provide reports on testing outcomes and show comparisons with previous test runs. This is a critical foundation for a continuous testing practice, but just a starting point.
In a continuous testing practice, testing is not done just in the “testing” phase and not only by test teams. Testing is performed at the level of the build (for example, Unit Test and Security Test) and does not stop after the change has been deployed into the production, where testing can ensure great customer experiences. Here are some other key aspects of continuous testing.
Continuous Testing Automates Test Environments
Let me clarify what I mean by environment. Basically, you want to replicate the real-world situation much as possible for testing. A test environment contains the test cases to be run, the right test data and services available, and the right application version deployed in the right environment. The juxtaposition of these elements of an environment must come together automatically to ensure a continuous testing practice.
Continuous Testing Amps Up Throughput
A key premise of continuous testing is that you are continually maximizing efficiency. This is a must to keep up with the velocity and frequency of today’s modern software factory.
You are doing testing earlier in the testing cycle (shift left) where problems are easier to fix and less expensive. You are doing more testing in parallel. You are automating the promotion and feedback during testing stages. And you are ideally able to prioritize the testing order based on the risk profile of the code so that you are testing higher risk areas first. All of this enables you to find issues quickly and roll forward if possible or roll back seamlessly.
Continuous Testing Applies Intelligence
Finally, continuous testing means you have the knowledge at your fingertips about the risk profile of a release so you can make more informed decisions. You should be able to see what was changed in the code and the implications. For example, was there a small change in the UI or a complex backend change that affected integrations and could break the APIs for the interface?
You also need a detailed view of the quality level of the release, including which tests have failed, which tests have not been executed, whether you have tested the new or the risky areas. All this intelligence will help you to understand the quality profile of a release and make faster decisions about whether additional tests are needed or if a release is ready to go to production.
Steps Toward Continuous Testing
There is no perfectly prescribed way to put a continuous testing practice in place for your organization. For example, you may find that your biggest bottleneck of the moment is managing test data, so you should tackle that first. But here are five steps critical in your adoption of continuous testing:
Step 1: Implement best-of-breed test automation tools—Functional, GUI, Performance, Security, etc.
Step 2: Automate test environments with the right test data and virtual services to ensure real-world, always available testing.
Step 3: Automate test requirements (you’ll find manual methods can no longer keep up).
Step 4: Automate the automation—Connect your testing toolchain and stages. Automate promotion from one stage to the next. Automate feedback loops end to end.
Step 5: Get smarter—Automatically align the right tests to the right app version. Automate testing sequence by risk profile of the code. Build release health profiles.
About the Author / Uri Scheiner
Uri Scheiner is a senior director of product management in the Continuous Delivery Business Unit of CA Technologies. He was previously a key architect and responsible for the product road map and vision at a leading release automation company Nolio, which was acquired by CA in 2012. Uri started his career as a software engineer at Mercury Interactive (acquired by HP in 2006) where he was responsible for developing several products for the DevOps and APM markets.
