The continuous demand market for flawless applications is being addressed through advanced software development techniques along with continuous delivery. As such, many organizations adopted the shift-left approach and agile best practices—which suggests the motto, “test early, test often,”—through which they get to imbibe QA practices and testing early in the software development pipeline. That also includes continuous testing for many organizations.

Continuous Testing in DevOps

Continuous testing today is the most preferred practice by leading software companies. Continuous testing implies testing through the whole software development pipeline, from product design to product release, by integrating the complete product cycle. Even after the product release, companies test to troubleshoot customer complaints to further improve the product.

Continuous testing in DevOps involves encouraging the development engineers and operations to practice testing as well, and also makes them a part of the whole cycle instead of being a separate team. Continuous testing involves testing not just from an earlier stage but also through each and every single process. Its goal is to automate almost all the processes as possible.

A Templated Approach to Continuous Testing in DevOps

A perfect approach to continuous testing in DevOps should be a right balance between manual and automation testing, and to achieve it, organizations must practice virtualization to plan out the whole product development pipeline successfully. Many organizations are not yet ready to risk it all with continuous testing, as big errors are still a possibility if they don’t really know how to execute it. But, errors in the continuous testing method can be easily rectified as it is practiced through each process.

Errors are common when it comes to product development; hence, the right set of tools to automate testing can be critical to finding and rectifying errors, since manual testing can miss certain errors. Manual testing also takes a considerable amount of time, which is a huge drawback. A well-planned but carefully structured templated approach can be practiced by companies to make a full transformation toward continuous testing.

Continuous testing is achieved through imbibing multiple use cases over certain parameters to determine the strength of this methodology. Some of those parameters that would prove to be a sure-fire way of determining whether continuous testing is worth it are distribution, technology, scale and load, environments and security and release.

Checking the Parameters

The parameters to completely check a dimension such as continuous testing can be done to create readings based on elimination and weighted scoring to derive results from a use-case scenario. Let us see a brief application of running a hosted public-facing service through these parameters:

Distribution – The distribution areas and the geographical location of this product are considered.

Scale and Load – The number of test cases (3,500 automated cases in this scenario, by way of an example), the suites that are used based on the browsers, cloud test platform and the numbers of users are considered in this category.

Environments and Security – Varied device uses, cross-browser testing, secured cloud environment and approval of releases are considered in this parameter.

Release – The number and frequency of release and the technology stack are considered in this parameter.

The tools that are used for this scenario are Selenium web driver, Selenium GRID, Jenkins, Sauce Labs, Sauce On-Dand Plugin.

The process through which checking the parameters is done is by triggering automated test cases from Bitbucket automation repository using Jenkins during deployments, and the execution of automated scripts in Sauce Lab cloud, as parallel threads are done by interfacing Sauce Lab Plugin with Jenkins.

The control mechanisms first include testing through smoke, regression and integrated automated tests execution. Then, the test evidence for all of the test cases is captured. Later, as the tests are completed and the screenshots which are captured by automated tests are validated, UAT sign-off is done.

Some parameters for test automation framework are compared within Selenium and tests completed over various categories, their scores are compared and then they are passed onto the test environment. Some suite, regression suite and integration test suites are carried out during Dev and QA stages, later when the changes from these two environments are deployed. Smoke suite is carried out during pre-production, editorial and the production stages. Finally, the proven results of these stages are derived successfully.

The proven results of this use-case scenario through these stages are:

• Parallel testing visibly reduced regression hours from 150 hours to 15 hours.
• Enhanced test coverage matrix addressed varied device and OS fragmentation, as well as browser combinations.
• Quality gatekeeper for code promotion to production.
• Reduced regression issues in production.

Conclusion

The example of going through such a use-case scenario helps us better understand how continuous testing can be integrated gradually into a product development pipeline through careful observation, planning and deployment through such parameters.

Engrafting continuous testing in a DevOps eEnvironment needs careful execution to avoid errors and future defects. And with the right set of tools and a perfect execution, it can be achieved to successfully experience faster time to market and reduction in time and cost.

For more information on how a templated approach can be applied in different scenarios, watch this webinar.