Continuous integration (CI) calls for code to be always working. Continuous delivery (CD) and continuous deployment mean that we must be able to roll out a working configuration to production at any time. As DevOps becomes the norm, bugs in production are now a developer’s personal problem. Developers (and their managers) want to know immediately if code changes they are working on can break something.
What is Continuous Testing?
The only guaranteed way to prove code and configurations are working is to test them. That’s why we need continuous testing. In organizations making this transition, performance engineers and test automation experts shift from minding a backlog of test-building work, to facilitating a democratization of testing. Everybody builds their code and everybody tests their code, continuously.
Continuous testing is becoming the “new normal.” In this modern and agile testing approach, developers are equipped with low-friction testing capabilities and tools and are expected to unit test any new or updated code. With testing shifted all the way left to developers in pre-production, and all the way right to post production monitoring, organizations are just seconds away from proving code is ready and working to acceptable service levels.
Although continuous testing is becoming a more standard practice, it certainly presents unique challenges for execution.
Here are five steps to help any company successfully implement a continuous testing policy:
Define Tests Early
Lack of clear requirements can result in wasted time and delays. Behavior-driven development (BDD), acceptance test-driven development (ATDD) and model-based testing using tools should be used so that all requirements are clearly documented and communicated.
Test cases need to be clearly defined and test scripts created ahead of time, to enable continuous testing of code at all stages of production.
Optimize Testing Processes and Test Coverage
Save time, money and resources by testing only what you need to test. Visual models allow various paths to be explored and optimized so that only your tests provide maximum coverage, without unnecessary testing.
It is also recommended to import test cases, remove duplicates and distribute and allocate optimized test cases in existing tools.
Shift-Left Your Testing (and Shift It Right as Well)
What do we mean by shift-left testing? Tests should be run earlier in the development cycle. Developers test as they go, with test automation including functional testing, performance testing, monitoring and security testing. This requires developer-friendly tools for easy adoption and use.
By shift-right testing, we continue to test through production and post-production, including continuous monitoring. Gartner recently released a report on the shift-right testing, calling it a “must” for adopting DevOps practices.
Provide Complete Test Environments
The ability to provide virtualized test environments is critical to achieving continuous testing. Eliminate blocks and reduce wait times by providing complete test environments on demand, with dev-friendly tools (as code, CI/CD integrations, supported open source). These environments should include test data on demand to ensure that teams can perform comprehensive tests with production-like data. These environments should be ephemeral; ready on demand, decommissioned after use.
Get the Right Test Data
The inability to get robust test data causes significant delays in many application release cycles. To accurately test new functionality, test data should be as close as possible to what the application will encounter in production. If the test data lacks certain real-world characteristics (i.e. actual fields, data specifications, negative scenarios, etc.), the tests are unlikely to find many potential issues or break the application where there are weak points. Understanding data models in order to extract the right data is a special skill in and of itself.
Production data is clearly the best data to use for testing, but data privacy regulations and team silos often restrict its availability. The answer to this dilemma is a robust test data management tool that lets you copy production data and mask it to protect sensitive information, while maintaining the characteristics that make production data desirable for testing.
Remember, continuous delivery and continuous testing are processes that can take time to implement successfully, but teamwork, continuous feedback based on data and the right tool set will help you on your journey. You can read more about implementing a continuous testing policy here.