Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
In my prior blog, Continuous Testing – The Quest for Quality at Speed, I described five tenets and some of the practices for continuous testing (CT) to help those understand what continuous testing is. In my consulting work, I find it necessary to use 15 categories of practices to assess an organizations’ continuous testing capabilities. Given the large number of practices, I am publishing the complete list in three parts.
This article is part one, which covers the following categories of continuous testing practices:
- Continuous testing tenets
- Leadership
- Culture
- Test strategy and plans
- Test management
Continuous Testing Tenets
• The continuous testing principle “shift left” is practiced, in which tests are conducted as early in the pipeline as possible.
• The continuous testing principle “fail early” is practiced, in which tests are arranged so that the most likely problems are found in the early stages of the pipeline.
• The continuous testing principle “fail often” is practiced, in which tests are run frequently and with many different conditions such as different variations of operating systems, production topologies, browsers and production versions of services.
• The continuous testing principle “test fast” is practiced, in which tests are run in quick cycles to get to a verdict as quickly as possible.
• The continuous testing principle “be relevant” is practiced in which tests are focused on the most important tests and results for each purpose and stakeholder.
Leadership
• Leaders establish a definition and vision for software quality.
• Leaders establish a definition for test coverage.
• Leaders clarify how everyone’s roles fit with the CT strategy.
• Leaders establish incentives for CT performance.
• Leaders proactively sponsor quality assurance testing as a strategic element of their budgets and operations rather than viewing testing as a cost to avoid. This includes budgeting time and money for DevOps test training, test resources, test frameworks, test tools, test management and establishing policies for assessments.
• Operational and capital expense budgets are planned together with all stakeholders to ensure all stakeholders agree there is enough opex and capex budget for the organization to accomplish quality goals for the budget period being planned.
• Cost accounting practices ensure the capex and opex expenditures are tracked and made visible to stakeholders.
• Schedules for CT improvement projects are identified and tracked.
• Skills development programs provide individuals and teams opportunities to learn CT practices.
• The performance of continuous quality assurance programs are monitored and made visible to stakeholders.
• Leaders participate in creating and maintaining guidelines for codifying and tagging business risk priorities.
Culture
• Quality is everyone’s concern. Responsibility for attaining high quality software products is shared by everyone on a cross-functional team.
• Cross-functional team members collaborate for testing. This reduces confrontation and friction between people who find failures and those who fix the code.
• Dev teams embrace the creation of good tests and analysis of test results.
• Ops team members contribute to test planning, test creation, test execution and test results analysis.
• Policies for test creation and test coverage are agreed upon by the cross-functional team including Dev, QA, security and Ops team members.
• A test architect is a role within the organization. The role has expertise in CT practices and helps cross-functional teams understand and apply them.
Test Strategy and Plans
• Test strategies and plans are created for each product or service and reviewed by cross-functional teams including development, QA, Ops and security teams.
• Test strategies and plans identify the type of tests, and the extent of testing activities, for each stage in the value stream.
• Test requirements and tasks are included in agile planning procedures for each theme, epic and user story.
• Test strategies and plans define a test coverage standard, test case standards, and input/output criterion for each pipeline gate.
• Test plans identify the test cases and all the supporting requirements to implement and execute the test cases as required satisfying the test strategy and test management goals.
• Test strategies and plans identify the approach for system and acceptance testing.
• Test strategies and plans identify any tests that are to be conducted in production.
• Test strategies and plans identify any tests that are to be conducted for security testing.
• Test strategies and plans identify a target for test automation and manual testing for each type of testing. For example, 100% of all tests in the integration stage and 85% of test in the pre-production stage is a typical goal.
• The test strategy requires that changes to a database or changes to an application that uses a database have tests defined for the database and the application use of the database.
Test Management
• All aspects of test management, such as test planning, test guidelines, test code version management and version management of test results are handled by tools that are well-integrated into the continuous delivery pipeline and any test framework being used for test automation.
• Schedules for all test tasks are identified and tracked.
• New features have a scheduled task to ensure test goals are met for each new feature.
• Each test case has associated attributes that can be used to find tests according to the attributes. For example, an attribute could be type of test and the feature purpose of the test.
• Test environments and resources needed to satisfy requirements to run each test are identified and available programmatically for scheduling purposes.
What This Means
Continuous testing practices are critical to success with DevOps. Many organizations underestimate both the importance and the practices needed to engineer continuous testing. This article is the first in a three-part series that provides a comprehensive list of practices that can be used to understand and assess an organization’s continuous testing capabilities. To learn more about my blueprint for continuous testing, refer to my book Engineering DevOps.
