One of the points of DevOps is to streamline development and focus on what is important. Things that don’t raise to a high enough level of importance end up in the category of backlog, and their tests are normally disabled or marked as “ignore results” in CI/CD tools.
With CI/CD, this is a good step in moving forward, because you have a list of the things that you haven’t gotten to yet, even if that list is not currently part of validation. The problem is that it can create the very kind of cruft that we adopted CI/CD to get rid of. Many organizations keep these tests in the queue for an indefinite amount of time on the assumption that they will come in handy when priority allows for time to address the issues at hand.
It Isn’t Simple
The problem is, of course, that some of those tests will never—and I do mean never—be resolved. Those are cruft, sitting in the way of productivity and cluttering up the build system. It can be a corporate decision that a given bug fix/feature request will never be implemented, but more often than not it is simply that the priority will never be addressed because there are always new features, more important bugs, etc.
Purists will already have started sweating while reading this, because the goal is supposed to be to address every single test in the system, resolve every single bug in the system, and implement every change request that made it far enough to have tests written for it. But we don’t live in a purist world. We never have. I’ve argued with purists of every possible stripe in my time in high-tech, because reality isn’t pure. It’s interruptive and ugly and prioritized.
So we have backlogs. And we have items in the backlog that won’t ever be addressed.
Why Old Tests Matter
The real problem with tests that are backlogged for more than a few sprints is, of course, the changing code base. As an abstract example, you have a dialog that asks the user for some input. The tests assume the input values that are designed in at the time the test was written. Six months later, a higher-priority test comes through that changes what fields are available on the dialog, and what data is collected. The original test is marked as ignored, so when it fails that doesn’t impact things. But now the test is invalid. And still in your system. Multiply these two requests by hundreds or even thousands, and you have the same type of flotsam that DevOps was designed to resolve, except now it is part and parcel of the CI/CD system.
Clean Up, Acknowledge and Move On
So what is a productive team to do? Have a frank discussion about your backlog and what will/won’t ever be implemented. Business leaders can agree that some items just never will raise to the correct level of priority, and you can identify tests that have aged out of usefulness while sitting in the backlog.
Set up a process to review them occasionally, to run parallel to the process that updates tests when assumptions change—just not as frequently.
Then document the tests that won’t ever succeed, and delete them. They don’t belong in version control, even as documentation, because eventually and inevitably, they will be inaccurate representations of the system. And that can create confusion.
Of Course It’s DevOps
This is an extension of all that DevOps did to begin with: eliminate wasteful activity and non-productive items on the list of to-dos. You prioritized once and cut with a knife, this is the same process, operationalized for CI/CD. And it’s necessary.
A slight change, another set of choices that are no harder than the others you’ve faced, and the code base (in particular the test base) is cleaner.
Leaving you more time to keep rocking it.
