Of the tried-and-tested frameworks that allow enterprises to assess DevOps in their organization and how it can be improved, the CALMS model remains particularly useful.
CALMS, which stands for Collaboration, Automation, Lean, Measurement and Sharing, is particularly helpful for analyzing an organization’s DevOps structure, and ultimately, its utility in any organization. The CALMS framework covers all stakeholders within DevOps, including business, IT operations, QA, InfoSec and development teams, and how they collectively deliver, deploy and integrate automated processes that make business sense.
“DevOps as a technique is slowly but steadily maturing, so it’s good to see general assessment methods like the CALMS model,” Holger Mueller, an analyst for Constellation Research, said. “The CALMS model gives a good frame of reference to compare the maturity of a DevOps team, and as such, is invaluable for assessing the state of teams for the transformational change that goes with it.”
Applying CALMS
Here are a few ways CALMS can be applied to DevOps for enterprises either thinking about shifting to a DevOps structure or seek to make improvements to an existing deployment.
Culture
It is long-established that technology adoption must be to serve a business need rather than investing in technology for its own sake. This shift in mindset applies to how the “culture” component of CALMS supports how the forecasted ROI associated with automating a process must be championed across the DevOps team. This mindset is especially important for getting the non-technical business teams onboard, Mueller said.
“DevOps must have the backing from the business executives. That’s a crucial factor in the culture part,” he said. “One thing software teams can do is use successful software deployments to help convince the business executives in DevOps to make the investments. Those are the kinds of tribal events that every DevOps organization needs.“
Automation
Netflix is well-known as an exemplary case study for DevOps automation. Among other things, it can deploy code and software updates in a matter of minutes across its entire massive infrastructure, many times throughout the day. But Netflix’s example of automation does not take into account what can go wrong at firms that want to follow Netflix’s example when automating their software deployments and updates, Mueller said.
“The risk is always deploying too fast and you end up deploying bad code,” Mueller said. “Automation, of course, needs to take this into account and compensate. But not everybody can do that.”
Lean
Once completed, it is relatively easy to document cost savings by eliminating resources. Within the CALMS framework, the corresponding “lean” component in DevOps is thus relatively easy to assess, Mueller said.
The good thing about DevOps, Mueller said, is companies can come to a consensus ahead of time regarding what they consider to be a lean operation.
However, he added, “There is always the risk of being so lean you begin to err on the QA side.”
Measurement
The success of software deployments is relatively easy to measure once completed. However, attempting to measure future deployments as part of a forecast model is, of course, much more challenging.
“It can be surprisingly tricky to measure how good DevOps is ahead of time,” Mueller said. “Measurements should thus allow errors to be pinpointed  quickly during the post-deployment stage to fix the code if needed very quickly.”
Sharing
Sharing, in many ways, overlaps with culture. This is where assessing whether all of the different teams within DevOps are working together and communicating comes into play.
“Thanks to  DevOps, the days of the lone developer are long gone. But sharing is not always easy,” Mueller said. “Even within separate teams, it is easy to fall into the trap of wanting to get your own work done without necessarily thinking about the team as much as you should.”
The End Result
While applying CALMS to assess DevOps never hurts, the framework is still not as important as one of the main objectives of DevOps—namely, using technology to make organizations’ operations more agile and reactive to customer needs, Mueller said.
“At the end of the day, assessment methods such as CALMS are great, but they are never as important as focusing on implementing next-generation applications,” he said.
— B. Cameron Gain