What Agile means to your typical operations staff member is, “More junk coming faster that I will get blamed for when it breaks.” There always is tension between development and operations when something goes south. Developers are sure the code worked on their machine; therefore, if it does not work in some other environment, operations must have changed something that made it break. Operations sees the same code perform differently on the same machine with the same config, which means if something broke, the most recent change must have caused it … i.e. the code did it. The finger-pointing squabbles are epic (no pun intended). So how do we get Ops folks interested in DevOps without promising them only a quantum order of magnitude more problems—and delivered faster?
The problem is exacerbated when the topic of DevOps comes up, as it is most often a developer doing the talking. And the topics under conversation are things every developer would have substantial interest in and pretty much nothing that would interest the operations part of the equation. Many Ops folks feel as though they have been dragged into this DevOps phenomenon by osmosis, without their consent, and now they are supposed to be reaping huge benefits from it (by magic, as near as anyone can tell). Developers working faster is traditionally not something Ops folks look forward to; rather, it may be their worst nightmare. So, if the advocates of DevOps all tend to be developers, how do we get operations folks to become our evangelists as well?
QA Impacts on Evangelism
It begins with quality—not the ethereal concept of quality, but rather the specific responsibility for who owns it. Finger-pointing be gone; developers now own QA. If the software development lifecycle (SDLC) environment is broken, developers own it. Ops can sit back, toast marshmallows and enjoy the show. Developers should be including whatever configuration file changes their application needs as part of the deployable package. So nothing is done manually anymore. This means no one could forget something, as machines do not forget; they do what we tell them. If by chance, some random operations staffer decides to doink with a config variable, the deployment software should simply overwrite the manual change with what is in the deployable package created by the developer. Even malicious changes can be overwritten, so now, all change is included in the development class environment in the one and only deployable package. That is the only package that moves forward.
If developers own the QA, there is now a shift in the force: The operations team has a new role, one of coach and adviser, instead of doer. The software now is the doer, the developers are the packager and the operations staff are the witnesses of how it works. If it breaks, we all know who must fix it (and who is to blame). But once blame has been permanently affixed, something else changes, namely the culture. Now that the operations team does not take the blame, it remains interested in seeing its operations run without error. This allows operations to take an active “helping” interest in getting things fixed faster. It does not really matter who is to blame, once that argument has been settled; now it only matters if they can make something work. Harmony between factions begins to emerge.
Scalability Impacts on Evangelism
It continues with scale. Self-service is the nirvana of your average operations staffer. If the responsibility for provisioning can be moved into a push-button self-serve automation, nirvana will have been achieved. Imagine the offload of repetitious work tasks that consume the life of your average operations staffer. Instead of setting up (and taking down) environment after environment of one class or another for one application or another, requesters can all do it themselves with the push of a button. Even the accounting or financial impact is something the accounting team can guide the user on; operations does not have to worry about the cost of its operations anymore (or, at least, worry less). If too many environments are being provisioned, the accounting team has the power to change the rules and keep it from happening automatically without operations having to be the bad guy.
At this point, Ops can resume a focus on something it tends to enjoy more: monitoring. Ops staffers enjoy being able to provide real-time reporting on where things stand at any point in time. It is even more rewarding to provide more meaningful information such as who is using this application, what users are doing with it and when are they performing that activity. To monitor that kind of information, operations must partner with developers at the outset of a project to determine what kind of application logs are needed later for monitoring. Without the daily barrage of mundane provisioning tasks, an operations engineer is free to focus on more rewarding (translate: higher skill-level) tasks. The days of changing tapes for the backup may have reached their end (or pick your similar set of mundane tasks which DevOps automation can fully eliminate).
And Speaking of Monitoring
It merges into the future. How do you create an operations staffer as a DevOps evangelist? Start by exploring potential that only DevOps can enable. All those nagging alerts that are created when an environment goes down while a deployment is made … all of that could be turned off by the DevOps automation, then turned back on when the deployment is completed. Again, it all would be machine-driven, with no human mistakes anymore, ever. This will begin to cut down “noise” in monitoring/alerting systems into only useful information. However, integrating monitoring systems into the normal DevOps continuum extends beyond just an on- and off-switch.
Forward-thinking organizations begin to consider testing and monitoring as two parts of the same evaluation discipline. When a test is needed and when a monitoring characteristic is needed now can be determined in total rather than just in a development capacity or operations capacity. A holistic view of what can and what should be evaluated about an application, and how that will be achieved, brings the operations team to the table with the development and testing teams to plot holistic solutions that will live on beyond the development project timelines. Examining monitoring relative to version-controlled artifacts allows for organizational constructs within the monitoring discipline that never were possible before DevOps enabled them.
The Masters of Hardware Abstraction
Containers, or similar types of technologies, abstract an application from the hardware details it runs on top of. Technology such as this is a trend in DevOps that greatly simplifies the world of hardware from your average application developer. So who will retain the knowledge about the jungle that lives and breathes underneath? Ops has an extended role in understanding what lives underneath the abstraction layer that lives on top. Over time, only Ops will understand these particulars. They will become the only in-house experts about which cloud provider to use, which sets of physical hardware performs best and under what conditions.
It is this kind on unique expertise that makes an operations engineer extremely valuable (translate: pricey). It is the kind of knowledge careers are made upon. It is ageism-neutral. Older employees do this today, and they will continue doing this in the future; the only difference will be fewer and fewer people will know how it is done. If the domain of application development is the province of our youth, then the domain of hardware abstraction is easily the province of our more experienced employees. These realities create personal value for our operations teams, and they are only possible in a post-DevOps embrace.
Impacts to the Bottom Line
The adoption of DevOps by the Ops portion of the equation is critical to overall success. Passive-aggressive resistance by operations staffers can delay or destroy altogether the value of DevOps. Overcoming this is a first step. But creating operations evangelists goes an order of magnitude further: With active operations staffers becoming internal DevOps evangelists, the entire operations team can begin to look forward at what DevOps will ultimately enable for them. The value they will carry back into development efforts becomes substantial and the entire company benefits. Careers can emerge that were not possible before it—and, just as importantly, so can profits.
To continue the conversation, feel free to contact me.