Assessing the ROI of DevOps normally is done against a company’s baseline cost. The problem is, most organizations do not collect or evaluate the cost of “automating and delivering” innovation; they usually are more focused on the innovation product itself. As an example, if a company creates an application hosted on the web that offers its customers the ability to purchase inventory, the company usually is obsessed with knowing how that application performs: How many people are using it? How often does it fail due to bugs? How many sales are made? The company typically focuses on the end product—on the application as it touches customers and potential customers. The delivery process that precedes the application being loaded into production is not usually the subject of intense scrutiny. At least, most organizations are not evaluating the speed of delivery and process effectiveness of delivery prior to an embrace of DevOps methods and tools.
This lack of scrutiny results in the lack of a baseline to compare against. Say, for example, a typical programmer (prior to DevOps) uses a mix of scripts, human intelligence and human verification to create and verify a build. Prior to DevOps, the goal is simply to create the build and get it out there to look at it, and decide what to do next. Most programmers are not obsessed with how efficient this entire process is; for them, it is a means to an end. So most programmers do not keep records of the average amount of time it takes to create the build from beginning to end. This is even more true of the deployment process. Once a build has been created (again prior to DevOps), getting it deployed to the target environment might involve a few different teams (depending on the class of test environment that is targeted). The process may involve email requests to other support teams or filling out a ticket to get it done. Once again, most organizations spend little time tracking this process from end to end; it is only ever a means to an end.
If, in fact, an organization was watching closely how long it takes to get a build created and how long it takes to get a deployment completed, that organization would be running toward DevOps at full speed to see improvements it otherwise could not achieve. It is the things we do not measure that receive little attention—until they break. Once our deployment process has become so onerous or so complicated or so burdened with approvals required that nearly nothing gets deployed, then we begin to look at it. This creates a culture of “squeaky wheel syndrome,” which makes both our development and our operations teams focus only on reactionary issues, while proactive concerns die on the side of the road. There is no time to invent proactive improvements when you are constantly fighting the fires of something else that is seriously broken. This culture must also change if DevOps is to succeed broadly in your organization, so keep note of how far it may be infected in how you run things today.
Preparing for Collection
Given that most companies lack a real baseline to compare against in this area, realize that collecting one will take an investment itself. This baseline metrics collection cost is usually added to the cost of the investment in DevOps. It is a bit unfair, as you are effectively saying, the soap company will also have to bear the cost of determining how dirty we are before we invest in the soap that will clean us. The makers of soap tend to think you already have a sense of how dirty you are, or you would not be looking at buying soap in the first place. But without an accurate assessment of knowing the extent of the problems (in this case, the investment of time you spend today), you will never know the ROI of having made the improvement of DevOps (the after picture).
Also, keep in mind that measuring anything manual is difficult. This sounds like a saying from Captain Obvious, but consider that manual work efforts require additional manual work efforts to capture them and record them in some kind of system. If, as an example, my typical programmer spends 15 minutes of every build doing human verification of the build, this amount of time is reflected nowhere. Hourly employees typically record how many hours they work per day, not what tasks they spent during the day. Even the employees who log work hours more discreetly tend to record only that they spent “x” amount of hours doing “development,” not what general task categories occurred within the heading “development.” So there are no electronic records out there that can be gathered automatically that will reflect the 15 minutes spent doing human verification of a generic build of an application. The only way to know this is a personal interview, followed by someone keying the data into a collection system of some sort.
Knowing that the information you seek for a baseline will come in human interactions as well as some automated logs, start the collection using a tool that allows for maximum flexibility for those efforts. Spreadsheets come to mind. When the effort to collect a baseline begins, you will have some ideas about the kinds of data you wish to collect and how it might exist in your organization. But it is impossible to know exactly how the resulting efforts will be completed. In short, you don’t know what you don’t know. While you collect metrics you will invariably come across someone who reveals critical metrics-related information you were not “planning” to collect. But once the revelation is made, you will need to incorporate the “new” information to really provide an accurate baseline assessment.
Determining the Baseline
The next step is to begin the collection process itself. The critical data to examine usually can be gathered by looking at “human” time involved in a given core “process” of the automation and delivery of innovation, and then the “machine” time involved as well. For example, the time a programmer must spend prior to the launch of a build (not the coding time)—all the prep work doubling checking that the system is ready to launch the build scripts (or ready to manually push the buttons)—would be considered “human” time. The execution of the scripts to create the build would be considered “machine” time. Then, all the work done after the build is created to verify it and make sure things are as the programmer expected, again can be put into the “human” time category. DevOps will systemically automate both the pre- and post-time activities the “human” currently invests, so all that time will count in the ROI.
The above example is good when one person has full control of a process from end to end. When other teams must get involved to complete a task, the “human” time begins to skyrocket. If, for example, a programmer or release manager, must get a request email (or ticket system equivalent) to deploy a particular build to a particular target environment, the entire life cycle of that ticket becomes part of the “human” time you need to track as your baseline metric. Even when shops create a two-day service level agreement (SLA) for turnaround, it still represents two days of delay (due to “process” time) in the life cycle of a deployment. The time “humans” actually spend reading requests, approving requests and routing requests is all time that DevOps will automate and eliminate. Finally, when the correct support team (usually Ops) pushes their own script buttons to do the deploy, that part of this process is once again “machine” time. As you can imagine, the end-to-end time of a given deployment can be days in the pre-DevOps methods and only minutes in the post-DevOps methods. Self-service deployments can reduce this time even farther, removing handoffs, reducing risks and delivering innovation at will.
Further classification of this metric data can be useful as well. For example, you may wish to break down “machine” time into its relevant subcategories such as build script execution, deploy script execution, test script execution and release script execution. You also may want to break down the “human” time a bit further as well into pre-build/deploy verification, post-build/deploy verification and test script results evaluation. Finally, you may wish to break “process” time down further into request generation, request routing, request approval, request declination and request notification. DevOps will be able to give you discreet visibility into the time you saved in these subcategories, although since baselining is generally a one-and-done effort, you may not want to hunt this level of detail. It is down the road where this kind of subclassification becomes useful—for example when you are trying to determine if one set of tests runs faster than another or one cloud provider runs faster than another.
Evaluating the Baseline
Having gone through this exercise, a hidden or secondary benefit emerges. Your organization begins to understand much better what it has been investing in the current delivery of innovation processes. Generally, immediate areas of concern emerge and provide the DevOps services delivery with a road map for which areas to attack first to provide the company with the biggest bang for the buck. Without a baseline effort such as this in place, your DevOps services delivery plan will be based on anecdotal thinking or on executive mandate that may not be substantiated by the data on the floor. Having a baseline provides a road map for delivery based on concrete evidence on what should matter most to the business in making improvements. No one can argue with the numbers (assuming they were collected with care). The road map may reveal which applications in your organizations portfolio should be targeted first for DevOps.
Also, your organization now understands much better “what” data in this process it sees as relevant to continue collecting even after DevOps is implemented. As DevOps is primarily an automation set of tools and methods, DevOps should be able to deliver these metrics in an ongoing basis. This allows continued metrics efforts to be automated without further human labor and can provide continuing proof of the value of DevOps to the organization, as over time the numbers continue to improve. Since the lion’s share of manual tasks disappear with DevOps in place, there should be no reason to have to manually collect metrics data as you move forward.
Finally, having an automated metrics collection system put in place by DevOps, should lead to the next evolution in thinking: simulation. Well after the initial ROI for DevOps services has received the accolades it is due, an automated metrics collection system should provide all the raw data needed for some predictive analysis. Tweaking variables or process points or “machine” effectiveness should result in increased speed of delivery that can be measured (even before it is reality) based on the data you have been collecting systemically. Understanding the impacts of these changes is worth more than its weight in gold, because you can weed out the planned changes that simulation advises against before they even hit the road map, let alone occur in your world.
Impacts to the Bottom Line
There is already enough anecdotal evidence in existence to entice companies to embrace the DevOps journey. However, as your strategy begins to look to deepen down the stack from application code level to middleware, database and operating system, each layer could be assessed against a baseline to determine if your road map is sound and, more importantly, in the right order for your best ROI. This applies as well as you look to expand your DevOps journey laterally with additional services such as automated testing and evaluation, gated progression, automated provisioning, release orchestration, configuration management database (CMDB) integration, production provisioning on demand and monitoring integration. Each new service can be assessed against a baseline to determine if its ROI still meets the demands of your finance department and if your road map here is in the right order for biggest bang for the buck.
The subtle win by performing baselining is the culture change it dispenses through your entire organization. Baseline metrics tell your development team and your operations teams (as well as the business units looking on) that you intend to measure performance and improve it systemically. Just because something was not examined before does not mean it will remain so, and a forest fire does not have to be the motivation to do so. DevOps is a proactive improvement to your delivery of automation. Collecting that baseline for your ROI, begins to seed the idea that proactive improvements are important to your executive teams. It is a demonstration of action over platitudes. It begins to eliminate the forest-fire responses in your employees, and creates a new forward thinking point of view. This cultural change is worth more to your company than any implementation of any one given tool or method, it goes back to changing how you think about delivering innovation in your individual job. That is what DevOps can do best.
To continue the conversation, feel free to contact me …
