The late great W. Edwards Deming, statistician, professor, author, lecturer, and consultant said, “In God we trust. All others must bring data.” The point is, if you want people to trust your information, then you need to present measured data. This is true for countless circumstances. As a DevOps consultant, I am often asked by my clients what data should be used to show the value of their DevOps capabilities, in a way that is acceptable to the stakeholders of those DevOps capabilities. A lot of excellent research results, published by DORA and others, describes metrics that demonstrate the value that DevOps provides to application development. More and more enterprises are embracing the idea of providing DevOps capabilities, such as pipelines, tools, hardened containers and tech stacks, as a centralized service to support their many application teams. The question then becomes, “What metrics should be used to show the value of DevOps-as-a-Service (DaaS)?”
Before measuring the value of anything, you need to know two things: “What is it?” and “Why is it important?”
What is DevOps-as-a-Service?
In high-level terms, DaaS is DevOps capabilities made available to the users of those capabilities through portals and APIs. There is no standard set of services; DevOps capabilities offered vary from organization to organization. In my SlideShare, DevOps-as-a-Service (DaaS) Values, I illustrate a generic blueprint for DaaS.
The DaaS blueprint has four layers, as follows:
• Stakeholders – There are three types of stakeholders: business, app-dev users and DaaS teams.
• Service access layer – This layer consists of user portals, APIs, and access security.
• Service layer – The service layer consists of user services and management services. User services include things such as repositories, CI/CD pipelines, sandboxes, test data, configuration data, guides and training. Management services include service catalogs, issues and change requests, SLOs, SLIs, governance policies, experience monitoring and automated tasks.
• Resources layer – This layer includes application stacks, infrastructure resources, tools, infrastructure code and topology data.
Why is DaaS Important?
The value of DaaS, as perceived by different stakeholders, can be summarized as follows:
• Business – DaaS makes the business competitive.
• App-dev Users – DaaS improves application development.
• DaaS team(s) – DaaS improves management of DevOps capabilities.
What Service Level Objectives (SLOs) Demonstrate the Value of DaaS?
The SLOs need to be chosen to best fit the requirements of each application. The following are examples.
Business value can be demonstrated by:
ROI based on tracking cost savings relative to a benchmark comparing prior non-DaaS, trend cost per developer and per release.
SLO agility of services – Lead time for changes, and frequency of releases.
SLO reliability of services – change failure rate, time to restore services.
SLO governance, compliance and security scores.
App-dev user value can be demonstrated by:
SLO availability of DaaS, service level indicators (SLIs) depend on service components and architecture.
SLO service response time.
SLO experience scores.
SLO change request response time.
DaaS team value can be demonstrated by:
SLO scope of applications – number of apps supported, number of users per month.
SLO number of requests served per month.
SLO critical events per month.
How Does an Organization Develop DaaS Metrics?
Start by picking an application that is supported by DaaS. Do not try to boil the ocean and measure all applications at once. Then, appoint a DaaS SLO/SLI team to work on metrics development for the application. The team should include stakeholders from business, app-dev and DaaS support teams associated with the application.
DaaS SLO/SLI team members determine the SLOs of DaaS that each class of stakeholder values the most, and then decomposes the services to identify sources of data that will form the most important service level indicators (SLIs).
With the SLIs in hand, instrumentation, telemetry and analysis requirements can be determined and used to design monitoring agents, communication, analyzers, alerts and dashboards.
Then, roles, policies, playbooks and responsibilities will complete the solution. Test the solution for the chosen application. As you gain confidence in the solution, adapt and expand the solution for use with other applications.
Continue to monitor the progress and performance of the SLO metrics, and continuously improve them with experience and as newer technologies become available.
What This Means
DevOps-as-a-Service provides many valuable capabilities to business, app-dev users and DaaS teams. Metrics that demonstrate value to each class of stakeholder should be determined for each application that consumes the capabilities offered by each unique DaaS. A DaaS SLO/SLI team can determine the best metrics and system design for the implementation for an example application. The SLOs can be adapted and evolved for other applications in the organization, and used as a basis for continuous improvement.
