Let’s get one thing out of the way. DevOps isn’t a panacea—it isn’t magic, it isn’t a requirement for any enterprise and it won’t work for every team in every situation. Above all, it isn’t a yardstick by which an outsider can measure great operations, development or IT. And if anybody tries to tell you it is, feel free to say thanks and politely change the subject. You’re the one responsible for delivering the services your business depends on, and if the way you do it meets the needs of your business and the team is happy, stick with it.
But if you’ve tried adopting DevOps methodology, it’s likely because your business needs to substantially alter itself to meet specific goals. And DevOps has demonstrated the ability to help many organizations. Unfortunately, just as many seem to abandon DevOps initiatives and roll back, sometimes more frustrated than if they hadn’t tried it in the first place.
IT might be the team responsible for reliability, but this is a big challenge in an increasingly “fail fast” era. Many businesses are encouraging their departments to experiment, learn from their mistakes and keep trying until they deliver business-transforming innovation. But in IT, the high visibility of issues—even those initially within the accepted risks of a new project—seem to be treated differently. Even minor and expected teething pains can be judged as failures in hindsight. And many of us have seen instances in which attempting something new and encountering difficulty seems to make it five times harder to get off the ground on the next attempt.
DevOps is a prime example. As its culture and practices gained popularity, companies were quick to jump on the bandwagon—only to stumble in the execution or not deliver anticipated “transformation” or measurable ROI. It’s not uncommon for these companies and decision-makers to be left with a bad taste in their mouths. They may be reluctant to revisit DevOps and try again, even with the gained value of previous investments and lessons learned.
This isn’t much of a surprise: Humans don’t like to investigate public failures, and everything in IT is public, assessed in binary absolutes. It either works or it doesn’t, apps are up or down, and everyone in the company can see them. It’s also natural for businesses to switch their focus to the next potential success if personal political capital was part of a DevOps project. And as engineers, looking forward is in our DNA. We’ve built our careers around finding new and better ways to meet customers’ needs, solving novel challenges, and permanently squashing nagging issues to make room for new technology. The trick is to avoid considering DevOps a passing fad.
There’s significant value to be gleaned from any investment in DevOps—a better understanding of the human dynamics of your organization, at a minimum. If your team tried to implement some of the processes of DevOps or took the advanced step of adopting DevOps culture, don’t forget you did it for a reason. Go back and review what led you to consider DevOps in the first place. Chances are the business needs behind it haven’t changed and may have even expanded.
Outline those earlier reasons—they’re likely still there, even under novel circumstances such as shifting the team to work from home. Did your DevOps effort meet its original objectives, regardless of any rear-view assessment? Drill into the details where it didn’t. Was it due to culture, communication, risk reassessment or operations challenges? Did your team end up building something different than what you intended—a compromise not meeting the business’s needs? Take your time; you likely had many good people on the effort who are eager to share what they learned. A previous misstep could be due to a skills gap, software, management support or something as basic as infrastructure. The first step in taking another swing at DevOps is analyzing the previous attempt with data instead of opinions.
And there’s a good reason you’re likely planning to try again: The stakes are high. According to Forrester, 80% of revenue growth will depend on digital offerings and operations by 2022. Organizations will survive and grow depending on their ability to design and deliver technology services, respond to issues and meet growing user expectations. At the same time, considering the environment brought on by COVID-19, IT teams are seeking new ways to sustain operations and use the time to build a head start for when business picks up.
With this in mind, here are a few ways businesses can approach evaluating the execution of a first DevOps attempt to inform a successful second try.
First: Deputize an Investigator
You’ll need a deputy to help you conduct a forensic analysis and your QA team might be an unexpected place to find the right candidate. QA likes many of the outcomes of DevOps practices because they’re often suffering horror-show weekends at the hands of big-bang monolithic deployments at several companies. If you don’t find a detective there, empower someone who focuses on quality service and considers it a peer to availability, security and resiliency.
Your first job will be to demark the fault lines. Remember, the task isn’t to assign blame and a good admin-side manner is required to prevent this. Remind the team the mission is to avoid making the same mistakes, not to point fingers. Perhaps you originally had fantastic buy-in but everyone got caught up in the wrong details. Was the problem operational? You’ll need to know.
Second: Admit Mistakes and Ask for Feedback
Once you’ve conducted your analysis, you’re ready to take it to the next step: humanizing past technical actions. Consider what the previous attempt might have been like for different teams and people. Remember what it was like when you rode the help desk, when you were sitting in the data center on a weekend or when you were working as a dev lead under an impossible deadline. Chances are you’ll remember the common frustration was external direction not taking people into account. DevOps success is primarily about people. Align your DevOps reboot with the humans on your team and the processes will follow.
For instance, your company may have hired outside consultants to “rub a little DevOps on it,” and it’s possible they evangelized a lot of dogma. “Do exactly these things, this exact way, or you’ll fail at digital transformation,” they might have said. There may have even been FUD and frightening charts. Perhaps they said everything would be neatly wrapped up in six months and you’d achieve a high-performing, blended, and fully active nirvana within a year. “You could be the next Netflix!” You’ve probably heard this story several times by now, and if it’s happened to you, know you’re not alone.
But your approach this time will be different because it will be yours. You will have noted sharp edges needing to be filed down. The broader team will be integral to conversations from the get-go. Internal champions won’t be under the influence of ballyhoo, fancy decks or hard sells. Your team will have a list of concerns but also possibilities reflecting a shared interest in modernizing operations. One look at salaries for job postings with “DevOps” in the skill set has likely piqued more interest from your team than a year ago.
Open conversation has always been the hallmark of good forensic analysis. Express what you think went wrong the last time and encourage frankness. Acknowledge concerns—previous outcomes may have validated many of them. Your team will be looking for proof you’ve given thought to a process that might—in extreme circumstances—temporarily affect availability or lead to a security concern. Maintain an open invitation for everyone on the team to be a part of the conversation and hear their concerns. And wherever possible, lean toward person-to-person communication. You’re changing hearts and minds, not reviewing governance policy.
Third: Unite Your Team by Confronting Obstacles Together
Don’t perform a shotgun wedding between ops and dev. Administrators and developers are drawn to their technology foci for personal reasons and interests. One of the most cited reasons for unsuccessful DevOps plans is a directive to homogenize the team, followed by shock this didn’t work. Developers are attracted to and rewarded for innovation and building new things, while admins take pride in finding ways to migrate the mission-critical apps everyone forgets about onto new hosting platforms. They’re complementary, integrable engineers, but they’re not interchangeable cogs.
Contrary to popular opinion, telling developers they’re going to carry a pager for escalation doesn’t magically improve code quality and can slow innovation. They may even quit, even in this chaotic economy. And telling ops they need to learn code patterns, git merge and dev toolchains will be an unwelcome distraction not related to keeping their business running or meeting their personal review goals. They also may quit.
It might be helpful to share with your team the simple idea you embrace: Unfounded stories of friction between dev and ops aren’t about the teams. They’re usually someone else’s war story about the organization, technology or process obstacles dividing them. At its heart, DevOps is about clearing these obstacles, not replacing them with new ones.
By 2020, you’ve likely met someone from a team who had an opportunity to work based on tenets of DevOps, and they likely have no intention of going back. They’ll tell you they built or modified interaction models and evolved their deployment methodology organically. They streamlined the way code is built, improved testing at the bench level, staging and production. Even some companies needing to maintain specific waterfall processes for business or regulatory reasons may benefit if the transition is organic and incorporates their specific needs.
Fourth: Set Your Plan in Action
The last step—and this time, hopefully, the easiest—is to set your plan in motion, especially if you’re doing so at a smaller scale with better instrumentation than before. Ops teams and developers are people of action. We like to research, test and discover, but we’re happiest when our work sees production and we have a positive impact on the business. And often with initial forays into DevOps, stymied progress is what makes teams give up. By tracking ops and business metrics together, all the stakeholders will be able to share a sense of progress toward the overall goals, even when you break a few things along the way. Think of it as fail fast insurance.
For example, consider adding development metrics and business metrics alongside your traditional infrastructure data. Track rate of change, error rate over time, deployment events and PR requests to get a measure of release times. Share your application performance management (APM) metrics with the whole team—real-time user-to-app-code-to-infrastructure-to-resources data sparks insightful cross-team conversation. But you should also track metrics affected by ops quality, including visitor time on site, page abandons, service desk ticket rate and CSAT. You likely already have the data you need, but perhaps it’s siloed in different systems. If your dashboards combine data mirroring the mix of teams and infrastructure flying in formation, it’ll be much easier to identify issues, evolve your initial plan to suit your unique environment and improve overall DevOps collaboration.
And this time, you’ll be able to see code check-ins, unit tests, builds, pipeline progress, deployment to staging and prod, and follow-through on performance, issues and anomalies. If the team gets nervous trying something new, you’ll have actionable data, the production feedback loops developers crave and common charts everyone agrees on. It’ll be easier to know your DevOps effort is working and to keep pushing forward as you address unexpected challenges.
You’ll Know It When You See It
I made a joke in a talk once that DevOps is a lot like being in a relationship: You kind of fall into it. You learn about it, adopt some of its practices and enjoy the experience when it works. You don’t set out to “get in love” and likewise don’t set out to “get in DevOps.” Rather, you find yourself in a situation developed with experience over time, and you wake up one day and realize, “Hey, we’re a DevOps shop!” That’s why you can’t buy it in a box, and nobody else’s will be like yours. It’s not about the technology, it’s about the humans working together to manage it.
Maybe this is the best reason to give DevOps another shot. It didn’t spring from a consulting team as a framework to drive engagement or from a vendor with an appliance requiring admins to bend to arcane workflows. It came from people; from developers and administrators who fought the same operations challenges we all have and who wanted to share what they learned. And this is why, outside of a bit of time investment from your team, DevOps is free. It’s made to help, not to sell you anything.
