Why DevOps Teams Need Security Engineers

In the episode of View with Vizard, Mike Vizard sits down with Om Vyas, chief product officer for oak9, as he explains why security engineers need to become part of every DevOps team. The video is below followed by a transcript of the conversation.

Vizard: Hey, guys. Thanks for the throw. We’re here with Om Vyas, who’s Chief Product Officer for oak9. We’re gonna be talking about all things related to DevSecOps and cloud security. Om, welcome to the show.

Om Vyas: Yeah. Thanks for having me, Mike.

Vizard: We, of course, have been talking about DevOps for the better part of a decade. And then we decided that the security people couldn’t be left out, but they wanted to have the word “sec” thrown in there to make things a little more confusing, so then we got DevSecOps. And some of them even argue that I guess it should be SecDevOps. But who knows? My question to you is it seems like we’re still struggling with getting this idea across to people. It’s become more important in the cloud era, obviously, ’cause more developers are programming stuff. But what, from your perspective, is the fundamental challenges that we’re seeing with getting people to really implement DevSecOps practices?

Vyas: Yeah. I think if we take a quick step back, I think you hit it on the head. There’s DevOps, there’s DevSecOps, there’s SecOps. There’s a lotta buzzwords and a lotta different ways that these adoptions are being thrown around. I think fundamentally, my take is DevSecOps or DevOps, it’s first and foremost a cultural transformation. The technology and the people process really come after. So really, one of the biggest challenges is that embracing this notion of being able to work together, being able to have shared responsibility and a collaborative environment.

If you think about kinda where we’ve come from, and in fear of dating myself, I’ll say that when I used to develop long ago with my DevOps, my development teams, our focus was only towards application development. And then it was like the packaging, the art effects, once they were created, it was like, “Hey, kinda ” the notion around, “Hey, throw it over the wall to IT ops people,” or having formalized security review processes, right? And really, this cultural transformation is, I think a key area where I think a lotta the folks in the organizations, they’re struggling. They’re struggling to embrace the culture of DevOps, culture of DevSecOps.

And then when you factor in, as you said it yourself in the beginning, that the security people are getting left behind. And what that really means is they don’t have the visibility and they don’t have the automation to be able to keep up with the amount of churn that’s happening with the development, the DevOps. Because the dev teams are using the automation, which is the technology part of this transformation, to really create this velocity in which they are churning over new features, doing rapid deployment into production. Five, seven years ago, we used to do maybe one release a month, one release a quarter. And now, I’m seeing organizations, ours included, where we’re doing tens of releases a day, hundreds of releases a day. So this is really where the security or the “sec” part of the DevSecOps is really struggling to keep up with that velocity and that change.

Vizard: Does that mean we need to slow down? Or can we build and deploy secure applications just as fast? What’s the tradeoff?

Vyas: Yeah. I mean, I think maybe slow down is not the right answer. Because I think velocity is really being utilized as a differentiation for a lotta the – lotta the organizations. I think the right answer is – I think, not necessary just slow down to a halt. The right answer in my opinion is really, how do we integration security into the processes of DevOps to truly give them the visibility at the point of time when these changes are happening and then really arm them with the automation that they really need to give them full visibility, give them ability to be able to really quickly identify how that application is changing? How are those new capabilities that are being added, and how is that impacting security of that application? And the sooner we give visibility to security engineering of all of those changes and where security matters in those changes, the sooner we can continue to operate with that velocity.

Vizard: How do we go about doing that? ‘Cause a lotta the developers, they didn’t really set out to build something that’s insecure. It’s just more often than not, they have a tool that lets them manage a cloud as infrastructure, as code. But then they misconfigure something ’cause they’re not security experts, and maybe a port gets left open or something like that occurs. So how do we lock all that down and how do we kinda tighten that up without necessarily requiring the developer to become a security expert, ’cause that seems unlikely?

Vyas: Yeah, you’re absolutely right. I think that a developer is not a security expert. And I think if you look at kind of where dev engineering is today, especially with the Cloud Native architecture, especially with DevOps, it’s dev engineering or developers are taking on lot more responsibility, right? But I do think as a developer myself, what I’m interested in is not that I don’t care about security. It’s that what I wanna know is, “Hey, help me understand what I need to do to move fast. What do I need to do? What are the five things or ten things or whatever number of things that are related to security that I can do to not only help my security team, but really quickly move this feature capability from development and implementation to testing and to production?”

So from that perspective, I think what really also helps is if I have the ability to, within my own workflow, within my own tools – if somebody educated me and gave me kinda the prescriptive guidance of, “Hey, here is all you need to do to ensure that all of these different security aspects of security-related things in this infrastructure’s code that you change. Here’s what you need to be aware of. Here’s what you need to change, and to really help you move forward.” I think that’s what, I think as a developer, I’m looking for. Because again, I have many, many different things that I’m managing today beyond just application code.

Vizard: Mm-hmm. How automated can all this get? Seems to me the key to this whole thing is if the developers still need to program the cloud environment, we need to find a way for them to make fewer mistakes, put more guardrails in place, and kinda just gently lock down the environment, I guess, is the phrase I’d say. So what is the challenge, what is the state of the art at the moment?

Vyas: Yeah. I think it’s interesting. You said “gently lock down the environment.” I think absolutely, right? I mean, I think that developers, what we’re seeing today with a lot of our customers is they want lotta freedom because that freedom will enable them the agility and the velocity that they are looking for. But at the same time, we do need guardrails because they’re important.

And I think how do we gently lock down the environments? I think that’s kinda the key. I think one way we can talk about this is that when we think about that infrastructure’s code, we’re really talking about codifying the entire environment under which your application is running in the cloud. And what that actually enables us to do is be able to then say, how do we actually remediate? How do we actually focus on automation that doesn’t always just point out all the challenges or all the problems in that code around security? But then taking it that last mile, as I like to think about it, which is actually fixing it for that developer.

So I think that’s really where the trends are. And I think that’s really where things need to be, is really say, “Hey, I analyzed this infrastructure’s code. I have identified where all of the security challenges are. Here’s how to risk-appropriately secure this code. Because all of this infrastructure’s code is declarative. So it’s predictable in how it can be changed and what the impacts would be.”

So I think that’s really where it needs to be, is when I look at some of our customers and infrastructure’s code, it’s getting pretty complex, right? So we have customers that have 40,000, 50,000, 60,000 lines of Terraform code. So even cloud infrastructure code or infrastructure code itself is getting pretty complex. So no security engineers or no number of security engineers or even DevOps or developers can actually keep up with understanding, what are all the security implications in this code? So the real answer in my mind is, how do we automate this thing, take it to the last mile, and say, “Hey, here are all the recommended changes in this code as far as the security aspects of it is concerned?” So as a developer, now I can focus on the functional aspects, and I have an automated tool, an automated process, that will really enable to me to then not have to worry about the security aspects of that infrastructure’s code.

Vizard: And maybe no one ever needs to know how much I don’t know about security. But do you think that we’re gonna see a lot more of so-called artificial intelligence thrown at this problem? And is that part of this automation curve? Or are we making too much of AI and security these days and it’s just gonna require something a little more fundamental?

Vyas: Yeah. I think the short answer is yes. But I think my opinion is that AI kinda gets thrown around a lot. And I think when we’re talking about some of the deeper learning concepts in AI, we’re talking about some neural network things, I think those things are coming. But I think we still have to recognize that we’re in the infancy stages of this. So I think really to me, it starts with the fundamentals first. Can we actually provide these recommendations, this automation, to really help educate the user, that developer, that practitioner, that is today, lot more often than not is kind of utilizing lotta the modularized code that’s available in the web without really understanding, what are the implications as it relates to their own world or their own applications or their own context?

And lemme sort of dive in deeper. So not all applications or workloads in the cloud are created equal. So if you are a health care company that’s securing health care applications that’s utilized by your patients, that’s using PII data, their requirements around security may be completely different than if I’m building a marketing website where most of the data that’s on that website is publicly available. Both of those applications are not created equal. So how do we secure those two different applications is actually different.

So there’s an element of this dynamic nature. There’s element of understanding, what are the requirements of that application? What are maybe the compliance adherence to those applications? So really then, when we talk about securing those two applications, that’s also different. So it’s really what I think about the context behind that application and how you secure that application really becomes important or a focal point of, then what are the recommendations that you’re providing to that end user, that developer, within that infrastructure’s code? So I think that has to take that into consideration.

And then I think, to your point, it’s not simply just saying, “Hey, here are the ten things you need to do, developer. Go and here’s a tool that will actually remediate it for you.” That’s great. But I think part of it can be used really as an educational tool to really better understand, “Hey, developer, help us a little bit. Help us understand the context of this application. What are, what is it that you’re trying to do?” And then I think once you kind of cover those fundamentals, then I think as we go deeper beyond the infancy stages of this cloud infrastructure journey, then I think what we can do is we can learn behaviors. We can learn from the industry that this developer or this organization is in. And we can say, “Yeah, this is a health care industry vertical or this is a particular type of architecture or workload that’s being deployed in the cloud,” and secured.

And then I think we can start to get into intelligence around deep learning and other heuristics to say, “Hey, how do we react to then risk-appropriately secure this application,” right? Because otherwise, if we start treating all applications, all workloads that are run in the cloud, in the same way, there’s a lot more that we’re asking of the developer. There might be hundreds of things that we need to do to risk-appropriately secure a health care application that’s using PII data versus we might only need to do a handful of things to ensure that a marketing website that has publicly-available data is secure.

Vizard: What will be the role of the security team in all of this? Because one of the things that comes to mind is if we are shifting more responsibility or at least the enablement left, then what exactly does the security team need to do to kinda work alongside, then? Or will they just kinda set the policies and the rest of it will be automated?

Vyas: Yeah. I think, first and foremost, about kinda the role of security engineering today in the DevOps world or DevSecOps world, is really understanding and gaining visibility into what’s happening. Because if you have engineering teams that are deploying tens of times a day into production, they’re fundamentally changing, potentially, the architecture of that entire application. So I’ll give you an example. Today, with lotta the sort of building blocks or the way I like to think about it is cloud service providers are really offering these Lego pieces. So they’re outta-the-box capabilities.

Now, I can plug in an analytics service into my architecture and seamlessly provide analytics dashboard into my application or into my workload that I didn’t have a week ago. But that fundamentally changes, or it fundamentally has the ability to change, the architecture, and therefore the security implications. So first and foremost from the role of security engineering is really getting the visibility and getting the visibility as early as possible so that they can understand how, fundamentally, this application is changing and what are the security implications of that. So it, and I’ll say for that aspect of it, this is the people process side of embracing DevOps, is really engaging that security engineer as early as having that requirement or having that capability from business that says, “Hey, we need this analytics dashboard,” really getting the security engineering involved as early as possible.

And from that point forward, again, then it goes back to technology and automation. How can security engineering really understand, what are the implications of adding this analytics service from AWS or some analytics service from GCP or Azure? What does that mean for the security of my architecture? So there might be hundreds of different configurations. There might be thousands of different implications around putting in that one analytics service that’s available outta the box as a feature or service from the cloud service provide. And the reality is no security engineering can actually keep up with that.

So having a tool that really understands down to the configuration level, what are the impacts or the configurations of that analytic service to securely deploy that service and then what security engineering can do is security engineering can say, “Hey, let me understand the implications and the architecture when my development team is adding this analytics service.” And if we start early, if we get the security engineering involved early, if we enable, then, automation to take away the implications of down to the hundreds of configurations of securely deploying that analytics service away from security engineering, but then give them the visibility, then I think we have a really great starting point to enable all of the teams to work together with the same operational velocity that the dev engineering really desires.

Vizard: Mm-hmm. What is your best advice to folks? I mean, should I just take all these developers and security people and lock ’em in a room till somebody comes to an agreement? Or is there some more intelligent way to go about this?

Vyas: Yeah. I think locking everybody up in a room is not the right answer, right? I mean, I think what I’ve seen work really well is there’s lotta different ways to sort of come to a solution. I think there’s no one silver bullet here. But I think depending on the size of the organization, depending on the size of the teams, I think one of the things that should be done, and I’m starting to see some trending around this, is really add a security engineer to the engineering team as a whole so you really have a dedicated security engineer that’s spending part of their time with these engineering teams.

So typically, we’ve seen, in more of the modern development shops, you see lotta the dev engineering, it started out with just developers in a single team. Then we kinda focused on scrum teams. So there was development teams, there was a product engineering or QA involve.We’ve seen a DevOps engineer or SREs get involved in these teams. And I think, to me, the answer is, again, being able to put that security engineer in these teams, as well, to work closely with the developers, work closely with the SREs. Because then, what will start to happen is that engagement, that early visibility and understanding. So I think that’s one.

And then I think the other area for lotta the larger organizations where this may not work is I think the notion around excellence centers , so you have visibility across these little center of excellence teams. So there’s cloud ops teams, there’s security teams. It’s really good to incorporate lot of your security engineering and security architects into these cloud center of excellence teams. Because I think then again, it goes back to that visibility, goes back to engagement, and it goes back to helping security teams really understand, what are the different architectures? What are the different capabilities that lotta the dev engineering groups are using today? Is it the serverless architectures? Is it the micro services that are being built with containerized architectures?

And then what security engineering can start to do is they can start to build these reference architectures or guardrails around 80 percent of the, “Hey – ” recognizing that 80 percent of the workloads across my organizations are built with serverless architectures, as an example, in the cloud, or it might be that containerized architectures in the cloud. And that will really help them start to build these guardrails in an effective way. And then I think, you talked about sort of, early on, gently locking down environments. So I think if we then start to think about, in the DevOps pipeline, how do we incorporate these security automation tools that enable us, both the development teams, the SRE teams, and the security teams, to identify, what are all of the security underpinnings, vulnerabilities, design gaps that we are identifying, be it with the application code, be it with runtime, be it with cloud infrastructure security?

So it’s really, where possible, building these automated tools within the pipelines without actually blocking the pipeline so that you’re actually what you’re doing is you’re really enabling the development engineering and the DevOps engineering to really start to build in the process around, how do we ensure that all of these security design and the best practices are actually incorporated within the pipeline? And then as the teams learn and grow, then these tools can become those guardrails. So maybe you’re not blocking or completely blocking the pipeline or failing the pipeline at or in the development or sandbox stages, but then you start to guardrail those as that feature capability’s moving through user acceptance testing or staging or before it goes into production. And I think those are some of the key things that can be done to really enable more of this collaborative, but shared responsibility culture that I think is really going to drive lotta the benefits that are needed to build security in.

Vizard: Hey, Om, thanks for being on the show. It’s all about making sure that if you find a pattern, build a guardrail, and rinse and repeat.

Vyas: Yeah, absolutely. Thanks for having me, Mike.

Vizard: All right. Back to you guys in the studio.