A Pragmatic Approach to DevSecOps

In response to market and competitive forces, many companies are using DevSecOps to develop higher quality and more secure software. Whether that software is their own product or is a component of their product, quality – and especially security – have become required rather than optional. To accommodate the security requirements, some companies have adopted DecSecOps overlays to their software development process that can, and has, caused significant disruption.

Barriers to DevSecOps Adoption

The main barrier to the transition from DevOps to DevSecOps is organizational rather than technical. Many companies are already employing special security teams tasked with ensuring applications are secure and that policies are being followed. However, they tend to act as gatekeepers to the release process – nothing gets released without security testing. This approach is prone to be “too little, too late,” as the security team is usually involved near the end of the development life cycle. Attempts to “test in” security rarely succeeds. Security needs to be built in from the start; “shifted left” so that security is fully integrated into the process from the start. Let’s break down these barriers:

• Lack of motivation and awareness:  There’s a lack of awareness about security issues, and about the risks and impacts security vulnerabilities pose to the organization in terms of liability and reputation. Vulnerability statistics make it clear that many companies are still not taking security seriously enough. This indifference remains, even when 63% of companies had, or nearly had, vital data breaches in 2019 and each breach cost an average of $3.9 million to deal with.
• Distrust: Lack of trust between teams is expected in most organizations, but it’s particularly problematic between software developers and security teams. Developers view security as impeding their success, and security teams view developers as incapable of delivering secure software.
• Poor Perception: The prevailing view of security is that it’s difficult to implement and somebody else’s problem. If security teams are solely responsible for security, then it’s easy for developers to delegate to them. Companies often think that security can be tested into the product at the end of development – with disastrous results. Vulnerabilities discovered at this late stage are more difficult to debug, with subsequent impacts on time-to-market, budgets, product quality and reputation.
• Gatekeeping: The security team sometimes acts as gatekeepers of release candidates, which is a source of friction between developers, testers and security experts. Placing security as a last, QA-like step before delivering a release is too little, too late. Developers also don’t like other teams imposing restrictions on their work, which is an issue when trying to put security controls, like secure coding practices, in place.
• Lack of integration: Without integrating security practices and tools through the entire development process, security improvement initiatives have trouble getting off the ground. In fact, the key tenet of DevSecOps is to integrate security practices into each phase of development.

These barriers might seem daunting, but it’s not unlike other corporate cultural shifts. Security teams need to delegate aspects of security to development teams and integrate secure practices as early as they can in their work. Developers need training and understanding of how critical security is to their work. Developers already have an appreciation for safety and quality, but lack some of the basics of good security practices.

It doesn’t need to be this way. However, removing these barriers won’t happen overnight, so a practical and gradual approach to improving security is needed.

Easing Your Way into DevSecOps

Transitioning to a more secure software development process across an organization rarely works when done with a “big bang” approach. There’s often lack of buy-in from the teams, and general resistance to change when new approaches haven’t proven themselves. The cultural barriers to adoption remain, and the initiative fails. As with any large task, it makes sense to start small; to learn and adapt before expanding to more teams. The move from DevOps to DevSecOps should be organic.

Start Small

The alternative to the doomed “big bang” approach is to start with a small, motivated team, where it’s possible to implement new practices from the very beginning of a project. By keeping the scope, and thus, the technical and business risk, of the project smaller, it is possible to allow time for learning and adjustments along the way. Think of this as an incubator project, where the results are analyzed and used to evaluate gains, losses, what went right and what went wrong. This team can learn, and then teach the larger organization what works and what doesn’t. Importantly, they set the standard for how to approach DevSecOps, and kick off the organic growth needed for a smoother transition.

Starting small extends to the use of new tools, as well. It’s one thing for a small team to adopt new technologies, but they can be quickly overwhelmed. Static application security testing (SAST) tools can produce a lot of information. It’s important that initial projects concentrate on high impact, high probability vulnerabilities, first and foremost. It’s easy to get bogged down in the details, and discipline is needed to focus on the highest impact, and to get the most ROI out of new tools.

Start at the Beginning (Or as Early as you Can)

This is where “shift left” plays an important role in the transition to DevSecOps. A small team can introduce new techniques without a large initial impact on productivity. They can also start integrating security from the beginning, right at the concept phase of development. They can introduce new tools, and become proficient at using them. It’s at these early stages where the payoff is the greatest. Finding and fixing possible vulnerabilities, and adopting more secure design and coding practices, greatly reduces the downstream impact. It’s not hard to justify the effort, when security attacks and data breaches in production systems can cost millions in fixes, downtime and reputation.

Measure Success

The interesting thing about measuring DevSecOps success is looking at what isn’t there; security vulnerabilities and potential software weaknesses. To understand how well a new DevSecOps transition is working, it’s important to have a baseline of vulnerabilities that exist in your other applications. For example, how many vulnerabilities were discovered during security testing? How many vulnerabilities made it into shipping products? If these statistics are known, then it’s a case of measuring how well the new project does versus the baseline performance.

Other metrics are important, too, such as measuring the number of vulnerabilities detected, identifying those that are true risks and those that were removed through corrective action. SAST tools provide information on a per-build basis, so it is possible to look at trends over time. You can evaluate the DevSecOps process in terms of vulnerability trends over time; are they converging to a goal? How quickly is the convergence?

The ultimate measurement of success is a more secure product and, ideally, security testing at the later stages of development should find fewer vulnerabilities than in the past. After building in security and starting at the beginning with a shift-left approach, vulnerabilities are caught sooner and removed more quickly.

Start your DevSecOps Transition

The critical time to detect new security vulnerabilities is as soon as developers write new code (or test cases) before it’s submitted to a build or software control system. It’s also critical that tools work in your favorite development environment (be that VI, Emacs, Microsoft VS Code or Visual Studio, or any of a number of Eclipse-based environments). It’s at this point where SAST tools shift security improvement via automation to the earliest point in the development cycle. Finding and fixing vulnerabilities here is less risky, cheaper and easier.

Equally critical is establishing an initial baseline inventory of your third-party software assets, whether that is open source, reused internal code from other projects and/or third-party commercial software. This is done at the very beginning of the project, about as shift left as it gets! Software composition analysis (SCA) is critical throughout the development process since the open source and third-party software that you depend on is likely to have updates and introduce new security threats over time (including after you release the product).

Early adoption of SAST and SCA tools can have many benefits as part of your DevSecOps adoption. These tools used in conjunction with a security “built in” approach make this transition easier:

• Improve security awareness by exposing vulnerabilities in existing code, third party software and newly developed code. SAST tools provide all the necessary information to fix potential vulnerabilities, but also provide an explanation of the severity and risk. Developers get more experience with the types of software weaknesses that lead to vulnerabilities, and learn to avoid them.

• Illustrate ROI by measuring success, and have a goal in mind. Eliminating just one serious vulnerability justifies much of the investment. However, communicating this ROI is important, and needs to be put in the proper context of how costly and risky vulnerabilities are once a product is released.

• Remove gatekeeping so that developers and the security team are working together. Security teams provide the security controls that developers need to follow with a “trust, then check” approach. Developers learn more about secure practices over time, and integrate that knowledge into their existing workflow. In the end, there is no security team gate at release, and all members of the team know and understand the state of the product.

• Integrate security practices into software development gradually, starting with a small team and small project. As experience is gained, these teams train the rest of the organization on best practices.

DevSecOps From the Ground Up

DevSecOps is a way to build security into an application, and the transition from an existing CI/CD and DevOps process is made easier with the right approach. Successful transition depends on breaking down the resistance to security practices that exist in many organizations. A “start small and start early” approach is recommended where new techniques, processes and tools, such as SAST and SCA, can be integrated with less downtime and cost.

Most importantly, you need to decide what the success factors are. Measuring success and the ROI of the effort is needed to grow DevSecOps organically within the organization. Tools and automation provide the data needed to show progress, but communicating the ROI needs to be done in business terms. In the end, success results in a more secure product, and reduces your company’s exposure to data breaches and cyberattacks.