How Log4j Becomes a Serious DevOps Problem

The recent discovery of the Apache Log4j vulnerability has wide-ranging implications for anyone who develops software, especially for those in the DevOps realm. What’s most troubling about the vulnerability (CVE-2021-44228) is how prevalent the use of Log4j is. The vulnerability is reported in a vast array of applications and directly impacts numerous Apache projects, including Druid, Dubbo, Flink, Flume, Hadoop, Kafka, Solr, Spark and Struts. There are also numerous Cisco programs impacted by the vulnerability. The list of impacted applications and libraries seems to be almost infinite, extending to Elastic LogStash, GrayLog2, Neo4J, Steam, Twitter and even the VMware Tanzu family of programs.

Simply put, any application that uses Log4J for logging and tracing is subject to the identified family of attacks known as Log4Shell. Those attacks prove easy to exploit and can be used to take control of vulnerable servers.

Why DevOps Practitioners Should be Concerned

The world of DevOps is all about reiteration, where features and improvements to applications are slipstreamed into production environments. To accelerate development, the QA process is usually limited to the most recent changes.

In other words, developers, via their pipelines, can often establish a baseline of quality and then speed up their CI/CD pipelines by testing only what has changed since the last iteration of the application. This process ensures an acceptable level of quality without introducing what are perceived to be unnecessary steps.

However, those processes do not take into account newly discovered flaws in older APIs and libraries, meaning that those flaws could telegraph into deployed applications and leave DevOps teams completely unaware of the impact and possible disruption. Detecting flaws after an application is deployed is like putting the cart in front of the horse.

Further complicating the situation is that many developers lack a well-defined and well-maintained software bill of materials (SBOM), which means developers may be unaware of the components and/or libraries being used in a deployed application.

Without an SBOM, it becomes increasingly difficult to remediate an issue with an application—especially if developers are unaware of the issue to begin with.

Where Should Responsibility Lie?

More often than not, vulnerabilities introduced into applications become a game of finger-pointing, where developers blame library suppliers or infosec staffers, and infosec blames developers for not performing due diligence on the components they are selecting to incorporate into their applications.

Of course, finger-pointing only adds to the angst and delays the process of remediation and does nothing to determine where the responsibility for detecting and remediating vulnerabilities lies. DevOps teams need to put these squabbles aside and establish a process that deals with unintentionally introduced vulnerabilities. What’s more, DevOps teams need to invest in DevSecOps to make sure that vulnerabilities are detected and remediated as quickly as possible.

To avoid the potentially massive disruption that a vulnerability like Log4Shell can cause, those running DevOps should consider:

Creating and maintaining an SBOM
Deploying automated tools that scan for discovered vulnerabilities
Automatically comparing the SBOM against vulnerabilities
Including DevSecOps processes within their CI/CD pipelines
Frequently testing applications for flaws
Coordinating with cybersecurity personnel when a flaw is discovered
Patching frequently

With more and more vulnerabilities discovered on an almost daily basis, DevOps teams must be ever vigilant about what components are incorporated into their applications. What’s more, with the increased use of low-code/no-code tools and integrated development environments (IDEs), DevOps will also need to consider if those tools use other libraries that can become compromised.