Merriam-Webster defines a secret as something kept from others or shared only confidentially with a few. Modern software development teams use software secrets such as passwords, tokens and API keys to properly set up, protect and maintain development environments and delivery pipelines and to enable programmatic access to services such as AWS Buckets or Azure Blob Storage for cloud-native apps. Yet, while secrets are critical for modern software to function, they are difficult to manage across delivery stages. All too often, software teams trip over the complexities and inadvertently leave these secrets exposed in both public and private software repositories.
Potential exposures can stem from many factors: Plain text passwords, weak cryptography, flawed CI/CD or packaging automation and compromised developer accounts or malicious insiders. Whatever the cause, bad actors are regularly scouring repositories and application binaries for hard-coded secrets they can exploit. Look no further than the CircleCI and CodeCov incidents, where exposed secrets left these organizations vulnerable to adversaries intent on attacking software supply chains.
Rethink Secrets Prioritization
According to GitGuardian analysis, 10 million new secrets occurrences were exposed on GitHub in 2022, a 67% increase compared to the company’s 2021 findings. As software becomes larger and more complex–with open source, third-party and commercial components–the number of detected secrets will only increase, creating a different challenge: Prioritization for remediation.
Regrettably, teams lack a scalable means to understand which detected secrets developers can’t or shouldn’t remediate. Today, teams must manually sift through potentially thousands of detections and understand their context to triage these threats. Gaining context means discovering where secrets reside (in your code, in a software dependency, in the documentation included with your software) and understanding their purpose (to access resources, to enable user testing, to enable intrusion detection).
As teams consider how to simplify and automate the prioritization of secrets, important questions to consider include:
â—Ź What secrets are your developers responsible for protecting? For example, secrets used to access internal CI/CD pipeline technologies. Depending on permissions, leaking these secrets enables malicious actors to change automated build processes with no one being the wiser. Similarly, API credentials to access data or services needed for their code to function. Leaking these secrets gives malicious actors programmatic access to those APIs, which can be a gold mine of customer data or further compromise the environment.
● What secrets are worth protecting? There are some you may be responsible for but which are not worth protecting. For example, to facilitate automated software testing, developers typically publish API keys and certificates in public repositories alongside their applications or in documentation bundled into release packages or software containers. Since these testing “secrets” are designed to be commonly shared, they should not be encrypted or protected.
â—Ź Which detected secrets are not actionable? For example, developers typically cannot manage secrets in third-party or commercial components and dependencies within software. Instead, these exposures represent a third-party risk management problem. Another example is canary secrets added to software by security operations; remediating these secrets hampers intrusion detection.
Once teams have established their secrets definitions, building a process to identify and prioritize them comes next.
Managing Secrets
Any organization involved in software development should have clear rules for handling and incorporating secrets into code and for communicating that process and those rules to developers. Here are secrets best practices every organization should take into consideration:
● Promote developer best practices — and monitor compliance: Secrets leaks are more likely to occur in development organizations that lack clear guidelines for properly managing secrets in their code and easy-to-use tools to implement those guidelines. DevOps teams should formulate policies based on an understanding of how secrets are generated within their development environment, how those secrets are protected once deployed and how (hopefully) they are eventually cycled out of use.
â—Ź Revamp how applications use secrets: Design applications in a way that eliminates the reliance on access tokens that reside in code. Use technologies like OAuth OpenID Connect (OIDC) tokens that allow developers to authenticate users without directly exposing user credentials in code. Reduce the risk of abuse by using multi-factor authentication, leveraging identity and access management platforms and setting IP ranges that limit inbound connections for APIs to known addresses or address ranges.
â—Ź Establish end-to-end security: Misconfigurations in CI/CD tooling or release oversight can result in the inclusion of secrets during final builds that might otherwise escape notice during a code audit. Consider implementing binary analysis to extend visibility into leaked secrets beyond raw code and into software packages and containers ready for release. This capability can detect exposed secrets resulting from build or packaging automation.
â—Ź Streamline secrets prioritization: Noisy scanning results create a ton of triage work just to weed out false positives (e.g., widely distributed testing keys or network credential placeholders) before remediation can start. Automating some of this triage with relevant file intelligence saves countless hours of effort and ensures developers focus on addressing actual risks.
No Silver Bullet
Unfortunately, there is no “silver bullet” fix for software supply chain risks and attacks, nor for leaks and exposures. Instead, organizations need a holistic approach that includes scoping the problem, addressing short-term risks posed by leaked credentials and then extending changes in all processes with an eye toward taking the toil out of identifying, prioritizing and remediating secrets leaks—and therefore preventing potential supply chain attacks before they can happen.