Without proper consideration given to security best practices, the continuous delivery of software changes facilitated by DevOps is risky. On the other hand, DevOps provides an opportunity to reduce security risks if security is integrated into the continuous delivery pipeline according to best practices.

Leadership

Leaders need to understand and sponsor a clear vision for security.

Leadership demonstrates a vision for organizational direction, team direction and three-year horizon including security practices.
Leaders intellectually stimulate the team status quo by encouraging asking new questions and question the basic assumptions about the work including security practices.
Leaders promote personal recognition by commending teams for better-than-average work, acknowledging improvements in the quality of work and personally compliment individuals’ outstanding work including security practices.

Collaborative Culture

Culture in organizations that work well with DevOps have a collaborative continuous security mindset.

The culture encourages cross-functional collaboration, shared responsibilities and avoids silos between developers, operations, project management, quality assurance and security.
The DevOps system (toolchain) is created by an expert team and reviewed by a coalition of stakeholders including security.
Changes to end-to-end DevOps workflows are led by an expert team and reviewed by a coalition of stakeholders including security.
DevOps culture empowers and trains team members to take personal responsibility for security, compliance and privacy obligations.
Security engineers/architects are involved in the design for modular components, and also consulted when security patterns change within modules.

Design for DevOps

Designing software for DevOps at speed requires application designers to master the best practices for continuous security.

Software source code changes are pre-checked with static analysis tools, prior to commit to the integration branch. This assures that the modified source code does not introduce critical software faults and security vulnerabilities such as memory leaks, uninitialized variables, array-boundary problems and SQL injection.
Software component analysis scans third-party components for known security vulnerabilities and identifies risk during the build process.
Security frameworks for technology stacks that are used (such as Apache Shiro or Spring Security) are documented and shared with developers for their respective technology stacks.
Software code changes peer code reviews include checks for defensive coding and security vulnerabilities prior to committing code to the integration/trunk branch.
Common security components such as identity, authorization, key management, audit/log, cryptography, protocols, etc. are maintained, published, readily available and used within module development.

Continuous Integration

Continuous integration (CI) within organizations that have multiple teams working concurrently on a project and different code bases is challenging. During the integration stage it is critical to assess the application and understand the impact of code changes from a security point of view.

A software version management system is used to manage versions of all changes to source code, executable images and tools used to create and test the software.
Incremental static analysis pre-commit and commit checks are wired into CI to catch common mistakes and anti-patterns quickly by only scanning the code that was changed. These checks identify security vulnerabilities through control flow and data flow analysis, pattern analysis and other techniques. These techniques find security-related issues such as mistakes in using crypto functions, configuration errors and potential injection vulnerabilities.
Binary artifacts are digitally signed and stored in secure repositories.
Changes to security patterns used within software source such as session management, authentication, authorization and encryption code trigger a notification or pull request to security engineers.

Continuous Testing

Continuous testing (CT) with DevOps has significant advantages for continuous security when best practices are followed.

New security tests that are necessary to test a software change are created together with the code and integrated into the trunk branch at the same time the code is. The new tests are then used to test the code after integration.
Security tests for each DevOps pipeline stage are automated and may be selected automatically according predefined criteria.
Release regression tests include security tests. At least 85 percent of the security regression tests are fully automated and the remaining are auto-assisted if portions must be performed manually.
Test results that indicate possible security concerns are tagged for security analysis.
Dynamic or interactive application security tests exercise the application for security vulnerabilities. Results of these tests are delivered to developers through tools and feedback loops native to their organization.
If containers are used, image repositories are scanned for images with known vulnerabilities, hash checks for image drifts and run-time checks for vulnerabilities during image deploys.
Attack patterns, abuse cases and tests are built for application module profiles.
Unit, functional and integration tests around—and especially outside—boundary conditions are run during CT. Tests include error handling, exception handling logic and negative tests.
Automated security attack testing to include the OWASP Top 10 in integrated into automated testing.

Continuous Monitoring

Continuous monitoring (CM) of security considers the dynamic nature of artifacts and infrastructure and a proliferation of objects and services to secure.

Metrics and thresholds are automatically gathered, calculated and made visible to anyone on the team that subscribes to them. Example security metrics include:
- Number of security defects identified pre-production
- Percent of code coverage from security testing
- Number of failed builds due to security checks
- Mean time to detection
- Mean time to resolution
Vulnerability information in consolidated to provide a comprehensive view into vulnerability risks and remediation across tools, pipelines and apps—and over time.
Metrics and events from production security controls such as WAF, RASP, etc., are used to improve security testing.
Insight into security threats and events are shared and visible across DevOps teams to enable “attack-driven defense” methodologies.

Continuous Security

Continuous security (CS) is itself a distinct pillar with standalone best practices that cross all of the other pillars.

All information security platforms that are in use expose full functionality via APIs.
Immutable infrastructure mindsets are adopted to ensure production systems are locked down.
Security controls are automated so as not to impede DevOps agility.
Security tools are integrated into the CI/CD pipeline.
Source code for key intellectual property on build or test machines are only accessible by trusted users with credentials. Build and test scripts do not contain credentials to any system that has intellectual property.
External penetration tests (done out of band) are scheduled either periodically or on a regular cadence are used to perform deep-dive analysis.
Telemetry from production security controls such as WAF and RASP are delivered back to development teams to inform application updates.
Accurate inventory of all software packages and version information is documented via infrastructure as code. Automated detection is used to identify whether any of the packages have known CVEs associated and define specific remediation actions.

Elastic Infrastructure

Elastic infrastructure environments offer advantages compared to legacy or traditional static infrastructure. However, elastic infrastructures need to follow best practices for continuous security because flexible infrastructures offer a broader range of attack surfaces.

Configuration code includes automated checks including:
- Ensuring unnecessary services are disabled and only ports that need to be open are.
- Permissions on files, audit and logging policies are enforced.
- Development tools are not installed on production.
Security-approved OS, software versions and frameworks are used to compose required infrastructure. Security-related controls such as ACLs and FIM are defined as a part of infrastructure where applicable.
A least-privilege model is enforced for processes running on shared infrastructure.
Smaller clusters are used to reduce complexity between teams.
Service provider partners security controls are validated to ensure they meet business requirements in their domains of the shared security model.
IaaS or PaaS service provider security controls are validated to ensure that they meet business requirements in their domains of the shared security model.

Continuous Delivery/Deployment

While DevOps enables new feature deliveries to users quickly, to minimize risk during deployment the following best practices are important:

Release-to-production decisions are determined according to predetermined metrics, which include security metrics.
A whitelist policy for application segmentation is enforced during deployment for each environment, especially production.
Continuous deployment processes trigger run-time security and compliance checks including:
- Ensuring unnecessary services are disabled and only ports that need to be open are.
- Permissions on files, audit and logging policies are enforced.
- Verify development tools are not installed on production.
All secrets used for deployment are vaulted and retrieved programmatically during run-time or initialization of the continuous delivery process.

In this article we detailed best practices for nine pillars of continuous security, otherwise known as DevSecOps. While is it clear that DevOps offers immense value for software deployment, the adherence to best practices is essential to reduce risk and assure security. Each organization is different and has different security postures. The specific practices, security tools and metrics that are appropriate may vary according to a risk assessment for each organization.

Aggregation of end-to-end security metrics, from planning through to operations, is important but beyond the scope of this blog. End-to-end security monitoring tools, such as value stream management applied to DevSecOps and enterprise security dashboards, is a topic that we intend to cover in a subsequent blog.

This article was co-authored by Imran Chaudhari. Imran is a principal security consultant at Trace3, where he assists clients in assessing and designing cybersecurity initiatives from a strategic, organizational, and technical perspective. His experience covers expertise in secure software development, application and cloud security, as well as identity and access management and DevSecOps assessments.

— Marc Hornbeek