How to bring security into agile development and CI/CD

Devops came about because of the cultural, functional, and technical walls between development teams that want to release frequently and operations teams that need to preserve reliability and stability. Devops culture addresses the mindset, collaboration, and practices to achieve both objectives, and devops practices—including continuous integration and delivery (CI/CD), infrastructure as code (IaC), and AIOps, which leverages machine learning in application monitoring, enable the implementation.

As more people and organizations adopted devops, it became clear that the term “devops” fell short of describing the full breadth of the movement, its practices, and requirements. I once called out the need for DevQaOps, and I have recommended shift-left testing practices where feasible.

But equally important, if not more important, is the need to make everyone responsible for security. Shifting security into development and operations, or DevSecOps, helps you achieve this.

Software security starts with developers

Before devops, development teams often implemented security practices in the final stages of an application release process, usually as a required step by a change advisory board (CAB). Because security teams were brought in late in the process, they had limited time to learn business requirements, understand technical changes, evaluate risks, and run security tests. When security teams escalated issues, there was limited time to remediate the problems without impacting timelines, and issues that required substantive code changes left development teams with hard choices.

Testing security late in the release process can be a critical risk for devops teams that are increasing the frequency of releases or investing in microservices. In the Accelerate: State of DevOps 2019 report, published by DORA and Google Cloud, 43% of respondents are identified as high or elite performers who release applications daily or weekly. That’s a significant increase in production deployments, a rate that requires an integrated approach for implementing security best practices frequently and early in the development process.

Collaboration between agile development teams and infosec is needed in the following areas:

• Reviewing security requirements, architecture, and coding practices
• Instrumenting automated security tests in CI/CD pipelines
• Monitoring applications for threats and resolving security issues

I’ll offer some guidelines for tackling each of these areas in the sections below. 

Collaborating on security requirements, architecture, and coding practices

Development teams and infosec must partner on security early in the agile development process—even before coding begins. In the 2019 State of DevOps Report, published by Puppet, CircleCI, and Splunk, the authors identify several best practices for how development and infosec teams should collaborate:

• Security and development teams should collaborate on threat models.
• Functional and non-functional security requirements should be prioritized in the product backlog. 
• Security requirements should be treated as design constraints.

Agile development teams can implement these practices by flagging higher security risk requirements and implementations for security reviews. Development should partner with infosec on the requirements, architecture, design, and implementation of the parts of the application that captures user information, manages authorizations, or processes sensitive data.

For less risky coding changes, agile teams should write user story acceptance criteria that address infosec’s security requirements and constraints.

Agile developers should also review OWASP’s security by design principles, which include several best practices:

• Establishing security-driven default policies in areas such as password aging
• Implementing the principle of least privilege when defining roles and entitling access to business processes
• Understanding security principles such as separation of duties, “don’t trust” services, minimizing attack surface areas, and avoiding security by obscurity
• Fixing security issues quickly by understanding the root causes and implementing holistic fixes

Lastly, the development and infosec teams should jointly establish a reference for coding best practices. Some good starting points include coding practices from Carnegie Mellon University, best practices from Safe Computing at the University of Michigan, and the best security coding practices for the programming languages and platforms used.

If you are deploying applications to public clouds, you should also review best practices such as AWS security by design, the site on designing secure applications on Azure, and the Google Cloud security overview.

Testing for security in CI/CD pipelines

The next stage to consider security is the CI/CD pipelines, where automated code and security validations can break builds and alert developers. Some of the more common security practices and tools to consider when establishing CI/CD pipeline standards:

• Static application security testing (SAST) platforms like SonarQube, Veracode, Sentinel Source, and Checkmarx scan code for different vulnerabilities and patterns. For example, SonarQube scans for bad inputs (taint analysis), cross-site-scripting, sensitive data exposure, and known vulnerabilities. Veracode states they have scanned more than 11 trillion lines of code and have a false positive rate of under five percent. Checkmark works with more than 20 programming languages and complies with PCI-DSS, HIPAA, FISMA, and other regulatory standards. All three tools work across many IDEs and CI/CD platforms. There are also open-source SAST tool options like CodeWarrior and NodeJsScan. OWASP lists more than 20 SAST tools and states that their weaknesses include finding configuration issues and vulnerabilities in authentication and access control.
• Dependency scanning tools review the underlying software components including open source libraries and report vulnerabilities. GitLab Secure has SAST and other security tools, including dependency checking, and works with Java, JavaScript, PHP, Python, Ruby, Scala, and Go. OWASP Dependency Check has integrations for Jenkins, CircleCI, and SonarQube. Snyk Open Source Security Manager enables developers to find and fix open source vulnerabilities. Microsoft recently released Application Inspector, a code analysis tool that reports against 400 patterns including features impacting security.
• Penetration testing has been around for a while, but traditionally, many organizations have security teams run these tests independent of the code, build, and deploy processes in the software development lifecycle (SDLC). One of the more popular tools, OWASP Zed Attack Proxy or OWASP ZAP, can plug into CI/CD tools like Jenkins and trigger off deploys. In the All Day DevOps series on ZAP in Ten, Simon Bennetts, ZAP project leader, notes, “The earlier it’s used, the better. ZAP really shines with automation.”
• Devops, cloud, and development tools generally offer their own security plug-ins. For example, both Jenkins and Azure DevOps have more than 40 security plug-ins, while CircleCI lists over 20. Microsoft Azure has published its continuous security methodologies, while AWS provides DevSecOps guidelines for CodePipeline users. As security technologies, integrations, and devops tools are all evolving rapidly, infosec and development teams should regularly review these tools for new security plug-ins. 
• One other important consideration is securing the CI/CD pipeline itself. For example, securing keys and parameters is critical for security, and CircleCI, Jenkins, and Azure provide tools and recommendations for locking these down.

Closing the security loop with monitoring and AIOps

There is a whole other set of DevSecOps disciplines tied to securing infrastructure as code, hardening containers, and configuring cloud services. In addition, there are specialized DevSecOps topics on data security, identity management, and securing IoT devices. If your engineering and development projects cover infrastructure, mobile, networking, IoT, or analytics, you will find specialized security practices and tools in these areas as well.

Going beyond infrastructure and data security, anyone working in application development must have a better understanding of how applications perform in production environments. Reviewing incidents, participating in root cause analysis, and remedying defects are all crucial application development responsibilities. For developers, this often means improving logging and reviewing analytics from application monitoring tools.  

One emerging operational technology is AIOps, which takes advantage of machine learning and automation to simplify devops and application monitoring. Operations teams typically work with a number of different monitoring tools, but juggling multiple tools can slow down efforts to resolve incidents — especially in complex, multi-cloud environments, and especially when development teams deploy changes frequently.

AIOps tools aggregate operational data from multiple monitoring tools, application log files, or infrastructure components. They then apply machine learning to help identify incidents, trigger automated responses, and reduce the time to resolve them. These tools also help discover outliers and slowly evolving issues by sifting through longitudinal operational data. Many security issues can be found using this type of analysis.

Reviewing monitoring and AIOps tools for security issues is how infosec and development teams bring operational security incidents back into the agile development process for remediation. This is a reactive security posture, but a critically important practice for agile teams and devops organizations striving to manage and improve the security of their applications.

Addressing software security requires a mix of proactive steps instituted at the beginning of the agile development process, best practices and instruments in the development pipeline, and reactive measures based on monitoring production systems. Security threats change rapidly, so agile teams and devops organizations need to review security practices and validate new methodologies continually.