12 principles for improving devsecops

I once transitioned from a SaaS CTO role to become a business unit CIO at a Fortune 100 enterprise that aimed to bring startup development processes, technology, and culture into the organization. The executives recognized the importance of developing customer-facing applications, game-changing analytics capabilities, and more automated workflows.

Let’s just say my team and I did a lot of teaching on agile development and nimble architectures. But we also had a lot to learn about deploying highly reliable, performant, and secure applications to our data centers. This was all before the days of cloud computing and devsecops.

Today, while many enterprises and businesses have robust software development and devops capabilities, SaaS companies have developed greater expertise in scaling applications, handling highly disparate customer use cases, and identifying performance and security incidents before they become customer issues. 

“CTOs understand the value of a product that is not only functionally robust but also consistently available, lightning-fast, and impregnable to security threats,” says Raghav Gurumani, CTO of SaaS company Zuper. “CTOs can teach enterprise teams to achieve this iron triangle of reliability, performance, and security by emphasizing the importance of striking a balance between the three and leveraging iterative approaches.”

In this article, I share 12 principles recommended by SaaS technology leaders that business IT leaders can apply to devsecops. I’ve grouped these principles into three areas:

• Shift-left operational practices into requirements and development, starting by adopting a customer-first mindset.
• Recognize that devsecops teams must do more test automation beyond unit testing to ensure application reliability and performance, especially with high usage and many user types.
• Achieve higher performance and reliability by defining service level objectives (SLOs) and leveraging tools for observability, monitoring, and cloud automation.

1. Adopt a customer-first mindset

Developing a customer-centric mindset and sensitivity to client needs is a must to retain customers and support growth. While many businesses develop customer-facing applications, devsecops teams also must learn to treat fellow employees as customers when developing internal applications.

A customer-first mindset includes a “non-negotiable focus on quick identification and remediation of customer issues,” says Claire Vo, chief product officer of LaunchDarkly. “Engineers must be as customer-centric as product, design, sales, and support, which means they spend time talking directly to customers, use the product as customers do, and hold a high-quality bar on behalf of customers. In my experience, having close customer relationships is highly correlated with high-quality, resilient cultures.”

Recommendation: Devsecops teams should schedule regular meetings with end-users to observe how they use applications and listen for ways to improve application performance.

2. Connect version control to agile user stories

While most enterprises have adopted version control,  David Brooks, SVP of evangelism at Copado, says developers tend to focus too much on the branch management in their repository. “Modern development is based on agile, and many devops tools manage changes directly based on user stories.”

Brooks says that tracking changes by starting with the user stories makes it easier for agile development teams to focus on delivering value, support test-driven development, and enable automated merge conflict resolution.

Recommendation: Besides connecting workflows between agile tools and version control, devsecops teams should consider standardizing CI/CD pipelines, developing with feature flags, and leveraging canary release strategies.

3. Release new features to alpha groups

Investing in devsecops automation provides flexibility in releasing features to small user groups and performing A/B testing on feature implementations. The automation can support continuous deployment, but an even more important benefit is the ability to validate features before overinvesting and capture end-user feedback during development.

“New feature alphas are being tested out in the open and at a rapid-fire pace to gather customer feedback, says Elliot Wood, CTO and co-founder of CallRail. “Teams are empowered to move fast because they’re able to ship small changes to limited sets of customers and minimize the risk of each individual experiment.”

Recommendation: Alpha and beta testing, where alpha testing happens internally within the organization and beta testing focuses on the user’s environment, is a long-standing software development practice. Devsecops brings automation and scalability practices to operationalize the technology operations. However, the key to successful alpha and beta programs is recruiting participants, communicating goals, capturing actionable feedback, and rewarding collaboration.

4. Require security by design

While many enterprises have robust information security programs, implementing security by design in the software development process remains challenging. Security best practices include automating penetration testing, triggering code scanning in CI/CD pipelines, and protecting APIs from injections, authentication flaws, cross-site issues, API leaks, and broken access controls.

Steve Touw, CTO at Immuta, says, “By implementing security by design and applying security as early as possible in our own product development, we found that our back-end maintenance and management were noticeably reduced from a vulnerability management standpoint.”

Recommendation: CIOs, CISOs, and delivery managers should clearly define non-negotiable requirements regarding what security practices, tests, and metrics are required when automating paths to production.

5. Recognize unit testing is insufficient

You can check the tire pressure, look at the oil level, and perform dozens of other tests on your car engine. But does the car meet your expectations in handling curves, bumps, and other road conditions?

The same can be said for software applications, and while unit tests help validate components and interfaces, they are insufficient for validating end-to-end functionality or user experience.

“Embracing a shift-left approach, developers often prioritize unit testing to ensure features and functionality work correctly,” says Peter McKee, head of developer relations and community at Sonar. “However, relying solely on unit testing may leave gaps in quality assurance, allowing bugs to slip through unnoticed. This compromises both the quality and security of software upon deployment.”

Recommendation: Many tools can automate front-end user experience testing, and a key requirement for agile development teams is to assign the responsibility, develop the skills, and invest the time to ensure robust functional testing.

6. Automate tests from subject matter experts

A commitment to more functional testing is only as good as the developed test cases. Quality assurance engineers have the talent to identify boundary conditions and the skill to test for error conditions, but they need guidance from end-users to better understand their goals, workflow, and journeys.

Brooks of Copado says, “Developers are concerned with delivering code that works as requested, but to ensure robust software that works with all variations of customer configuration, subject matter experts (SMEs) must generate tests illustrating how users use features in the real world. The best way is for the SMEs to perform exploratory testing using a tool to capture the steps and then create automated tests.”

Recommendation: Leverage the same alpha and beta groups to be part of the apps testing community, but don’t expect testers to perform repetitive user acceptance testing. Use tools to capture their testing patterns, automate the most important tests, develop a continuous testing strategy, and leverage synthetic data to scale test patterns.

7. Validate code for security and quality

Using copilots and other genAI code generators has increased the importance of reviewing code for vulnerabilities and flagging issues that may become tomorrow’s technical debt. Other code quality issues that should be flagged before code makes its way to production include checking for proper documentation, error conditions, logging, and naming conventions.

“To bolster QA efforts, developers should integrate static code analysis into their workflow,” says McKee of Sonar. “Automated static analysis examines the internal structure of an application, complementing unit testing by uncovering additional issues. Combining both, developers can proactively manage code quality throughout the development lifecycle, swiftly identify and address bugs, and enhance overall software reliability and security.”

Recommendation: Reducing technical debt is a major issue for enterprises, so finding tools that scan for security and code quality issues and integrating the steps in CI/CD should be a non-negotiable requirement.

8. Establish nonfunctional operational requirements

When considering the iron triangle of performance, reliability, and security, it’s important to identify requirements specifying acceptable operating conditions. Development teams often express these as nonfunctional requirements that can be expressed inside agile user stories as acceptance criteria. Nonfunctional requirements can also guide how infrastructure components are selected and managed.

“Nonfunctional operational requirements are equally important as the functional requirements,” says David Coffey, VP of product management in software networking and NS1 chief product officer at IBM. “Everything matters in a tech stack for a cloud service, and overlooking details like which DNS service or network connectivity can adversely impact the availability and scale of a cloud service.”

Recommendation: Architects, operations, and security experts should draft standards on nonfunctional requirements and acceptance criteria that agile development teams reference from their user stories.

9. Channel SLOs and alert priorities meaningfully

An outdated method for setting expectations on application performance is expressing target service level agreements (SLAs) around an operating metric, such as 99.9% uptime. A modernized approach is to define SLOs and error budgets that govern when devsecops teams should prioritize operational improvements.

“If engineers drive performance remediation without a broad vision of which SLOs matter—the ones tied to user satisfaction profit—then we’re limiting any success that we can expect from an observability platform,” says Asaf Yigal, co-founder and VP of product at Logz.io. “CTOs have the responsibility not only to drive meaningful results from the engineering team, but also reduce stress for engineers by setting alert priorities that clearly identify the vital role they play in the success of the business.”

Recommendation: Product managers should be involved in setting SLOs and use customer segments, journey types, and critical periods to express when outages or poor performance have more significant business impacts.

10. Implement observability and monitor data pipelines

Most applications today leverage data pipelines to connect other data sources and move data in and out of the application’s environment. Workflows can be compromised, and there’s the risk of making erroneous decisions when there are pipeline delays and data quality issues.

“Shifting left in data reliability checks allows for early validation of data quality and integrity at the source and avoids costly repercussions,” says Ashwin Rajeeva, co-founder and CTO of Acceldata. “Continuous monitoring and incident management facilitate proactive responses to potential data incidents, enabling uninterrupted data flow and ensuring the continuous reliability of data across the data supply chain.

Recommendation: Rajeeva recommends implementing comprehensive monitoring across the entire data supply chain and employing proactive, automated checks and alerts to identify pipeline performance issues and data quality anomalies.

11. Lock down admin controls and external access

SaaS companies lock down administrative access to their applications and environments to avoid exposing customer data or compromising application availability. The same is true for businesses, and security specialists must review the business’s proprietary applications’ administrative functions, admin roles, access rights, and audit controls, especially when they contain sensitive data. 

“Create dual admin controls over critical functions in your applications so your admins can’t make single-handed mistakes that essentially change the uptime and availability of your platform,” says Igor Jablokov, founder and CEO of Pryon.

Jablokov also recommends other fundamentals, including implementing multifactor authentication and locking down unnecessary external access.

Recommendation: Information security specialists should review the latest vulnerabilities, such as the OWASP Top 10 and the SANS 25 most dangerous software errors. These can be used to provide security checklists, training, and support to devsecops teams.

12. Configure hot standby environments

Businesses deploying applications should ensure a robust cloud infrastructure by deploying infrastructure as code, using cloud automation to scale environments, configuring multizone deployments, and automating failovers. While these are standard practices for deploying enterprise applications, many applications developed in business units and departments may have been deployed without these infrastructure best practices.

Jablokov of Pryon adds, “Have your application running in hot standby in another cloud vendor because no matter how much redundancy and failover that got built into the hyperscalers, they’re not impervious to faults on their own.”

Recommendation: Devsecops teams who develop standard architectures, platforms, and configurations can more easily bake high availability practices into their infrastructure patterns.

Balance is key

The principles and best practices presented here are guidelines for devsecops teams. The hard part of improving application reliability, performance, and security is prioritizing which operational areas need investment and balancing this work with functional requirements. Agile development teams that track operational metrics, debate priorities, and track where they make investments are more likely to deliver better experiences and operational performance.