How to improve CI/CD with shift-left testing

Testing applications used to be a technically challenging, time-crunched activity scheduled days or weeks before an application’s release. Development teams were given the leeway to code until the eleventh hour, and testers, who did much of their work manually, had little choice but to make do with the bit of time given to them. The result was that many applications underwent substandard testing, and technology teams were forced to respond to the production issues and defects escalated by end-users and application monitoring systems.

Devops continuous integration practices, unit testing frameworks, and test automation practices have upended this paradigm. Instead of performing quality assurance at the end of the development process, many testing practices now start and fully execute during coding, integration, and deployment. Devops and agile teams automate testing scripts, and CI/CD pipelines call out to run the tests during their code integration or delivery phases. The net result is that developers are alerted when their code changes break the build and can take immediate steps to address the reported issue.

Automating testing and integrating testing scripts into CI/CD pipelines is known as shift-left testing. It implies that more quality assurance practices can be done in the development phase to catch issues earlier in the release timeline. Automating testing is one of the predeployment priorities for agile and devops teams who want to increase deployment frequencies.

At the introduction of new functionality, the constructed test scripts validate the new capabilities. These tests can then be automated and included in the build or deploy steps. Instead of having QA engineers run regression tests at the end of a release process, you can run and validate many of these tests as part of development. These tests shift left from the end of the release process to earlier development and coding phases.

Shift-left testing enables agile teams’ commitment to quality

Shift-left testing not only drives efficiency and improves quality, it also creates a significant culture change in the agile development process.

Some development teams perceive quality assurance and testing as a barrier to getting their code delivered to production. After all the hard work in satisfying agile product owners and completing the code, QA teammates identify a list of bugs requiring remediation. If QA finds a lot of bugs, it can impact the release timeline to fix them. Even worse is when significant sections of the code need re-engineering because flaws expose logic, security, or performance issues. In this scenario, developers and QA engineers may be on the same agile team but are not acting as a team.

Shift-left testing enables agile teams to shift quality responsibilities to the full team of developers and testers. When testing runs as part of the CI/CD pipeline, it provides a better opportunity for developers to address quality issues at the point in time when they are working on the relevant code. The CI/CD pipeline alerts the developer of the failed build, and most self-organizing development teams require fixing these issues immediately.

Shift-left testing also provides developers and QA engineers opportunities to automate more of the testing. A best practice is for teams to decide who implements the automation depending on the types of tests required for the functionality developed. For example, developers are often responsible for automating unit and API tests, but QA automation engineers often develop end-to-end user experience testing and transaction tests that simulate multistep API calls to multiple services.

When to apply shift-left testing

Shift-left testing works best for the more unit-level, atomic tests that have short execution times. It’s essential that tests automated in the CI/CD pipeline, and that run whenever developers trigger a build, execute quickly and not slow the build processes.

More complex and time-intensive tests such as end-to-end user experience tests, transaction testing, static code analysis, and security testing often run better outside of CI/CD pipelines and on daily or more frequent schedules. These tests still provide early feedback to developers on quality issues, but they are automated outside of CI/CD to avoid slowing or bottlenecking builds.

Thomas J. Sweet, a VP in IT Services at GM Financial, shared with me his personal insights on the limits of shift-left testing strategies. He suggests, “Shift left is always a strategy, except for when performing integration testing on third-party vendor deliveries, as you often do not have access to their source code. Even when you have internal apps with legacy monolithic architectures, you can start by enforcing basic check-in policies that require a code review and security scan. The check-in should be rejected if the scan includes essential warnings and failures.”

One potential solution to downstream testing with integration partners is to implement service virtualization. This technique enables development teams to simulate a downstream system’s response to different inputs. It works well when the downstream systems are well defined. Tools from Micro Focus, Tricentis, and others enable this capability.

Rob Pociluk, a highly seasoned quality assurance manager, is a strong proponent of shift-left testing in agile development. “Being ready and able to test small sections of code keeps QA flexible and in the loop during the progress of a sprint. Teams should still guard against using shift-left too much as you can lose the purpose of the code itself.”

So, even when teams fully embrace shift-left testing, there are good reasons to still schedule a testing window on a code-complete build targeted for release. It ensures that all automated tests get performed on the final build, but also enables scheduling additional testing that requires a fully functioning system.

One of those tests is UAT (user acceptance testing), where selected end-users and subject matter experts validate and provide feedback. Some UAT can be done during development, but it may not be easy to get people to perform this testing frequently or when functionality isn’t fully ready.

Prerequisites to shift-left testing strategies

Shift-left testing is a growing devops practice, but it has its prerequisites and upfront investment. Some essential capabilities and practices are required.

• Sufficient testing capacity and environments are needed to support the number of builds and tests that run concurrently.
• Agile teams require a toolkit of testing products that easily integrate into CI/CD pipelines and job scheduling tools, and that can validate functionality, code quality, security, and performance.
• Architects, infosec specialists, QA leads, and other senior members of the organization should establish testing standards and service-level objectives that form the default acceptance criteria.
• When applications require user input, testing teams need sufficient test data and patterns to validate enough personas, use cases, and input patterns.
• At sprint commitment or earlier, scrum teams, including QA test automation engineers, should set a testing strategy on what capabilities get tested, which types of testing are implemented, what automation processes are updated, and who develops the tests.
• Devops teams should measure the duration of CI/CD pipeline runs and flag when automated testing steps impact productivity. Devops teams often require additional testing schedules outside of CI/CD pipelines to execute longer-running tests.
• Teams should regularly discuss gaps in their automated tests, especially validations that require subject matter experts, UAT, or testing with partners. If agile teams cannot address these gaps with automation, then release cycles should factor in the overhead to reduce risks and complete these tests.

Lastly, agile teams and devops organizations should regularly measure and discuss their test coverage. Employing a shift-left testing strategy doesn’t work if the development teams and quality automation engineers do not actually implement, automate, and integrate sufficient tests to catch issues and address risks.

Speeding up release cycles or enabling continuous delivery without sufficient test automation can result in significant quality issues that degrade the experience for end-users. Agile development teams pushing releases too frequently then find themselves addressing production issues and defects instead of investing in more and better automation.