10 more big devops gotchas to watch out for

For a recent article, 10 big devops mistakes and how to avoid them, I interviewed industry leaders to uncover 10 devops “gotchas” that can sabotage your software development efforts. Those interviews turned up more mistakes than I could cover in one article, so here are 10 more potential roadblocks for devops teams and IT leaders to avoid.

10 more ways IT teams and leaders fail with devops

1. Resistance to change
2. Missing skill sets
3. Shadow IT
4. Quantum complexity
5. Gaps in monitoring and observability
6. Low-quality deliverables
7. Runaway costs
8. The backlog keeps growing
9. The human impact is overlooked
10. Management can’t keep up

Resistance to change

The devops rollout can be jarring, and some team members across the organization will struggle to accept the changes. Devops proponents need to get buy-in, not only from executive decision-makers but from development and operations teams.

“People get comfortable with their way of working, so devops can meet resistance,” says Anh Nguyen, a software developer and founder of footwear provider 365 Crocs. “Start small with a pilot to get the ball rolling and show people the benefits firsthand.”

Sean Spittle, lead software developer and managing partner at InspectNTrack, which provides a cloud-based product for managing safety devices, notes that developers, operations, security, and other teams all may resist adopting new processes and tools.

“Leadership needs to clearly communicate the benefits of devops and get buy-in across the organization,” Spittle says. “Providing training, celebrating successes, and designating devops champions on each team can help drive the cultural shift.”

Missing skill sets

Successful devops requires a wide range of skills, which many organizations don’t have in their workforce at the outset.

The Computing Technology Industry Association (CompTIA), a group that advocates for IT professionals, says key skills needed for devops engineers include Linux fundamentals and scripting, knowledge of relevant tools and technologies, cloud services, coding, automation, testing, security, proactive monitoring, containerization, continuous integration, and version management.

“Finding or training professionals with the necessary knowledge can take time and effort,” says Taylor Dolezai, CIO and head of ecosystems at Cloud Native Computing Foundation (CNCF), part of the Linux Foundation that runs open source developer conferences and hosts critical components of software stacks such as Kubernetes.

“Leaders should invest in upskilling their existing teams through training programs, partnerships with devops service providers, and promoting a culture of continuous learning,” Dolezai says. “Encouraging close collaboration between development and operations teams can also foster skill-sharing and a more holistic understanding of devops practices.”

There are many devops-related certifications available that can help provide the necessary skill sets for a devops implementation. But companies can also look beyond formal training and certification to obtain the skills needed.

“Formal training isn’t everything,” says Taimur Ijlal, an information security leader at Netify, a marketplace specializing in SD-WAN and cybersecurity. “Learning through trials and testing projects cultivates knowledge over time. What helps most is a clear vision for transforming how work gets done.”

Shadow IT

The deployment of IT systems by departments or groups other than the central IT department—shadow IT—remains a big risk for organizations. The growth of the cloud in particular has enabled many within organizations to procure software-as-a-service (SaaS) offerings or other products without oversight by technology leaders.

“Many organizations are trapped in a complex underworld of legacy systems, undocumented processes, and rogue applications,” says Maksim Muravev, devops engineer at game developer and publisher Wargaming.

“These systems are not merely technological artifacts but are deeply entrenched in the organization’s cultural and operational fabric.”

The challenge is not to eradicate these shadow systems “but to understand their purpose, integrate them where possible, and provide superior, compliant alternatives where necessary,” Muravev says. “This demands a devops leader who is not just a technologist but an anthropologist and psychologist, capable of navigating the intricate human networks and their affinity to these systems.”

Quantum complexity

Devops involves many moving parts, as well as connections among these parts. This can render decision-making complex because a change in one area will impact multiple others, possibly in unexpected ways.

“Decision-making in devops environments is akin to the [physics] principle of quantum entanglement, where choices made in one part of the system can have instantaneous effects across the entire organization,” Muravev says. “This interconnectedness, especially in large-scale deployments, can create unpredictable outcomes from seemingly straightforward decisions.”

Addressing this challenge requires an approach Muravev refers to as “quantum leadership,” or the ability to make informed decisions by considering a spectrum of outcomes and their probabilities.

“This necessitates investing in predictive analytics, fostering a culture of continuous learning, and developing a resilient mindset capable of adapting to rapid, sometimes unexpected results,” Muravev says.

Gaps in monitoring and observability

Those same moving parts require frequent or constant oversight in a devops system, to quickly catch problems before they significant impact on development projects.

“With devops, changes are deployed to production much more frequently,” Spittle says. “This makes robust monitoring critical to proactively detect issues. But many monitoring tools aren’t well-suited for dynamic, distributed applications.”

Teams need to instrument their applications and infrastructure to collect granular metrics and logs, Spittle says. Solutions such as artificial intelligence operations (AIOps) platforms—machine learning analytics technology that enhances IT operations analytics—can help correlate data to swiftly pinpoint the root cause of problems, Spittle says.

“The goal is unified observability across the entire stack,” Spittle says.

Low-quality deliverables

Devops can certainly bring improvements to the software development process. But nothing guarantees the final product will stand up to quality demands. That’s especially true at a time when users are more discerning about the products they buy.

“A pervasive issue within centralized devops teams is the delivery of work that, while functional, falls short of optimal standards in terms of optimization and reusability,” says Dan Schaefer, an independent software developer and devops engineer.

“The root of this challenge often lies in the approach to problem-solving,” Schaefer says. “Solutions are crafted to address immediate issues without considering long-term implications or potential for reuse. Such practices lead to a codebase that is difficult to maintain and scale.”

This not only impacts the efficiency and agility of the devops practice, Schaefer says, but also detracts from the overall quality and sustainability of the software development lifecycle. “I frequently find myself compelled to refactor infrastructure-as-code and pipeline code to enhance cleanliness, configurability, and reusability,” he says.

Runaway costs

Increasingly complex and diverse devops environments, which might include cloud, on-premises, or hybrid systems, can lead to cost overruns that might negate some of the gains from deploying devops in the first place.

“Integrating devops practices also brings financial challenges, particularly in cloud environments where costs can quickly spiral,” says Andy Lipnitski, IT Director at Science Soft, a provider of IT consulting and software development services.

“We implement rigorous monitoring and budgeting controls to keep expenses in check,” Lipnitski says. “We also leverage solutions that dynamically adjust resources based on demand, rightsize the cloud resources, [choose] the right storage type, and more to optimize operational costs without sacrificing performance or scalability.”

The backlog keeps growing

Another significant challenge is managing the extensive backlog that can arise from a centralized devops team’s need to accommodate requests from multiple stakeholders, Schaefer says.

These include various development teams, management, and product teams. “This often results in a prioritization conundrum, where the most vocal requesters get their needs met first, rather than those aligning most closely with strategic business objectives,” Schaefer says.

Such a scenario can lead to suboptimal prioritization, delayed projects, and a general sense of frustration among team members who feel their projects are sidelined, Schaefer says.

The human impact is overlooked

For many organizations, a big part of devops is introducing automation into the software development and delivery process. But that can present a challenge in terms of understanding and responding to the human element of system interactions, Muravev says.

“Each line of code, deployment pipeline, and automated process touches humans differently, impacting work routines, job security, and morale,” Muravev says.

To address this, organizations need to cultivate “automation empathy,” a deep understanding of automation’s human impact, Muravev says. “This involves engaging with stakeholders at all levels, incorporating feedback loops that gauge emotional response to automation, and tailoring automation strategies to enhance human value rather than replace it,” he says.

Management can’t keep up

“Management would spend half a year creating a detailed project plan for the coming year, while the waterfall project teams executed on the previous year’s commitments,” says Dan Krantz, CIO at Keysight Technologies, a provider of electronics test and measurement equipment and software.

“This methodology prevented nimble pivoting week to week or month to month, as conditions warranted,” Krantz says. “Shifting to fusion product teams requires management to abandon their comfort zone of prescribing an annual project portfolio to define key business outcomes for each digital product.”

That means setting key performance indicator (KPI) targets for each team, regularly reviewing KPI achievement, and adjusting developer capacity as needed, Krantz says. “The devops teams need to self-organize their sprints to achieve the defined targets that management reviews and adjusts continuously,” he says.