Our cloud complexity bias may be hurting business

I was listening to my podcast feed again this weekend, and beyond the true crime shows, I did listen to something more work related. The question asked on this episode was a profound one, not typically asked in the tech press today: “Cloud was supposed to make computing easier, but it’s now as complicated or more complicated than legacy data centers and apps. Is there any future in a simpler cloud?”

Those of you who have followed me here for a while or taken my courses understand that I’ve been attempting to figure out the balance between making cloud architectures complex versus making them optimized and efficient. The more I’ve researched this space, the more I think I’m onto something: We need to understand what the trade-offs are.

Core to this issue may be a people problem, not a technology one. Most architects build and deploy cloud solutions that are often too complex and too costly. They do so influenced by a few conscious and unconscious biases.

No need to look further than complexity bias: “Faced with two competing hypotheses, we are likely to choose the most complex one. That’s usually the option with the most assumptions and regressions. As a result, when we need to solve a problem, we may ignore simple solutions—thinking “that will never work”—and instead favor complex ones.”

I’m not an expert to opine on the psychological issues of making things, including cloud architectures, too complex. It’s interesting that the simpler solutions with the fewest moving parts (cloud services) are typically far better than attempting to push every type of technology into the final deployed architecture. Don’t choose four types of storage when two will do. Opting for 10 different cloud-native databases because some of those have features that may be needed at some point in the future…well, maybe.

The issue is that complex architecture works just fine—initially. However, it costs three to six times more to build, deploy, and operate. There is no failsafe in terms of other company executives pointing out that although the solution is needed, it costs too much because it’s way too convoluted and overengineered. In other words, cloud architects get away with it, and are likely praised for deploying a solution where innovation is mistaken for overly complex.

Cloud architects (like myself) who favor simplicity or abstraction and automation to manage complexity that is unavoidable need to find a balance with those who naturally gravitate to overly complicated cloud architectures. Also, I prefer almost fully optimized and minimum viable solutions, which I know to work better than complex ones.

I suspect that a few things will likely occur: 

First, just through trial and error, those who design and build cloud solutions in an overly complex and costly manner will be identified and their negative impact managed better. This is why I always insist on peer reviews of cloud solutions in order to have some checks and balances. Unfortunately, for most enterprises, internal or external reviews are more the exception than the rule.

Second, postmortems on IT/cloud disasters will become more common. Could excess cloud complexity have caused security operations issues that led to accidental data exposure? What happens if an investor audit identifies “complexity and cost issues” that result in a whole new IT executive team? Neither of these options is good for the business.

Is it time to start thinking about how to reduce complexity? I think so.