The cloud as supercomputer

YellowDog, a cloud workload management specialist based in the United Kingdom, assembled a virtual supercomputer using many cloud-based servers. At its peak, which lasted about 10 minutes, the system had leveraged more than 3.2 million virtual CPUs. To be more precise, 33,333 AWS 96-core C5 24xlarge (bare metal) instances. This is one of several instances used during the run and costs $1.6013 per hour, which they used for a six-hour run time.

The reason for the tossed-together, widely distributed supercomputer was to run a drug discovery application as a single cluster, solving many problems quickly. All for about $60K.

If you think this sounds excessive, high-performance computing geeks like me who used supercomputers back in the ’80s and ’90s were looking at a total bill of many millions of dollars, at a minimum, to do about a tenth of what they are doing here. By using this on-demand cloud-based supercomputer, the researchers were able to analyze 337 million compounds in just six hours.

We’re going to start seeing more of these high-performance computing uses of cloud-based platforms. Enterprises understand that the cloud is an economical place to run traditional business applications and store data, and that the purpose-built and highly expensive world of supercomputers can exist in the cloud as well. Moreover, it’s not only that traditional supercomputers can be replaced; they are able to compress the time it takes for deep analytics, such as the drug discovery application but also exploration, medical research, advanced simulations, and other compute-intensive applications.

Small businesses are no longer limited by the price tag of a supercomputer. This allows them to punch above their weight and even leverage this technology as a force multiplier to create market innovations that the larger players may not even think of. I suspect disruptions will be the norm during the next few years, with smaller companies eating the lunch of larger companies that have been market leaders for 100 years or more. Previously, only the big guys had the resources to buy or lease supercomputers. Now it does not matter.

Also, this creates a new category of cloud talent: those who specialize in putting together these widely distributed compute instances that can be coupled to support intense processing needs. High-performance computing skills are already in silent demand as companies attempt to lock up some of the more experienced players out there.

Cloud-based platforms have been moving beyond what we could accomplish on-premises for the last several years. Innovation budgets are focusing on cloud computing, and I suspect that the more traditional on-premises systems are suffering because of it.

You can fight progress, but in this case, embrace it.