SpiderLightning: Making WebAssembly cloud applications portable

The combination of WebAssembly and Kubernetes is an interesting development. Where traditional containers can be large and take time to deploy, even using slimmed-down dedicated host Linux distributions, WebAssembly applications simply require a standard runtime and, as they are dedicated binary files, need much less in the way of system resources. This makes them an attractive alternative to containers for applications that need to scale rapidly or must operate in constrained environments.

Microsoft’s support for WebAssembly in its Azure Kubernetes Service is a big vote of confidence in this technology, and its recent addition of support for the runwasi containerd shim simplifies deployment by using Kubernetes to manage WebAssembly code directly. That support builds on new WebAssembly application frameworks that help developers use WebAssembly, working with the Wasmtime runtime for the browserless WebAssembly System Interface (WASI).

Introducing SpiderLightning

One of those frameworks is Microsoft subsidiary Deis Labs’ SpiderLightning. Designed specifically for building distributed applications and named after a type of lightning bolt that can travel for hundreds of miles through clouds, SpiderLightning is an implementation of a set of common application interfaces for WASI applications, using the WIT interface definition language.

There’s a strong correlation between the design of WebAssembly (and more specifically, WASI) and the needs of developers building cloud-native distributed applications. It gets around the differences between the different cloud vendor platforms, but not as a way of avoiding lock-in and enabling portability. Instead, it focuses on how we can deliver a true service-agnostic multicloud platform that can reach from off-the-shelf edge hardware to hyperscalers, using everything from Raspberry Pi hardware to the latest multicore, multiprocessor servers from Intel, AMD, or Arm.

WASI gives us many of the pieces needed to deliver that vision, albeit in a fairly rudimentary form. That’s not surprising; We’re still in the very early days of a new platform and we shouldn’t expect it to have the maturity of something more than 20 years old, like the JVM or the .NET CLR. Even so, it’s clear that the platform’s designers have considered this and are providing the tools necessary to accelerate the development of platform extensions.

Extending WASI with WIT

A key element of this extensibility is the WebAssembly Component Model. Defined by the WebAssembly working group as the Wasm equivalent of an OS process model, it’s the foundation for how WASI implements its interfaces. A key element of any low-level approach like this is an interface definition language, which provides a way to specify how interfaces interact with code. For Wasm, and especially for the Component Model, the standard IDL is wit, which gives us a concise and human-readable way of defining interfaces that are expanded into WebAssembly code.

To use WASI to build distributed applications, we need a set of extensions that lets us abstract provider-specific services as interfaces. Instead of having to use separate APIs for S3 on AWS and Blob storage on Azure and the code to manage them, we could have a single storage component that would provide a common set of interfaces on all platforms, with the underlying WASI instance managing service-specific implementations.

This is where SpiderLightning comes in, with a set of interfaces that implement many common distributed application capabilities. You can write code that works with these interfaces and be sure that it’s portable. There’s no need to worry about infrastructure, and you can simply write the code that implements your business logic. Deis Labs describes SpiderLightning as “a set of Lego pieces” with compoapinents that provide features like key-value stores, gRPC APIs, message queues, and more.

Having a set of WIT definitions is only the start; to make a truly portable environment we need an implementation that’s designed around common cloud APIs and services. Deis has implemented a proof-of-concept SpiderLightning framework called slight that sits on top of the familiar wasmtime WASI runtime environment.

Getting started with slight

Like many parts of the cloud-native development toolchain, slight is a CLI tool. You can install it by executing a GitHub-hosted install script on a UNIX-based system (Windows developers can use Windows Subsystem for Linux). The script downloads a tar file that contains the slight binary then extracts and installs the CLI.

The tool will create and populate a WASI application. All you need to do is define the version of the SpiderLightning interfaces it will use. You can use either Rust or C, making sure that you have the appropriate compiler targets installed. Once your application is compiled, you’re able to run the code using the slight command, targeting the compiled Wasm file for your application and using a SpiderLightning configuration to map interfaces to implementations.

That configuration file is key to working with slight. It’s how you detail the capabilities your code needs to use, in the form of a resource type and a name. It’s important because it helps you switch between supported infrastructure features without changing your code. You can switch a storage provider to one that’s more suitable for your target environment. Code running on the edge might use local resources, one intended for your own private cloud could target a familiar element of your infrastructure, and one intended for a public cloud could use one of the provider’s platform services.

The result provides portable code that will run on AKS consuming Azure resources, at the same time scaling out to edge hardware and down to your own data center. It also helps you ensure that account information is kept outside your code repository, using the slight configuration to hold this data and manage connections.

For this to work, your application needs to import SpiderLightning’s WIT definition for the feature you’re using. This describes how your code needs to work with the interface: how it calls the service, the supported commands, what payloads are sent, and what returns to expect. The actual service used is handled by the slight runtime, allowing you to focus on the problems your code solves rather than the specifics of working with Azure or any other supported cloud.

Building new SpiderLightning capabilities

Currently SpiderLightning is very much a work in progress, with support for a subset of the planned capabilities and even then, with only some of the proposed services. The key-value store is the most mature at the moment, along with its messaging support. It’s an open source project and designed to be extendable, with a process for creating new dependencies for new services. With AKS support, there’s an incentive for Microsoft to add its own capabilities to the platform, and a rough road map suggests that these and a selection of AWS services are planned.

We’re still in the early days of using WebAssembly to build distributed applications, but tools like SpiderLightning show a lot of promise. By providing a level of abstraction from platform services, it’s an intriguing way to build portable cross-cloud applications that scale up and down as well as out. It’ll be interesting to watch the evolution of WASI tools and frameworks like SpiderLightning as they exchange ideas and concepts. The WebAssembly community is clearly aiming to deliver a standard set of tools to support portable code that can interact with as wide a range of host services as possible. SpiderLightning may well be a first step on a lengthy journey, but it’s clearly a long and confident stride.