An open-source collection of low-level tools helps you troubleshoot cloud-native applications by delivering key data from the heart of the Linux kernel. Credit: Shutterstock Platform engineering is becoming a compelling concept for enterprises, as they’re devoting increasingly large amounts of resources into cloud-native application development. It doesn’t matter if you’re using your own Kubernetes instances in your data centers, or working with managed environments in public clouds, you’re going to need to understand how they’re operating, taking advantage of a new generation of observability and security tools. Container orchestration tools like Kubernetes are hard to manage using traditional tools. As they instantiate new containers based on resource usage or in response to events, their behavior is not predictable. If you’re using an agent-based monitoring tool, then agents need to register with monitoring tools as containers start up, and de-register as containers stop. The process adds complexity to tools, and it’s hard to use the resulting data to get a complete root cause analysis, or to trace an anomalous activity. There are cloud-native concepts that help, like observability, but it’s becoming clear that we need a way to step outside the orchestration environment, while still tracking its compute and networking operations. With containers hosted by modern hypervisors or by low-level container daemons, we need a lower-level way of monitoring systems. Seeing from inside the Linux kernel In the past that would have involved working with kernel-level code, adding your own or third-party kernel modules, and recompiling the Linux kernel each time a new tool or an update comes along. If you’ve ever compiled Linux from scratch, or written a kernel module, you know that it’s not the easiest of tasks—and often it’s extremely time-consuming. If you’re using a ready-to-run image from a cloud provider, you’re unlikely to have the necessary permissions or access. Luckily, we have an option that avoids all that complexity, giving us controlled access to kernel operations, in the shape of eBPF, extended Berkeley Packet Filters. With eBPF we can inject code into the Linux (and Windows) kernel, using a verifier to ensure security. Hooks in the kernel generate events that trigger eBPF programs, allowing access to networking and other low-level operations. This approach allows monitoring tools to get data on the operations of the hosted Kubernetes environment, tracing key operations and allowing external applications like Prometheus, Cilium, and Retina to log the data, tying it to individual containers and to the applications that are orchestrated by Kubernetes. With eBPF support in the Linux kernel, you can use it to monitor not only physical instances on your own hardware, but also virtual machine hosts on a cloud service. Introducing the Inspektor Microsoft subsidiary Kinvolk is best known for its Flatcar Linux distribution, but it’s rapidly becoming one of Microsoft’s open-source centers of excellence. Whereas Deis Labs focused on cloud-native development, Kinvolk is more aligned with platform engineering and operations, providing tools like the Headlamp Kubernetes UI. Over the last five years or so, Kinvolk has been working on a set of eBPF monitoring and capture tools, bundled as a suite of kernel “gadgets” with user space management and display services. Named after the part-robot French detective, the Inspektor Gadget project offers a growing set of gadgets, as well as a framework for developing your own. The mix of gadgets ranges from tools that audit your Kubernetes configurations to others that capture performance information about your cluster, either for profiling or for real-time analysis. The most useful set allows you to trace a number of key metrics, including low-level networking operations. Your issue might not be DNS, but Inspektor Gadget can help you prove that it isn’t. While tools like Falco and Retina are intended for longer-term observations of a Kubernetes infrastructure, a stand-alone install of Inspektor Gadget is most useful for ad hoc investigations and explorations. That’s not a bad thing. Platform engineers need tools that can be used quickly to diagnose specific problems, without having to invest time in setting up and configuring a complete observability solution. With Inspektor Gadget you can get the information needed to show what’s going wrong and why, and then explore potential causes, all from the Kubernetes command line. The data you get back, while low-level, is associated with namespaces in your Kubernetes cluster, allowing you to quickly isolate specific pods and containers. You can use Inspektor Gadget on its own, or through a tool like Microsoft Defender for Containers, which recently added a sensor component that uses Inspektor Gadget tools to look for threats like container escapes. These are some of the riskiest compromises for Kubernetes, as they allow malicious code to break container isolation and access the host OS. Microsoft is using Inspektor Gadget to replace Defender for Containers tools that were previously powered by Sysdig’s eBPF probes. Running Inspektor Gadget in your cluster Getting started with Inspektor Gadget is easy enough. The toolkit installs both a kubectl plugin and a daemonset in your Kubernetes cluster. You can use the krew installer tool to install the kubectl tools, or install it manually by downloading a binary or compiling the source code and then adding the resulting executable to your path. Once you have the gadget command-line tool installed, you can deploy the daemonset using its deploy command, creating a pod for your gadgets and applying the appropriate security controls. You can deploy the daemonset to your entire cluster or to a specific node or nodes. It can then be configured to start alongside new containers. To uninstall, simply use the undeploy command. You don’t need to install Inspektor Gadget in Kubernetes. It can be installed as a Linux application on a host server, allowing you to trace containers from outside Kubernetes. While this approach isn’t suitable for a cloud-hosted Kubernetes, it can be useful for monitoring experimental installs and on-premises clusters. Once installed you can use Inspektor Gadget’s gadgets to install eBPF code in your host OS kernel, attaching the gadgets to specific kernel functions. The eBPF code runs when specific syscalls are made, allowing the gadget to trace the operation. Trace results are written to kernel buffers, and then read back by the userspace gadget code, displaying streamed data through the Kubernetes CLI. In the background a tracer manager keeps track of the various gadgets in use, with a map of the containers and namespaces they’re associated with, so data is associated directly with a specific container. Results are filtered by container and by event, giving you a way to quickly dive into the events you want to monitor. Inspektor Gadget provides a Prometheus gadget that delivers metrics to a Prometheus instance, allowing them to be visualized with tools like Grafana or analyzed by other Kubernetes management tools. Support for Prometheus allows you to run gadgets in the background, especially if you’re using network tracers or profiling tools to get a picture of the overall performance of an application and a cluster, as well as the underlying network. Extending the gadgets Usefully Inspektor Gadget can work with other eBPF programs, adding additional information to basic queries. You can write a simple eBPF probe to look for a specific event, and then call Inspektor Gadget to add details of the current mount namespace to the event log. This way you can quickly add code to look for new critical vulnerabilities, allowing you to pass that data to your monitoring application via the Inspektor Gadget CLI. Tools like Inspektor Gadget are important for dealing with the unpredictable nature of cloud-native applications. By using eBPF we can get down into the weeds of complex networking meshes and understand exactly what is happening when containers communicate with each other, allowing us to build the right infrastructure for our code. There’s a lot to like here, both in how Inspektor Gadget integrates with familiar Kubernetes tools and in how it can be extended with your own gadgets to address your own issues. As Kubernetes matures, it’s important to have ways of seeing what happens under the hood. Without that data we can’t be sure that we’re delivering the enterprise-grade architectures that are essential to support our code and our users. eBPF and Inspektor Gadget are an important way of delivering that maturity, alongside observability tools that allow us to interpret and analyze the data and deliver a manageable cloud-native platform. Related content feature 14 great preprocessors for developers who love to code Sometimes it seems like the rules of programming are designed to make coding a chore. Here are 14 ways preprocessors can help make software development fun again. By Peter Wayner Nov 18, 2024 10 mins Development Tools Software Development feature Designing the APIs that accidentally power businesses Well-designed APIs, even those often-neglected internal APIs, make developers more productive and businesses more agile. By Jean Yang Nov 18, 2024 6 mins APIs Software Development news Spin 3.0 supports polyglot development using Wasm components Fermyon’s open source framework for building server-side WebAssembly apps allows developers to compose apps from components created with different languages. By Paul Krill Nov 18, 2024 2 mins Microservices Serverless Computing Development Libraries and Frameworks news Go language evolving for future hardware, AI workloads The Go team is working to adapt Go to large multicore systems, the latest hardware instructions, and the needs of developers of large-scale AI systems. By Paul Krill Nov 15, 2024 3 mins Google Go Generative AI Programming Languages Resources Videos