Getting started with MQTT in Azure Event Grid

MQTT is an important technology for the industrial internet of things (IIoT), building on concepts from IBM’s venerable MQ Series message queue technology. MQTT was initially designed to deliver telemetry from SCADA control systems, with IBM handing the protocol over to the OASIS standards body in 2013.

The standard is deliberately intended to evolve slowly, as it’s embedded in industrial device firmware, and used in hardware that may not get updates—ever. That’s because organizations typically deploy not just tens, or even mere hundreds of MQTT-enabled systems, but many thousands. Plus, MQTT devices are often deployed in inhospitable and hard-to-reach environments, like undersea pipelines, with rollouts often lasting years. MQTT is also relatively simple, with implementations for most common microcontrollers.

MQTT support in Azure Event Grid

Because MQTT is a publish-and-subscribe protocol, where endpoints publish messages that listeners subscribe to, it’s an obvious fit for Azure Event Grid, Microsoft’s pub-sub message handling service. Designed to scale to support massive device deployments, Event Grid is perhaps best thought of as a message routing broker, supporting IIoT and other event-driven applications, feeding events from devices to your applications and to Azure services. While Event Grid is perhaps best known for its implementation of the Cloud Events protocol, the service is able to support many different messaging standards. (Read more about Azure Even Grid here.)

Azure Event Grid’s protocol support now includes a public preview of MQTT, with support for MQTT 5 and MQTT 3.1 unveiled at BUILD 2023. MQTT support for both incoming and outgoing messages means Event Grid can serve as the hub of an IIoT control system. Events sourced from edge devices can be used to deliver new events to both sources and MQTT-ready applications, as well as in Azure’s own stream analytics tooling. Those events could also be stored in Azure Data Lake, where analysts can use tools like Data Explorer to extract insights from device data and use that data to train machine learning-powered control systems.

Azure Event Grid is an important component of any large IoT infrastructure, whether you are supporting consumer or enterprise devices. That’s partly because it’s an implementation of a many-to-one messaging pattern, allowing architectures to consume many thousands of inputs with Event Grid as message manager. Because Event Grid is a two-way architecture, applications can use it to broadcast alerts and information to selected clients. You can even use Event Grid as a relay so that a message from one client can be broadcasted to all clients, or to a distinct subset. Microsoft has developed a reference architecture to show how Event Grid might be used in practice.

The result is a flexible way of connecting many devices in a hub-and-spoke network, where clients and services are linked by a scalable broker that manages authentication and authorization, reducing the work needed to build and secure services, and encapsulating functionality in defined namespaces. Namespaces are a useful tool for managing messages at scale, as they allow you to group clients and then wrap their associated topics into topic spaces. This then lets you apply permissions at a granular level so that clients need authorization before they can publish or subscribe to a topic.

Using MQTT in Azure Event Grid

Once MQTT messages are delivered to Azure Event Grid they can be routed to Azure services using built-in APIs. Custom services and your own code can use webhooks to receive messages and then process them accordingly. Some MQTT message types and features aren’t yet supported by Event Grid. One missing feature, support for message ordering, might cause problems. If so, you will need to add your own code to ensure that messages are processed in the correct sequence.

There’s a lot of scale in Azure Event Grid’s MQTT support. Each namespace handles up to 200,000 MQTT clients, delivering 20,000 messages a second. That’s only the preview release, too, as Microsoft has documented plans to rapidly increase this to 1 million clients and 100,000 messages a second.

Working with Event Grid is relatively simple. You can accomplish most of the tasks in the Azure Portal, though you can use the Azure CLI if you prefer (and if you want to build reusable scripts for future operations).

Building a MQTT broker

As your endpoint devices will be using MQTT to connect to Azure resources, start by opening port 8333 in your firewall, both from your network and into the Azure VNet used for your application. This is the standard port for MQTT and should allow any compliant device to connect to your Event Grid. It’s a good idea to use an X.509 certificate to authenticate client connections. You can generate X509 certificates using a tool like the open-source Step certificate authority on most platforms, including Windows, Mac, and Linux.

In the Azure Portal start by creating an Azure Event Grid Namespace in the resource group you’re using for your MQTT application. Namespaces are DNS entries, so they need to be unique to an Azure region. It’s a good idea to use names that are related to the purpose of the application you’re building, so use the name to tag it as one that supports MQTT. Finally, choose a region for the namespace before creating it.

Once the namespace has been created, enable MQTT support from its configuration page in the portal. You can now start to add clients to your Event Grid. For simple test applications with a handful of clients, you can add them using the UI, but for larger deployments consider automating the process with a script and using generated names for clients. You should tag each client with the thumbprint from your X.509 certificate, as this will be used for authentication.

You can now start to add Topic Spaces to your Event Grid, along with the filters used to select messages that have been published into its topics. With Topic Spaces in place, you can add permissions for clients, giving them publisher access to the topic space.

Configuring MQTT clients

Clients will need to be configured with the certificate you’ve created and will then use the topic names you’ve added to Azure Event Grid to publish messages. You’re now ready to add subscriptions to these topics, for example adding a connection to an Event Hub in Event Grid to automatically translate MQTT content into other formats, such as Cloud Events. This is perhaps one of the more useful aspects of Azure Event Grid’s MQTT support, as it moves messages and events out of the realm of operational technology into more flexible protocols.

Of course, if you’re working with existing MQTT systems, you can configure your Event Grid with the fingerprint of existing certificates. This will allow you to upgrade without major updates to device firmware.

As Event Grid is a publish-and-subscribe service in its own right, routing to alternative protocols requires mapping your MQTT topics to Event Grid topics. As part of this process you’re able to add new properties to messages, which can be used to enhance Event Grid-sourced Cloud Events messages received by tools like Azure Event Hubs, for example providing content metadata that isn’t present in a basic MQTT message from a remote device that can then be used by Stream Analytics or as additional labels for a machine learning model.

Adding MQTT support to Azure Event Grid is a sensible move by Microsoft’s Azure IoT team. With long-lived industrial devices an essential component of any IoT platform, MQTT support will allow companies to quickly migrate existing device deployments to a cloud native environment. And by making use of AI support for device monitoring, they can spot outlying readings and use them to drive control systems and to order predictive maintenance. The result could well be a significant upgrade for any large-scale messaging environment.