When you have some number of IoT devices connected to an MQTT broker, one of the major concerns we had is - how do we know what devices came online, what topics they publish to, and what are its other params (device details). Most solutions involve maintaining a separate device registry database. But the idea here is to use MQTT broker itself as a device registry.

In typical MQTT deployments, you need to maintain device inventories separately: a database to store device metadata. Still, at this point, the path to write this info to the database write/read has to be created.

The initial thought would be to send the init message to the broker, from where the MQTT client reads and store/write to a source. This setup is what I’m trying to skip.

Understanding MQTT Retained Messages

Before diving into any further, let’s understand about retained messages. In MQTT, when you publish a message with the “retain” flag set to true, the broker stores that message. Here’s the key behavior:

Any new subscriber to that topic immediately receives the last retained message, even if it was published hours or days ago.

Think of it as the broker remembering the “last known state” for each topic. This is typically used for things like “what’s the current temperature?” where new subscribers want the latest value immediately, not wait for the next update.

The core idea is that - if every device publishes a manifest describing itself to a well-known topic with retain=true on boot, the broker becomes our device registry.

When a new monitoring service or dashboard comes online, it can instantly discover all devices by simply subscribing to the manifest topic pattern. The broker will immediately deliver all retained manifests from every device that has ever connected.

Use a consistent topic hierarchy for device manifests:

$registry/{device_id}/manifest

Examples:

  • $registry/sensor-living-room/manifest
  • $registry/thermostat-bedroom/manifest
  • $registry/pump-controller-01/manifest

The $ prefix is borrowed from MQTT’s $SYS/ convention to indicate these are metadata topics, not actual sensor data. End of the day, these are just topic names, we can create it however we like.

What Goes in a Manifest?

Each device publishes a JSON document describing itself:

{
  "device_id": "sensor-living-room",
  "device_type": "temperature_sensor",
  "firmware_version": "2.1.4",
  "boot_time": "2025-01-15T08:30:00Z",
  "capabilities": ["temperature", "humidity", "battery"],
  "topics": {
    "publishes": ["home/living-room/temperature"],
    "subscribes": ["home/living-room/config"]
  }
}

This tells any observer:

  • What the device is
  • What it can do
  • Where to find its data
  • When it last booted

When a device boots:

  1. Connect to MQTT broker
  2. Immediately publish manifest to $registry/{device_id}/manifest with retain=true

That’s it. No registration API to call, no database to update.

Any service that needs to know about devices:

  1. Subscribe to $registry/+/manifest (the + is a wildcard)
  2. Immediately receive all retained manifests from all devices
  sequenceDiagram
    participant Device
    participant Broker
    participant Discovery Service

    Note over Device: Device boots
    Device->>Broker: PUBLISH $registry/device-01/manifest<br/>(retain=True)
    Note over Broker: Stores retained message

    Note over Discovery Service: Service starts later
    Discovery Service->>Broker: SUBSCRIBE $registry/+/manifest
    Broker->>Discovery Service: Sends all retained manifests
    Note over Discovery Service: Now knows all devices

    Note over Device: Another device boots
    Device->>Broker: PUBLISH $registry/device-02/manifest<br/>(retain=True)
    Broker->>Discovery Service: Forward new manifest
    Note over Discovery Service: Device registry updated

How to handle offline devices

Another core challenge we face is to know and differentiate active devices, what are currently online/offline or is it even alive. Retained messages persist even when devices are offline. A device that crashed three weeks ago still has its manifest in the broker, making it look “available” when it’s not. But, we can solve it with LWT feature of MQTT.

Last Will and Testament (LWT)

MQTT has a built-in feature called Last Will and Testament. When a device connects, it can register a “last will” message that the broker will automatically publish if the device disconnects ungracefully.

The trick: Set your LWT to publish an empty retained message to your manifest topic.

On connect:
  - Set LWT: topic=$registry/{device_id}/manifest, payload="", retain=true
  - Publish actual manifest with retain=true

When the device crashes or loses connection:

  • Broker automatically publishes the empty retained message
  • This effectively deletes the manifest from the broker

This way, the broker always has alive devices with it.

I mean, there are other methods that you can try - a timestamp based approach. Or, let the device send manifest info every N minutes or such like a heartbeat. But, LWT seems to me like a neat solution. This architecture felt very sleek and no-infracture setup to me. The way I was dealing before was to create a client which will sub to a wildcard topic, write devices to a table having unique constraint to avoid duplicate records. With this way, I don’t have to care about that as well.

I use Linux, just install Mosquitto (no setup required) and playaround with it. mosquitto_pub --help for more info. tldr: just add -r flag at the end of each publish message to retain them.