Playing with some basics aspects of MQTT (MQTTS)
Hi friends,
here we are again with another lab...but first thing first, look around out there for this song, "The Whole of the Moon" by The Waterboys (1985).
In this new lab, we will test some basic aspects of MQTT. Interested? Let's get to it.
Disclaimer: I’m not an expert, it’s clear, I’m testing and sharing my notes, perhaps with some errors.
What is MQTT?
A little bit of history: The MQTT protocol was invented in 1999 for use in the oil and gas industry. Initially, the protocol was known as Message Queuing Telemetry Transport due to the IBM product MQ Series product line, where it stands for "Message Queue". However, the protocol provides publish-and-subscribe messaging (no queues, in spite of the name). In 2010, IBM released MQTT 3.1 as a free and open protocol for anyone to implement, which was then submitted, in 2013, to Organization for the Advancement of Structured Information Standards (OASIS) specification body for maintenance. In 2019, an upgraded MQTT version 5 was released by OASIS. Now MQTT is no longer an acronym but is considered to be the official name of the protocol.
Yes, yes... MQTT is the fucking master, but show me a high-level topology of its basic architecture.
M QTT Components
MQTT client (publisher/subscriber)
An MQTT client is any device that runs an MQTT library. If the client is sending messages, it acts as a publisher, and if it is receiving messages, it acts as a receiver/subscriber. Basically, any device that communicates using MQTT over a network can be called an MQTT client device.
MQTT broker
The MQTT broker is the backend system which coordinates messages between the different clients. Responsibilities of the broker include receiving and filtering messages, identifying clients subscribed to each message, and sending them the messages. It is also responsible for other tasks such as:
MQTT connection
Clients and brokers begin communicating by using an MQTT connection. Clients initiate the connection by sending a CONNECT message to the MQTT broker. The broker confirms that a connection has been established by responding with a CONNACK message. Both the MQTT client and the broker require a TCP/IP stack to communicate. Clients never connect with each other, only with the broker.
MQTT topic
A topic is a hierarchical string used to categorize messages. Topics provide a way for publishers to label their messages and for subscribers to express interest in receiving messages related to specific topics.
All in one
Messages in MQTT are published to topics by the publishers, which are hierarchical strings used for categorizing messages. Subscribers can subscribe to entire topics or specific subtopics using wildcards, allowing for flexible message routing.
1. Publisher (Client sending messages to specific topic): A publisher is any device or application that sends messages to a specific topic on the MQTT broker. MQTT clients publish messages that contain the topic and data in byte format. The client determines the data format such as text data, binary data, XML, or JSON files.
2. Subscriber (Client that receives the messages from specific topic): A subscriber is a device or application that receives messages by subscribing to specific topics on the MQTT broker. MQTT clients send a SUBSCRIBE message to the MQTT broker, to receive messages on topics of interest. This message contains a unique identifier and a list of subscriptions.
3. Broker: The MQTT broker acts as an intermediary between publishers and subscribers. It receives messages from publishers and forwards them to the appropriate subscribers based on topic subscriptions.
In the previous introduction to MQTT, we covered the basics of this lightweight messaging protocol for the Internet of Things. But when it comes to sensitive data or mission-critical applications, security becomes paramount. That's where MQTT TLS, also known as MQTTS, comes in.
What is MQTT TLS?
MQTTS, or Message Queuing Telemetry Transport Secured, is the secure version of the MQTT protocol. It leverages Transport Layer Security (TLS) to establish an encrypted communication channel between clients and brokers. This ensures:
MQTT TLS plays a crucial role in securing MQTT communication, providing encryption, authentication, and integrity mechanisms to protect data exchanged between MQTT clients and brokers. By implementing MQTT TLS, organizations can ensure the confidentiality and integrity of their IoT deployments, safeguarding against potential security threats...BUT take into account some considerations for Implementing MQTT TLS:
MQTT QoS
MQTT (Message Queuing Telemetry Transport) provides Quality of Service (QoS) levels to ensure reliable message delivery between MQTT clients (publishers and subscribers) and the MQTT broker. QoS levels define the guarantee of message delivery and the message handling behavior between publishers and subscribers. There are three QoS levels in MQTT: QoS 0, QoS 1, and QoS 2. Let's explore each level:
1. QoS 0 (At most once):
- Also known as "fire and forget."
- The message is delivered to the broker without any confirmation.
- There is no guarantee that the message will be received by the subscriber.
- This level provides the lowest reliability but the highest efficiency, suitable for non-critical messages where occasional message loss is acceptable, such as sensor data where the most recent value is more important than older data.
2. QoS 1 (At least once):
- The message is delivered to the broker and stored until the subscriber acknowledges receipt.
- If the subscriber doesn't acknowledge receipt, the broker resends the message.
- This level ensures that the message is delivered at least once but may result in duplicate messages.
- Suitable for scenarios where message duplication is acceptable, but message loss is not, such as controlling actuators or commands where duplicate execution is harmless.
3. QoS 2 (Exactly once):
- The most reliable QoS level.
- The message is delivered exactly once to the subscriber by using a four-step handshake process.
- The publisher and broker exchange a series of messages to ensure that the message is delivered exactly once, even in the presence of network errors or failures.
- This level guarantees no message loss and no duplicate messages but has higher overhead due to the handshake process.
- Suitable for critical messages where both message loss and duplication are unacceptable, such as financial transactions or control commands where exactly-once processing is essential.
When choosing the appropriate QoS level for MQTT communication please consider factors such as the importance of message delivery, network reliability, bandwidth constraints, and application requirements. By selecting the right QoS level, MQTT applications can achieve the desired balance between reliability, efficiency, and delivery guarantees.
The MQTT TLS LAB
In order to view all of these running I deployed a MQTT TLS lab in my home. There are 3 clients that can act as Publishers and/or subscribers, and one broker.
The clients are → https://github.com/eclipse/paho.mqtt.python and https://github.com/emqx/MQTTX
The broker is → https://github.com/eclipse/mosquitto
In this LAB we will need certificates for the clients (publishers and subscribers), and for the broker. Remember all the communications will be secured through TLS.
In order to issue the certificates I use EJBCA Community (EJBCA CE), that is a PKI free and open source, OSI Certified Open Source Software. EJBCA covers all your needs – from certificate management, registration and enrollment to certificate validation. Here is a post where I describe the setup of this PKI: https://www.linkedin.com/pulse/certificates-certificatesoh-mammaejbca-open-source-gonzalez-diaz?trk=portfolio_article-card_title
Please take note here, if you go to production consider the use of EJBCA Enterprise (EJBCA EE) - commercial and Common Criteria certified.
Lab Topology
Broker configuration (Mosquitto)
In the acl.conf we define some restrictions to publish and subscribe actions:
MQTTX Client Configuration (MQTTS)
PAHO Client Configuration (MQTTS)
Testing MQTTS
I’ve created a python script that can subscribe and publish multiple messages to a specific topic in order to test the setup. The subscriber “homemqttclient01” will send multiple messages to the topic “HomeArpa/MQTTLAB01/publish01”. The same client (homemqttclient01) will subscribe to that topic, as result we will have the messages back. Here is the output:
➜ MQTT python mqtt_publish.py
Creating instance & connecting to the broker --> homemqttbroker01.home.arpa
Subscribing to topic --> HomeArpa/MQTTLAB01/publish01
Publishing message to topic HomeArpa/MQTTLAB01/publish01
MQTT message received = {"msg": "Message sent by MQTT publisher (homemqttclient01).", "variable": "109 units"}
MQTT message topic = HomeArpa/MQTTLAB01/publish01
MQTT message qos = 2
MQTT message retain flag = 1
MQTT message received = {"msg": "Message sent by MQTT publisher (homemqttclient01).", "variable": "101 units"}
MQTT message topic = HomeArpa/MQTTLAB01/publish01
MQTT message qos = 2
MQTT message retain flag = 0
Publishing message to topic HomeArpa/MQTTLAB01/publish01
MQTT message received = {"msg": "Message sent by MQTT publisher (homemqttclient01).", "variable": "102 units"}
MQTT message topic = HomeArpa/MQTTLAB01/publish01
MQTT message qos = 2
MQTT message retain flag = 0
Publishing message to topic HomeArpa/MQTTLAB01/publish01
MQTT message received = {"msg": "Message sent by MQTT publisher (homemqttclient01).", "variable": "103 units"}
MQTT message topic = HomeArpa/MQTTLAB01/publish01
MQTT message qos = 2
MQTT message retain flag = 0
Publishing message to topic HomeArpa/MQTTLAB01/publish01
MQTT message received = {"msg": "Message sent by MQTT publisher (homemqttclient01).", "variable": "104 units"}
MQTT message topic = HomeArpa/MQTTLAB01/publish01
MQTT message qos = 2
MQTT message retain flag = 0
Publishing message to topic HomeArpa/MQTTLAB01/publish01
MQTT message received = {"msg": "Message sent by MQTT publisher (homemqttclient01).", "variable": "105 units"}
MQTT message topic = HomeArpa/MQTTLAB01/publish01
MQTT message qos = 2
MQTT message retain flag = 0
Publishing message to topic HomeArpa/MQTTLAB01/publish01
MQTT message received = {"msg": "Message sent by MQTT publisher (homemqttclient01).", "variable": "106 units"}
MQTT message topic = HomeArpa/MQTTLAB01/publish01
MQTT message qos = 2
MQTT message retain flag = 0
Publishing message to topic HomeArpa/MQTTLAB01/publish01
MQTT message received = {"msg": "Message sent by MQTT publisher (homemqttclient01).", "variable": "107 units"}
MQTT message topic = HomeArpa/MQTTLAB01/publish01
MQTT message qos = 2
MQTT message retain flag = 0
Publishing message to topic HomeArpa/MQTTLAB01/publish01
MQTT message received = {"msg": "Message sent by MQTT publisher (homemqttclient01).", "variable": "108 units"}
MQTT message topic = HomeArpa/MQTTLAB01/publish01
MQTT message qos = 2
MQTT message retain flag = 0
Publishing message to topic HomeArpa/MQTTLAB01/publish01
MQTT message received = {"msg": "Message sent by MQTT publisher (homemqttclient01).", "variable": "109 units"}
MQTT message topic = HomeArpa/MQTTLAB01/publish01
MQTT message qos = 2
MQTT message retain flag = 0
Hasta la vista baby!
Of course, if we check other subscriber of that topic the messages are there…
Captures
The next image shows a traffic capture between the subscriber and the broker:
MQTT over Websockets TLS (WSS)
MQTT over WebSockets is a protocol adaptation that enables MQTT communication to be carried over the WebSocket protocol. WebSockets provide a full-duplex communication channel over a single TCP connection, making them ideal for real-time, bidirectional communication between web browsers and servers. By leveraging WebSockets, MQTT can be used in web applications without the need for specialized MQTT libraries or protocols.
When using MQTT over WebSockets, MQTT messages are encapsulated within WebSocket frames, allowing them to be transmitted over standard HTTP/HTTPS ports (80/443) or not. In this lab I’ve used TCP 9001 port.
Here is the setup with MQTTX client:
The next image shows a traffic capture between the subscriber and the broker using websockets:
Publishing Nagios checks results to a topic
I’ve deployed multiple nagios plugins for my use, thus is very easy to implement new functions. For example, I’ve created a python script (when a script is used by nagios that script is called plugin) that checks if there is any rogue AP in my home “area”. Well, I added to that python script a function that publish the result of the nagios check. Now I can read the result securely using a MQTT client...I know, it seems silly, but I love it.
Conclusions
MQTT provides a robust and efficient messaging solution for a wide range of applications, particularly in IoT, where scalability, reliability, and low overhead are essential requirements. By leveraging its features, developers can build resilient and responsive systems that facilitate seamless communication between devices and applications.
Documentation