The Eclipse foundation started a Machine-to-Machine working group at EclipseCon 2012 looking at forming a community around the growing Internet of Things – hardware devices and sensors that are becoming networked on a massive scale. Recently, the working group renamed to IoT since the Internet-of-Things is more inclusive a term a than Machine-to-Machine.
A key part of this environment is a means to communicate and share data, and instead of requiring high power ethernet and WiFi connections, the growing trend has been to use MQTT as the de-facto communication protocol.
To find out more about where it came from and how people use it, InfoQ caught up with Andy Piper, project lead of the Eclipse Paho project, which provides clients for the MQTT (Message Queueing Telemetry Transport) protocol in a variety of languages.
InfoQ: The Paho project is specifically aimed at clients for the MQTT protocol. Can you tell us a little about what MQTT is, and where it has come from?
Piper: MQTT originated in the industrial sector in the late 1990s – the first implementation was in 1999, to be precise. Companies wanting to monitor industrial and mechanical systems (think pipelines, oil pressures, water levels etc) needed a very lightweight technology, suitable for running on constrained systems and devices.
At the time, the primary protocol for dealing with those environments was SCADA (Supervisory Control and Data Acquisition), and the early versions of that protocol were monolithic or tightly-bound, proprietary and closed systems. IBM worked with some partners, primarily a company called Arcom, to devise a lightweight messaging-based protocol as an alternative, enabling better scalability and ease of integration.
So essentially, MQTT is a very light, publish/subscribe, asynchronous messaging protocol, ideal for the Internet of Things, mobile, and sensor networks. It started out at IBM, but in the past few years it has gone open source, with a standard being agreed at OASIS – so it is much more widespread than just being an IBM thing these days.
InfoQ: Is MQTT a proprietary protocol?
Piper: Once upon a time, MQTT was only implemented by IBM and a small number of their partners. However, the protocol specification was published "royalty free" for many years, and as a result, a few free implementations emerged – most notably, the very popular mosquitto broker from Roger Light. Since 2011 MQTT has been at Eclipse as part of the Paho project, and during 2013 the specification was submitted to OASIS for standardisation. The list of vendors and interested parties in the standardisation process is impressive! The MQTT OASIS Technical Committee is being formed and welcomes participants.
So, no, MQTT isn't proprietary, it is absolutely open, with many free and open source implementations, as well as a number of commercially-licensed and supported brokers and clients. IBM have a very high performance appliance which is being used to back a number of projects like the Internet of Things in connected cars from some manufacturers; one of the newer commercial brokers I'm most excited by is HiveMQ, from dc-square out of Germany.
InfoQ: What makes MQTT a good choice for embedded systems or high latency networks?
Piper: As I mentioned, MQTT was born in an era – and in a specific environment – where computing power and bandwidth were at a premium. It was designed for systems with limited CPU cycles and memory, and also for networks with minimal bandwidth or high latency (dial-up networks or satellite links), where users were potentially being charged for every byte transmitted.
As a result, even today in our era of broadband connections and smartphones with many more capabilities than the computers used to get us to the moon, MQTT can run very efficiently on small devices or in areas where network coverage is patchy. Analysis has shown that MQTT is very efficient on network and battery consumption on mobile devices.
The lightest implementations run on devices like the mbed and Arduino. The smallest possible packet size is 2 bytes, and there are a bunch of built-in features for dealing with situations where devices lose contact with the network (called the "last will and testament" feature, where a device or client can specify a final message to be sent on its behalf if contact is lost).
Publish/subscribe is actually a really good mechanism for distributing data in these kinds of systems, too – you're not making individual point-to-point HTTP-style calls to every node on the network, so this scales very effectively. Any system interested in the data can subscribe on a relevant topic and make use of it; other devices can carry on with the tasks that are important to them, and ignore it.
One other thing we should mention is that at the heart of an MQTT system, is a broker – brokers can often be bridged and clustered to share topics and publications (depending on the implementation), to avoid single points of failure – devices talk to the broker, and the broker distributes messages to subscribers.
InfoQ: What kinds of Quality of Service does MQTT provide?
Piper: Simple answer – there are three of them:
- QoS 0 is essentially "fire and forget" – the message may arrive, it may not, but the data probably isn't critical anyway (in most cases the message will be delivered somewhere, of course, but there's no double-check to make sure that happens).
- QoS 1 makes a bit more of an effort to ensure a message arrives: if an acknowledgement of receipt isn't received from the broker, we'll re-send the message again but with a duplicate header byte set, just in case.
- QoS 2 is closer to the two-phase commit world and tries to offer once and once-only delivery, for data where you want to be sure the broker did receive the message for onward delivery.
InfoQ: How does MQTT compare with other message broker technologies like JMS?
Piper: JMS is actually an interesting example to choose – really, JMS is an API, not a protocol or transport. When Sun specified JMS they essentially allowed any vendor or open source project to plug a suitable transport under a common API (so you ended up with IBM MQ, Tibco, etc. which didn't interoperate at the network layer, even though programs were broadly portable between them).
A better analog would be to compare MQTT with, say, AMQP, STOMP, or ØMQ. The answer is, it sits alongside all of these. It isn't a full-featured enterprise messaging protocol with fine-grained controls, ultra-reliable queueing, like AMQP; but it also isn't a queue grafted on top of a TCP socket like ØMQ (apologies to the ØMQ guys, I'm over-simplifying here for the sake of comparison!). It tries to provide a simple API – there are essentially 5 methods; a compact payload suitable for constrained networks; and a publish/subscribe "push" mechanism useful for sensor and mobile scenarios.
InfoQ: Who is using MQTT at the moment?
Piper: There are really a huge number of use cases and examples, and I'll pick on a few of the more high-profile and well-known ones:
- Probably the most recognisable name on the list is Facebook – they blogged about how they use MQTT in their Messenger and mobile apps, due to the lightweight network characteristics (they don't want to drain phone batteries or use all your data).
- St Jude Medical in the US are using MQTT for remote monitoring of patients.
- Consert Energy are using MQTT for their smart home meters.
- The Isle of Wight ferries tweet their location as they leave and enter port, driven by an MQTT system.
- Andy Stanford-Clark (co-inventor of MQTT) has made his house into a sort of living laboratory for home automation using MQTT, and monitors everything from which lights are switched on, to whether the garden gate was left open – oh, and whether he needs to re-arm his mousetraps!
InfoQ: How did the Paho project get created, and why was it created at Eclipse?
Piper: Eclipse started as an IDE, but the truth is that the Eclipse Foundation has always supported a wide range of open source projects, both around developer tools, and increasingly also runtimes and frameworks.
Paho came about when IBM was looking to open source their MQTT client code – and IBM has always had a strong affinity with the Eclipse community. At the same time, two other projects, Eclipse Koneki (a Lua tooling extension for the IDE) and Eclipse Mihini (an embedded runtime for IoT) were spinning up, and these three projects formed the start of the Eclipse IoT ecosystem. That has now expanded significantly, as the importance of IoT and the Internet of Things has been well-recognised.
We now have the mosquitto broker combining with a newly open-sourced IBM broker ("Really Small Message Broker") at the Foundation, in the Eclipse Mosquitto project; and there are many other projects like Eclipse SmartHome, Kura, Krittit, SCADA, OM2M, Wakaama (formerly liblwm2m), Californium, and – importantly, as it bridges between protocols – Eclipse Ponte. In fact, we recently took the decision to rename the Eclipse M2M Industry Working Group to Eclipse IoT, to reflect the industry direction on terms and understanding of this space.
Ian Skerrett from the Eclipse Foundation recently wrote "Eclipse is well on its way to being the community for open source implementations for IoT standards" – and the Paho project is excited to be a core part of that community!
InfoQ: What languages does the Paho project have libraries for?
Piper: Java, C, and Javascript are currently our "core" or implementations, and came from the commercial IBM libraries originally. The Python client was contributed from the mosquitto project and is also great quality and well-tested. We also have clients in Go, Lua and C++ which are being developed, an Objective-C client which is just being contributed and we are always happy to look at new contributions.
The goal here is to incubate high-quality, specification-compliant clients which can be reference implementations. The mqtt.org website also lists a wide range of other implementations in many other languages and for different devices – the Arduino client is particularly popular, but is not part of the Paho project.
InfoQ: Are there any available MQTT brokers for developers to work with?
Piper: Yes, there are lots, these days! As the community lead, it is a pleasure to be able to say that, since we used to just have a couple of examples, some of which were not entirely free to use. I mentioned that RSMB and mosquitto are coming together under the Eclipse Mosquitto project, and in fact you can already point an MQTT client at iot.eclipse.org:1883 and use a free, hosted mosquitto broker.
There are some other less well-known free examples around too, such as moquette (a Java-based broker). In the "mature OSS projects" space, RabbitMQ and ActiveMQ both support MQTT as well as their other established messaging protocols. Developers can also get a copy of the commercial HiveMQ broker to try out, or even download a virtual machine image of the full IBM MessageSight appliance (the actual hardware is probably out-of-reach for the majority of standalone developers... IBM won't even send one of the boxes to me!).
It is worth noting that while there are a wide range of language implementations of MQTT – not all of them are hosted at inside Paho – and a number of brokers, interoperability remains an important consideration. Not all of the brokers support the same features. At EclipseCon in California this March, there will be an interoperability testing day, where developers and hardware vendors can come together and see how well their implementations conform to the specification and work together.
InfoQ: Are there any good documentation or tutorials on the web to find out more about MQTT and how to use it with Paho and Mosquitto?
Piper: Yes, but we can always do better! A couple of my favourite recent articles are by DJ Walker-Morgan, who wrote some nice tutorials. Take a look at Practical MQTT with Paho and Using Eclipse Paho's MQTT on BeagleBone Black and Raspberry Pi. The HiveMQ guys have some great tutorials on their website, too. The really nice part about the MQTT community is that it is very active, and there are many places to come together and share ideas and ask questions. http://mqtt.org/get-involved lists some of the community spaces and mailing lists. Chances are, if you've got an idea or a question, someone on the forums can help you!
One thing to note is that at the moment, we're in the process of moving the mqtt.org community website and wiki to be hosted at Github – see http://github.org/mqtt for the new organisation. This is a great time to get involved and help out with revamping the website and reorganising the wiki to be a great source of information for newcomers and experienced developers and users alike!
As Facebook shows, MQTT is suitable for low powered or battery devices, as well as for smaller devices or embedded computers. There's a special track for the internet of things at EclipseCon on March 19, 2014. What could you network with MQTT?
About the Interviewee
Andy Piper is part of the Pivotal Developer Relations and Community Engineering team, and works as Developer Advocate for Cloud Foundry, the Open Source Platform-as-a-Service. He is probably best known online as a “social bridgebuilder” spanning a number of areas of technology and interest. Andy has a passionate interest in small and mobile devices, cloud, the Internet of Things, and Arduino and related technologies. He was previously with IBM Software Group for more than 10 years, as a consultant, strategist, and WebSphere Messaging Community Lead.