BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Interviews Interview with Andy Piper on the Internet of Things and the Eclipse Paho project

Interview with Andy Piper on the Internet of Things and the Eclipse Paho project

Bookmarks
   

1. I’m here with Andy Piper at QCon London 2014, Andy looks after the Paho project at Eclipse which is one of the new Internet of Things projects that’s happened there. Andy, can you start off by explaining what the Internet of Things is?

It’s one of those topics, I guess, that means a lot of different things to different people, and it’s sometimes somewhat emotive because people get concerned about connected devices, but the way I look at the Internet of Things is simply the transformative technologies the Internet has comprised in connecting people and computing and data and then adding into that all kinds of other objects, devices, senses, useful streams of data whether that's from mobile or city environments and so on. The ability for those things to communicate, transfer data over Internet networks and then have those decisions and choices and analytics performed on that kind of information.

   

2. A lot of people think of the Eclipse Foundation as just being concerned with IDEs, how did the Internet of Things start happening at Eclipse?

You’re right. I first came into contact with Eclipse shortly around its formation, I was in between jobs, I was about to join IBM at the time when the Eclipse IDE was donated and set up as a foundation, but if you look at the history of the Eclipse Foundation over time it’s developed into runtimes as well and various other projects, various tooling projects that plug into the IDE, especially runtimes; additions to app servers, things like Virgo, OSGi, those kind of things. Actually, what happened in 2011 there was three projects that were starting up, a project called Koneki which was a Lua tooling, so it was a traditional IDE project but it was aimed at Lua, which is a scripting language which compiles down to C code and performs very well; and there was also another project called Mihini which was being proposed by the same people from the Koneki team which is for embedded devices, runtime embedded devices, a framework; and then there was a protocol called MQTT which IBM and some other companies had invented and donated to the Foundation as a new project to go with those others. And that was our initial set of three projects, Paho, Koneki and Mihini, those are all actually Maori words, people think they sound a bit strange, they do, but they have meaning, I’m not going to go through them now, and they were the original M2M, Machine to Machine projects, because Eclipse was thinking “we want to play in the embedded industrial space”.

But over time, over the last couple of years, that whole Machine to Machine space has begun to intermingle in many ways, as it naturally should, with the concept of Internet of Things and the number of projects in that space has really grown out. And I think, I don’t work for the Foundation, but speaking as someone who works on one of their projects, the Foundation really wants to be the best place to go for open standards, open source software around the Internet of Things and the Machine to Machine space. So we now have I think 13 or 14 different projects that cover protocols, we have not only MQTT, but we’ve got another protocol called CoAP, Constrained Application Protocol, we have SCADA, which is a very traditional one, we’ve got lightweight M2M, we’ve got OM2M, which is an ETSI standard, so a range of protocols that we feel are going to be part of the Internet of Things, alongside traditional things like raw TCP/IP or HTTP, those things which are automatically part of the Internet. And then we’ve also got some tooling, things like Koneki, we’ve got additional runtimes, so OSGi containers designed to run in very small footprints and we’ve got some bridges and gateways, as well, we mentioned the alphabet soup of protocols, but it’s going to be really important to integrate and bridge between them, so we have a bridge project as well.

   

3. You look after the Paho project which is concerned about MQTT. Can you tell us a little bit about what MQTT is and whether that’s standard as well?

Yes. MQTT has a long history, it’s interesting you mentioned it as a new project, actually Paho’s been around since 2011-2012, MQTT has been around a lot longer, it’s been around since 1999 and it came from the industrial world where IBM and some of their industrial partners were in that time. To cast your mind back to 1999, people don’t recognize how quickly technology has changed, but if you think about 1999 you didn’t have broadband networks, you didn’t have super-computers in your pocket like an iPhone, you didn’t have those technologies, you had much slower processing speeds than we have today, we had much more constrained networks and especially in the industrial space you were looking at very small devices with limited CPU and memory and probably on the end of a very unreliable network, it may have a long latency, maybe a satellite connection or dial-up connection, or companies may have been charged for those networks per packet, so tremendously expensive and important to reduce network usage as much as possible. So MQTT plays in that exact space, it’s a lightweight publish/subscribe messaging protocol that was designed in the late 20th – early 21st century for constrained devices and as a result it’s extremely efficient on the network, it’s a building block really for this kind of space. Over that period of time, 14-15 years, its popularity has increased, its range of uses has grown and that’s really where that’s came from. Is it a standard? Actually it’s at OASIS right now, they are currently ratifying the first “OASIS-ified” version of the MQTT protocol. MQTT the protocol was released under a royalty-free license by IBM and their partners, Eurotech, who co-invented it in the mid-2000s, so it’s been open in the sense that you can pick it up and use it for free and nobody is going to sue you, in theory, but we’ve now got it with one of the open standards bodies, OASIS, there are many more participants in that conversation, they’ve done a lot of work to tidy up the protocol document, clarify all the questions people have asked over time, “if I do this, what’s the expected behavior”, those kind of things, which an organization like OASIS is very good at, as are organizations like the IETF, actually coming up with those standards in such a way that they can be written, understood and be inter-operable. So, yes, it’s in the final stages right now, as we speak, of being standardized at OASIS.

   

4. You mentioned the MQTT was designed in an era where networks were very expensive, battery powered devices maybe out in the field. These days we’ve got much more powerful networks, much more powerful devices, so is MQTT used today in any industrial or mobile systems?

Yes, I mentioned we came from a different era, but actually when you find is when you look at something that was designed with efficiency in mind is that it scales extremely well, even to much more high performance systems, and in fact if you think about the constraints I described, many of those still exist today either in developing countries or in particular domains or in mobile. So, let’s talk about some of those. The big (I wouldn’t say flag bearer as such), but one of the most famous uses of MQTT is Facebook, they wrote about their adoption of the protocol for their messenger two or three years ago because they discovered, as it was designed to be, that is very efficient on the network, it doesn’t burn their users’ data, it’s also very efficient on the CPU, so it’s not request/response, it’s not constantly polling to see if data is coming from the server, it’s a push and publish/subscribe protocol. So they use it, not only use it for messenger and for notifications in their mobile apps cross-platform, I’ve also just learnt about a week ago that they’ve got this new app called Paper which is only available in the US at the moment, which is a similar thing to Flipboard, I think, for producing a daily digest of news and they use it in that as well, so they are heavy adopters. But that’s a consumer facing internet mobile technology, let’s talk for example industry or other sectors. MQTT in use today in healthcare, there are customers of IBM’s I know and other companies that are using it for example remote monitoring of pacemaker patients, by which I don’t mean the MQTT is embedded in the pacemaker, but there are devices around that pacemaker, so when a patient, for example, goes to bed at night, the pacemaker can be monitored locally and have a trace of data uploaded to the remote clinic over MQTT, over a dial up connection, over a low power, low bandwidth connection to improve that patient’s quality of life and monitoring. It’s also being used in energy monitoring, some people are using it for home automation, in fact it’s one of the other areas where it’s caught on, really a whole range of industries, we’ve got a wiki where we’ve got a number of case studies and use cases where people talk about the kinds of things they’ve applied it to.

   

5. Of course, all those clients need something to talk to, so what is it that they talk to on the other side?

I mentioned it’s a publish/subscribe push broadcast protocol, the clients are the end points, they are either going to be publishing data themselves somewhere, or they are subscribing and receiving data from somewhere. That somewhere is something called a broker and when we think about brokers we mentally start to leap into single point of failure, but actually there are some semantics which enable brokers to be bridged together and federated together and there are some other typologies you can construct, which would reduce the fact of single point of failure. So, we have this central object called a broker, now let’s get back to my earlier statement which is that the initial systems the MQTT was written to run on were very small and were embedded possibly less than a MB of memory, low power, low speed CPU. So, those brokers themselves have to be efficient, so there are a number of those around, IBM was the first implementer of a broker, it was a very big expensive commercial product, obviously industrial quality, but way beyond the parameters of many hobbyists. So, we are actually really blessed now that we have a number of open source brokers and we have a number of other commercial brokers.

So, the best known open source broker is one called Mosquitto which was created by this great gentleman called Roger Light, he literally looked at the protocol, decided to implement a broker, it’s written in such a way that it fits within a very limited amount of memory, depending on the options, whether you’re compiling in SSL libraries and other things, you can compile it down to 80k or less than 100k of binary, it’s written in C. Actually that broker, Mosquitto, and IBM’s offering something called “Really Small Message Broker”, with IBM’s usual flair for naming, have come together at Eclipse in the Eclipse Mosquitto project, so that’s actually now under the Eclipse umbrella. There’s a Java broker which is free, called Moquette, which we are actually also discussing potentially bringing into Eclipse into the future which is exciting, and there are some really great commercial ones. IBM has an appliance, a piece of hardware you can buy, very popular with car manufactures I understand, who want to automate their cars in the sense of collecting data from on-board sensors and distribute that back to the mechanics or home-based checking for the systems on the car. There is also another one called HiveMQ out of Germany which is a Java based broker, it’s commercial, it’s fantastic to see that come along, and again it’s one of those spaces where it’s open source, it’s a lot of popularity, people are beginning to see real value in Internet of Things solutions, so we’re starting to see more and more of these come along.

   

6. Do you see these MQTT brokers being talked to directly by hardware devices like sensors, or do you see the embedded systems being run on things like Arduino’s and Raspberry Pi’s?

The answer is that it varies. The typical thing is a sensor needs to connect to a computer system. Actually where MQTT came from was a kind of competed with another protocol which is now part of the Eclipse as well, called SCADA. SCADA systems typically were almost like a kind of token ring or closed ring kind of system where lots of sensors were communicating with proprietary system and that was the point of collection and data analysis. MQTT was invented to break out of that what they used to call the SCADA jail, SCADA itself has evolved in many different ways since then. So, typically, back in the day on an oil rig you would’ve had a small Linux box with a bunch of sensors connected to it and data coming into that, and the box, the computer would do the MQTT broadcast rather than the sensors themselves.

Today sensors are becoming more capable themselves, every device has got some kind of CPU in it very frequently now. So, you are starting to see some devices like for example an energy meter Flukso, which you can purchase for your home to monitor your energy usage and that will natively talk MQTT to a broker. So, you absolutely can start to get small devices that do that and some of the hobbyist supporter manufacturers have started to produce prototyping boards which have inbuilt or onboard MQTT libraries, OpenPicus is one of them. They’ve got a little Flyport device which has an MQTT library ready to go, so you can immediately start enabling your system. But as I said it’s a bit of both, those where with the Raspberry Pi for example, you are connecting sensors to the Pi, receiving data through the GPIO ports and then publishing data from it on the Pi as a Linux system; the Arduino is a good example, it doesn’t have an operating system on it as such, it runs C-like code, but there is an Arduino MQTT client, so you can do that. But it’s a spectrum, you can buy some devices that have it enabled and built in and others are coming along.

   

7. Coming back to the Paho project, what libraries do you ship available for different systems?

Paho is still pre-1.0 release at Eclipse, but we are moving in that direction, we are going to join the Luna release train, we have got Java, JavaScript and C libraries, which are really high quality, those are the ones that came from IBM, so they’ve got 10 years’ development behind them when they originated initially, so they are in good shape. We’ve had a number of other ones contributed, we’ve got the Python library which came from the Mosquitto project originally, also great, and I think that will probably be in our 1.0 release, there is another library in Go which came from some IBMers as well, I’m going to forget some here, but we’ve also got C++ library, we’ve got a Lua library that fits in with Koneki world, we’ve got an Objective-C client that’s just been contributed as well from an organization called 2lemetry who themselves have built a business on MQTT in supporting customers wanting to use cloud-based MQTT services. So we’ve got a nice collection coming in, we’re always looking at more, we saw some tweets today talking to one of the guys who have written the .NET supported library to see whether he’d be interested in joining us and open sourcing the library under the Eclipse public license. Paho’s about clients and connectivity for apps and devices, maybe the Arduino client or something similar in the future; at the same time I am always happy to see innovation outside of Eclipse, as well. You mentioned I am the leader of the Paho project, I also run the mqtt.org community website, much more thinking about the broader ecosystem as much as I am keen to see the success of the Paho project and Eclipse IDE project.

Alex: And so the plan is for the Paho 1.0 release to be in Luna which is July 2014.

I think it’s going to be interesting, we will have to up our game to get there, but that’s our plan at least have some of those libraries available in what we believe is a production consumable quality by then. They are widely used already, so a lot of projects are already consuming the Paho Java client themselves and are implementing based on our pre-1.0 code which as I say is good quality anyway, we just haven’t gone through that Eclipse process.

   

8. So there is a JavaScript client for MQTT as well, how does that fit in with, say, something like WebSockets for communicating data to a browser asynchronously?

Actually, it’s MQTT over WebSockets, in fact. So, the way that works is the broker needs to be WebSocket aware as well, in Mosquitto you would typically put a WebSocket server in front of it configure as a proxy, but there’s an agreement, although not a standard, of how you tunnel MQTT over WebSockets. So, essentially the JavaScript library is a wrapper, it looks MQTT-like as an API, but essentially it’s talking WebSockets, I believe and I haven’t looked at the detail, that one of the headers in the WebSockets connection specifies it’s an MQTT binary packet that’s coming over, would have to go check the wiki and make sure I understood how it’s working, but that’s how it works. The idea being the WebSockets as a protocol on the web is ideal for streaming real time data, MQTT is also ideal for streaming real time data. We do see those two things converging, not one replacing the other, because obviously WebSockets is built on a full web stack, whereas MQTT is often embedded in something much smaller or lightweight.

   

9. Where do you see the Internet of Things going and Eclipse for the future?

At Eclipse, I know Ian Skerrett from the Eclipse Foundation has said that he'd love to see Eclipse being the home for open source, open Internet of Things protocols and projects and I think we are well on the way there. A number of the projects I mentioned earlier are still in proposal stages, but we already having community calls with them as a group and seeing them start to consume one another, how do we for example say, we’ve got an embedded OSGi runtime, how come we use the MQTT Paho JAR inside that? We’ve got the bridge project Ponte, how can we ensure there are bridges between CoAP and SCADA and MQTT and all those other things? So certainly a lot more cross pollination among those projects. I’d be really interested to see given that the Eclipse IDE has some great tools for things like data analysis built on top of it and one of the important aspects of this is not just connectivity and embedded devices, it’s also that pool of data, that massive sea of data that’s being generated. How do we evolve the Eclipse offering in the direction of actually analyzing that data and making some useful decisions, applying rules engines, those kind of things, we probably have thought about it from the enterprise perspective before but we may not have applied it to the same thing.

So, if you’ve look at something like the Node-RED project which isn’t part of Eclipse, that’s a really lightweight integration system written in Node (JavaScript) for integrating Internet of Things projects or scenarios. So, for example I’m reading tweets and as hashtag tweets arrive, I would like to light up these LEDs in my Arduino in different colors and also publish a message to another MQTT server which does something else, it’s really a flow integration and orchestration engine for the Internet of Things. If Node-RED itself doesn’t come to Eclipse, obviously I have no say, I am just going to express an interest in seeing that happen, I imagine we will see more of those kind of solutions come up around the Internet of Things and possibly at Eclipse, because it’s an integration problem as much as anything else. We can enable light bulbs to come on over wireless, but what does that mean? It needs to happen in a sensible, constructive way, should it respond to the event of sunset outside, should our heating systems respond to cold temperatures outside, we would probably all like that within our smart homes, but we still need to build those integrations and build those decision flows.

Alex: Andy Piper, thank you very much.

You’re welcome, thank you.

Apr 29, 2014

BT