ELIoT: Distributed Programming for the Internet of Things

ELIoT (Extensible Language for the Internet of Things) is a simple and small programming language aiming to make distributed programming easier. A program in ELIoT may appear as a sigle program, but it actually runs on different computers, so, e.g., a variable or function declared on one computer is transparently used on another.

Among ELIoT goals are enabling configuration and control of swarms of small devices. As an example, in less than 20 lines of code, you can have two temperature sensors talk one another and, in case the difference in temperature between them is greater than a given threshold, inform a remote controlling application. This is expressed by the following single code snippet:

invoke "sensor2.corp.net",
   every 1.1s,
        sensor1_temp -> ask "sensor1.corp.net", temperature
        send_temps sensor1_temp, temperature

   send_temps T1:real, T2:real ->
       if abs(T1-T2) > 2.0 then
           reply
               show_temps T1, T2

show_temps T1:real, T2:real ->
    write "Temperature on sensor1 is ", T1, " and on sensor2 ", T2, ". "
    if T1>T2 then
        writeln "Sensor1 is hotter by ", T1-T2, " degrees"
    else
        writeln "Sensor2 is hotter by ", T2-T1, " degrees"

At the heart of ELIoT are a few primitives, such as:

• tell: it sends a program asynchronously
• ask: it sends a program synchronously and waits for the result
• invoke: it sends a program and opens a bi-directional channel.

InfoQ has talked with ELIoT’s creator, Christophe de Dinechin.

Could you explain what led you to create ELIoT? What was the need to create a new language for distributed programming?

ELIoT is the convergence of three ideas:

1. A little over a decade of research in language design, with XL and its derivative Tao3D. The objective of my research is to create a language that is flexible enough to adapt to practically any usage scenario. To validate those ideas, I purposely selected use cases where other programming languages do not fare well, like describing interactive 3D documents (Tao3D) or distributed systems (ELIoT). I also validated the extensibility postulate by implementing in a library most of what other programming languages implement in the compiler/interpreter (e.g. loops, tests, arithmetic, optimizations etc). So the first key idea is “an extensible programming language”.

2. Discussions with engineers in the embedded space, notably from Intel, who gave me the idea of a language with a minimal footprint, but capable of dealing easily with a multiplicity of sensors or actuators. In embedded systems, languages like Lua are quite useful. But the value of small systems lies beyond the computations they can do locally. For example, an Apple Watch requires an iPhone to be useful, and most iPhone applications derive value from cloud-based servers. Hence the idea in ELIoT to exchange tiny program fragments very easily. And curiously, not that many languages do that well. So the second idea is “distributed programming with an eye on embedded systems”.

3. Discussions with former colleagues at HP who are designing something called The Machine, based on memristors (fully persistent memory). They are designing a clean-sheet operating system called Carbon. I know nothing about Carbon, but that got me thinking about what I would like to see in a clean-sheet OS. A recurring theme was: “how do you program that thing?”. Machines today are very different from back when Unix was designed, with heterogeneous CPU/GPUs/XPUs. Communication and storage became as important as computations. So I started dreaming about what I’d like in a system language for a machine made of 10000 CPUs each with a different architecture. And here again, I saw the same recurring theme of being able to send computations along with their data (“compute along the way”). So the third idea is “exchanging programs and data transparently, at the language level”.

In summary:

• An extensible programming language
• for distributed programming (with a focus on embedded systems)
• by exchanging data and programs transparently at the language level.

What problem does ELIoT specifically try to solve?

Let’s be honest: I’m first and foremost doing it for fun and personal development. If it happens to be useful, I’m thrilled, but in that case, the journey is really its own reward. And by that metric, ELIoT is getting closer and closer to my ideal programming language. For example, I’m relatively happy with ELIoT’s definition of complex numbers. In my biased opinion, it’s really short and to the point, and I’m pretty convinced it can someday be optimized to really good machine code. Of note, being designed to please me, myself and I makes ELIoT very “different”, and some people just hate that. It certainly does not look like C or Java and never will…

But with respect to what I saw as an unfulfilled need, it’s really: how do you control a small fleet of sensors and actuators easily, not knowing ahead of time what kind of data you are going to ask or what actions you are going to require. That led to the desire for a small, lightweight language applicable to embedded systems, that could be extended easily, as well as be used as an extension language for applications, and that would make distributed programming really easy. Several languages had one or the other property. None I know has them all, at least to my satisfaction.

Languages that had an influence on the design include Lisp (programs = data), Lua (small footprint), Python (as an extension language), Erlang (communication between processes), Pure (pattern matching), BASIC (no stinking parentheses in the syntax), and dozens of others, e.g. Ada, Hop, Haskell, C/C++, Java, Occam, XC to cite just a few. ELIoT tries to borrow what I see as the most salient strengths from each of these languages.

Could you explain how ELIoT works its “magics”?

ELIoT is a fully homoiconic language, i.e. programs and data are the same thing. All programs and all data in ELIoT (and its ancestor XL) are represented by a single data structure, the “tree” (an abstract syntax tree, aka parse tree). A recent redesign of the language compiler/interpreter ensured that all non-transient interpreter data structures were themselves represented using the same trees. That includes the symbol table, i.e. the way to tell that the value of variable ‘X’ is 28 in the current context.

Once you have built a language with these properties (and that’s the big effort, I don’t think it can ever happen for C or C++ for example), then the “magic” becomes very simple: you simply send over the wire a serialized format representing the parse tree for your program, combined with the parse tree representing its symbol table. The other side receives that data, reconstructs a symbol table and a program parse tree from it, and executes the program in the transmitted context.

There are a few details that matter, e.g. filtering which symbols are sent over the wire. In the current implementation, we stop at the source file level, i.e. you don’t send anything from other files. Built-in definitions and imported modules are assumed to also exist on the other side, possibly with a different implementation.

Finally, what do you envision for ELIoT’s future?

ELIoT comes with great ambitions. But I also made the deliberate decision to involve the open-source community long before it’s complete, while it’s still tiny and weak. I want this thing to have as many contributors as possible. In other words, I don’t have a preconceived notion of where it’s going. Hopefully, the community will take it places I would never have thought of. I’d be happy to have it taking pictures of Pluto one day ;-)

According to Slashdot and other people who contacted me, there is a big gap in security. And granted, the security model is not defined yet, mostly because a person I was discussing with at Intel is a big security guru and I wanted to discuss various ideas with him first. In any case, security is probably what I will focus on next. But there are a few other open issues for those who want to help.