Interview: François-Régis Sinot on Opa, a Web Development Platform

Opa is a web development platform including a language, Opa, based on OCaml, a built-in web server, database and distributed execution engine taking care of most infrastructure needs when writing and deploying a web application, including cloud distribution. Upon compilation Opa generates jQuery-based JavaScript for cross-browser compatibility on the client side with native code on the server side.

Opa was started back in 2001 by Henri Binsztok as a research project, part of his PhD thesis at Machine Learning Lab in Paris, and it became commercial in 2007. The platform is open sourced under the AGPL license which means any code written with Opa needs to be open sourced, but the company behind it provides paid licenses for those who want to keep the source code closed. Opa’s source code is available on GitHub.

With Opa the developer can focus on the domain of his applications, the platform taking care of the plumbing, including generating a custom webserver, creating and populating a database, and providing communication with the specified cloud. Following is the source code of a real time chat room web application:

type message = { author : string ; text : string }
@publish room = Network.cloud("room") : Network.network(message)
								
								
									

								
								
user_update(x : message) =
  line = <div class="line">
            <div class="user">{x.author}:</>
            <div class="message">{x.text}</>
         </>
  do Dom.transform([#conversation +<- line ])
  Dom.scroll_to_bottom(#conversation)
broadcast(author) =
   do Network.broadcast({~author text=Dom.get_value(#entry)}, room)
   Dom.clear_value(#entry)
start() =
   author = Random.string(8)
   <div id=#header><div id=#logo></></>
   <div id=#conversation onready={_ -> Network.add_callback(user_update, room)}></>
   <div id=#footer>
      <input id=#entry onnewline={_ -> broadcast(author)} />
      <div class="button" onclick={_ -> broadcast(author)}>Post</>
   </>
server = Server.one_page_bundle("Chat", [@static_resource_directory("resources")],
       ["resources/css.css"], start)

To find out more details about Opa, we interviewed François-Régis Sinot, a software architect working on Opa and related projects.

InfoQ: Please give us some details on the history of the project: when did it start, who started it, who is working on it now?

François-Régis Sinot: Henri [Binsztok] began the project ten years ago. Quoting the MLstate blog:

"Indeed, the first draft and first lines of code of what would become Opa were written in 2001. Back then, I was starting my PhD program. After several prior internships at the Machine Learning lab at Paris VI, I already knew quite a lot about graph models (Hidden Markov Models, Bayesian Nets, and the like) and I was struggling storing them in a database.

Graph databases that would fit within the challenging university budget (0 Euro = $0) were not legion, so I started using MySQL with two tables, one for nodes, the other for edges. While working on the learning of graph structure, I used a programming language to generate SQL queries. But as the (generated) queries grew bigger and bigger, MySQL became slower and slower until it purely and simply segfaulted. Therefore, I started writing a simple but safe database implemented in OCaml. Why OCaml? Because OCaml has a strong static typing system with type inference that allows to write software with less bugs than C, Python, Java or any other imperative programming language.

Years passed and the database had been integrated within a new, clean, ML programming language (hence the name MLstate) and gained web server basic functionality. In 2007, MLstate was born as a company to transform the Opa prototype into a production-grade technology. And the work was tremendous. More than 100 man years of work, now with a team of more than 10 PhDs in computer science and as much research engineers were necessary to build the Opa we have today."

InfoQ: What is your vision regarding Opa? What do you want to accomplish with it?

François-Régis Sinot: We think that the web can be an even better place, with richer, more powerful applications. The cloud, beyond the buzzword, is a considerable progress in IT. Computer and network hardware gets commoditized, and available through software interfaces. It means not only storage, but even more importantly computation becomes a resource actionable through software. With the cloud, the only job left to run applications through the web is to write them.

And with Opa, we aim at providing the best high-level programming language and environment to give more power to web application developers. By automating many of today's web programming tasks, we allow the developers to focus on features, user experience. What was repetitive is done once and for all, in a clean way thanks to strong static typing. As the term DevOps takes off, we also think beyond development: How to deploy, monitor easily applications in the cloud.

We have even more ideas how to improve both development and runtime of this application. This involve several complex research problems, because we're interested on having the best of two worlds: Distribution and fault tolerance. As the Opa community grows, we are committed to push the boundaries of web development even further.

InfoQ: I understand that Opa itself was written using OCaml. Why OCaml? Did OCaml influence Opa, the language?

François-Régis Sinot: OCaml is an excellent language. It allows to combine several paradigms (functional, imperative, object-oriented); it is strongly, statically typed, thus avoiding a number of bugs; it is compiled to efficient native code. It is well-suited to implement language-manipulation programs such as compilers, thanks to its ability to easily represent and manipulate complex data-types such as abstract syntax trees. Finally, among the few programming languages with similar qualities, OCaml is the most popular one in France (many students learn it as their first programming language).

Opa shares a number of properties with OCaml. It is essentially functional, allowing the imperative paradigm in a controlled and safe way (inside sessions and cells, because imperative states pose a number of technical and even semantic problems in a client-server and distributed setting). It is also compiled to efficient code on both sides: to native code on the server, and to JavaScript on the client. It is as well strongly and statically typed, but in a more flexible way than OCaml. For instance, it is often not needed to define types: many values and functions can be typed with "anonymous" types. This is similar to the polymorphic variants of OCaml, but our approach goes further and is compiled to more efficient code.

Opa was also influenced by other languages. For instance, we do not stick to OCaml syntax. Generally speaking, we adopted a more lightweight and permissive syntax. We also write function application in a way more familiar to users coming from C or Java, with parentheses and a fixed arity [the number of parameters in a function]. We also got inspired by Erlang's high scalability in designing our notion of session.

InfoQ: The Opa compiler generates JavaScript on the client side. Does it include a UI library (widgets, themes, etc.) like jQuery does?

François-Régis Sinot: The client side library includes, and heavily relies on jQuery. For instance, all DOM manipulation and effects are done through jQuery. This has the standard advantage of using jQuery, for instance a more unified behavior with different browsers; it also allows to use very easily standard jQuery plugins, such as jQuery-UI. It's just as well very easy to use external style sheets for themes.

In addition to this compatibility layer, we also provide an extensive library of widgets and components, written in Opa. All the standard widgets, such as panels, tabs, notification, date picker, auto-completion, are here. They are fully customizable by passing them a typed configuration. Regarding theming, we are currently adding facilities to make it easier for users. For instance, we recently added a library to include Twitter's excellent Bootstrap CSS toolkit. (There's a post related to his on our blog: http://blog.opalang.org/2011/08/bootstrap-in-opa-hello-dear-readers.html )

InfoQ: What’s on the server side?

François-Régis Sinot: The server side code is compiled to native code. At the moment, we do this through OCaml, to take advantage of all its optimizations. We also have a prototype implementation of an LLVM back-end, but it is not ready yet for release.

This compiled code is linked to a rich runtime library (written in OCaml), which includes our own web server and scheduler, as well as our native database engine, which can be either local or remote.

InfoQ: The Opa platform takes care of managing database and cloud access for the programmer. How does it do it? Where is the database and cloud connection information specified?

François-Régis Sinot: All connection information (IP, port, Unix account) are passed as command-line arguments to either Opa compiled server, the Opa database server, or the Opa-cloud deployment script. We are also planning to support configuration files, but it is not such an inconvenient to pass arguments on the command-line as it is common and advisable to write small scripts to launch and deploy applications. The Opa-cloud script is able to automatically deploy and distribute an application on several accounts and/or machines. In the future, we may include this feature in the compiler itself.

To communicate either with our database server or other servers in the cloud, there is no user/password (as for instance to connect to SQL servers). Instead, we use standard HTTPS with client certificates. This is fully secure and avoids writing password in the code or any versioned file.

Following popular request, we are currently in the process of implementing a compatibility layer for other databases. For these, we will either store access information in a configuration file, or advise storing it in an instance of Opa's native database. (Of course, we support multiple DBs in the same program.)

InfoQ: There is a lot of help having Opa generate an embedded web server and a distributed database management system upon compilation, but that could be a weakness also. Can the developer specify his own DB and web server if the default ones are not good enough for him?

François-Régis Sinot: The web server is not a very customizable aspect. Opa handles multiple threads of execution with fibers (co-operative light threads). The compiler transforms user code to a CPS representation, so that our scheduler can balance the pool of computation and I/O operations (we use epoll on Linux, kqueue on Mac OS). The developer can also explicitly push asynchronous tasks. So, it doesn't rely on the OS scheduler, there is only one main OS thread for all computations and I/O clients (our scheduler does the job). This puts a strong restriction on the runtime library that we can link to: it should not use OS threads either. Our web server is built exactly to fit our requirement, and cooperate nicely with our scheduler. We probably cannot expect this from a third-party web server. However, our web server very nicely fits under a HAproxy or Nginx, to handle redirection our load-balancing. That's exactly what we do for our deployments.

The database is a more arguable aspect, since most companies moving to Opa will have legacy data in other databases, and may have a hard time migrate all their infrastructure at once. So, as mentioned above, we are currently implementing user-level bindings to standard databases. They are not yet available, but they should be in a few weeks. Doing so may expose again users to e.g. SQL injections, so we will do our best to expose an API as secure as possible, automatically providing escaping where needed.

InfoQ: How does Opa handle scalability?

François-Régis Sinot: Any application written in Opa can (almost) automatically take advantage of additional cores or additional servers to distribute treatment, storage or delivery. Once you have compiled your Opa program (not caring much about scalability), you can deploy it automatically on several servers with the opa-cloud script, specifying accounts, machines and ports on the command-line. The sessions and cells (declared as cloud) are then distributed with our home-made, safe communication protocol (HLnet), and local databases turn to database clients. The script then launches a database server and an HAproxy for load-balancing. At the moment, we do not have much elasticity though, but we are working to improve it.

InfoQ: Are there real-life projects based on Opa?

François-Régis Sinot: Yes, we already have a few. First, opalang.org, the home base of Opa is itself written in Opa; and it is essentially designed as a generic CMS. On August 28th, after publication of a news on slashdot.org, it had 12000 visits and dealt with them nicely.

The most complex application written in Opa to date, is certainly jetleague.com, a travel search and booking site with an innovative interface. It features communication with many travel partners, based on various protocols; as well as a rich Facebook integration (login via Facebook, ability to invite friends, and to share trips and comments with them).

We also have Opa poker, a free Texas hold 'em peer-to-peer poker game available on the Mac App Store. It allows to host a standalone server on your Mac to play poker on a LAN, through a standard web browser (it has in particular a mode for smartphones). It also uses a peer-to-peer communication mode similar to Skype to allows gamers to join the online Opa Poker community, even through firewalls.

At the moment, we also have some ongoing project in Opa, among which a secured webmail for the French Department of Defense (DGA), and a data-intensive web application of off-line to online links, for plug-up.com.

InfoQ: Opa is licensed under AGPL, a license that forces developers of derivative works to make public the respective source code. This is not very well received by many companies, including Google which does not accept AGPL software to be hosted on Google Code. Why was Opa licensed under AGPL? Are applications built with Opa under AGPL too?

François-Régis Sinot: Indeed, some people don't like the AGPL. And as you say, Google still bans AGPL internally, although it finally accepted it on Google Code.

We chose the AGPL first because our goal is to build a strong open source community around Opa. Technically, as not only the compiler but also the standard library is released under AGPL, every program written in Opa must also be released under the same license. We are convinced that Opa, thanks to its advanced typing system, allows better code reuse - and we hope that as the quantity of open source in Opa increases, there will be many highly-reusable components allows better code reuse that will be a strong asset for the Opa community.

And at the same time, this licensing provides a simple business model for MLstate, the company that funded the development of Opa. Indeed, MLstate sells proprietary licenses of Opa that allow developers to keep their applications written in Opa closed-source. This way, everybody contributes to success of the open source Opa, either by bringing new code to ecosystem, or funding its development.