As part of the upcoming Eclipse Ganymede release, scheduled for June 25th, InfoQ will cover a series of Eclipse subprojects. Today, the subproject is Equinox p2 (Provisioning Platform), which is a framework for provisioning Eclipse-based applications. InfoQ spoke with Jeff McAffer and Pascal Rapicault to learn more about p2 and what it provides.
In describing the capabilities that p2 brings to Eclipse, Rapicault said that the 1.0 version of p2 is designed to replace the existing Update Manager (UM). McAffer iterated through several features:
- Automatic retry of download across all available sources
- Transparently picks the best mirror
- Bundle pooling allows sharing of plug-ins across multiple eclipse instances
- Ability to manage the complete installation (exe, ini, etc)
- Ability to manage and update an Eclipse instance without running it
- Makes is easy to create headless and custom update user interfaces
- Validates plug-in inter-dependencies so you only install plugins that work together; if you can install it runs
- Multi-threaded download making downloads faster
- Only installs plugins that you need, so it reduces the number of plugins installed
- Create uber-update sites that know how to get plug-ins from multiple sources
- p2 means you never have to install Eclipse again, you just need to update
- An installer that can be run as a regular Java application or using Java Web Start. This installer can be used to install and configure all aspects of Eclipse
- Cleaner end-user workflows
When asked what problems with UM p2 was meant to address, McAffer said that the list of features above corresponded very closely with UM issues. Rapicault pointed out that UM did not evolve along with the Eclipse platorm, and that UM was rarely used by the development teams. One of the goals of p2 is to be an attractive choice for both end users and platform developers, and Rapicault said:
This turns out to be a big challenge as we have to convince a crowd that never really bought into UM that we can actually help them, and they have to gain confidence in the operations that we are doing for them. This is actually a very hard thing as people have learnt to cope with UM with various workflows that we now ask them to revise.
p2 also brings new online install capabilities to Eclipse, with simple installers that weigh in at a few megabytes and a customizable Java Web Start installer which will be integrated into the new web-based install wizard which is being created by the Eclipse Packaging Project. McAffer also mentioned that these installers are only "the tip of the iceberg" for the capabilities that p2 offers for provisioning Eclipse-based and OSGi-based systems. Rapicault added that those who wish to extend p2 will find that it has a very modular set of building blocks which alow for managing an OSGi-based application on desktops, servers, or mobile devices. McAffer indicated that p2 is not an Eclipse-specific technology, and said that it is designed to run in web applications, in headless environments, as part of larger applications such as the Eclipse IDE, and as a standalone item. McAffer also added that "there is nothing in the design or implementation that prohibits it from provisioning arbitrary applications".
McAffer also gave more details about running p2 inside of a web application:
p2 can be packaged inside of a web app. As you may know, Equinox, the OSGi implementation that underpins Eclipse, is being used quite extensively on servers these days. In some cases it is as the implementation of the app server and in others, the OSGi model and Equinox functionality is exposed to the application writers. Using the Equinox servlet bridge it is possible to get all the power and flexibility of Equinox and OSGi inside a WAR. For example, you can deploy a small, standard WAR that contains just Equinox and p2 and then use the p2 provisioning technology to configure and run the real web applications. These applications can be dynamically extended and updated.
As part of the development of p2, several major challenges were faced. InfoQ asked both McAffer and Rapicault to describe the challenges and the solutions to them, and McAffer answered:
This is a great question. The team was lucky in that we were able to learn quite a bit from UM and other provisioning solutions. Here is a sample of the kinds of challenges we encountered and what we did to address them.
Metadata - Directly and indirectly metadata structure and format consumed months of our time. The first question that many people ask is "what's your metadata format (aka XML) look like?" or "Can I see the schema?". Fundamentally our answer is "we don't specify that" and "no" (respectively). The problem with spec'ing a format is that you then assume you are "the one". In today's world that is just not true. There are old Update Sites, OSGi Bundle Repositories, Maven repositories, ... End users want to connect to all of these and more. Our answer was to specify metadata at the API level and leave the format details to people who supply repositories. As long as you can implement a few simple interfaces, you can supply a repository. Of course we supply a repository implementation that enables all of p2's bells and whistles but p2 works fine with traditional Update sites and other contributed repository implementations.
Repositories - We determined that repository flexibility was going to be key to the use and adoption of p2. Many people have existing stores, formats, systems and p2 is not going to supplant them. Key to this is the separation of the metadata and artifact structures. This has turned out to be a real win by and gives p2 considerable flexibility in create new, and reusing existing, data stores. We also decided to make repositories pervasive in p2. When you install bundles they actually go into a repository that can in turn be used to provision other configurations or systems. The configuration of components that make up a system is also a repository. To enable this we use the notion of mirroring everywhere. You don't really download things so much as mirror them. As a result, you can provision a system using another, already provisioned system. We also abstracted out the notion of artifact format so one artifact repository can store multiple forms of the same artifact. For example, the standard JAR, a JAR packed using Pack200, the binary diff between two versions of the artifact, ... can all co-exist. p2 then selects the most efficient from to use when provisioning a given system.
Constraint resolution - The base set of dependencies that can be expressed in an OSGi-based system is relatively simple. However, there are quite a number of other constraints needed to define and provision real enterprise systems containing a variety of different elements. We opted for a generic "requires and provides" style constraint system but the really cool part is the pseudo-boolean constraint engine, SAT4J, we use for selecting components to build a system. In essence the generic constraints get mapped into a number of pseudo-boolean equations and sets of weights that express the desired outcome. This is fed into SAT4J and out pops a solution. The solution is mapped back onto the components to tell p2 what to install, uninstall or update. This is simple and fast. Even better is that the solutions can be optimized by using different fitness functions. So, for example, one could write a function that optimized for space (installing the smallest configuration) or the least change or... There are a number of active research teams looking at this kind of thing and p2 is set to leverage those results.
Flexible architecture - Eclipse is used in a great number of different scenarios from embedded systems to rich client applications to tooling to server side. This in turn puts a huge demand on the provisioning infrastructure to be flexibly configured. Pluggable repositories, different provisioning policies, multiple transports, provisioning logic located on multiple machines, remote management, ... To enable these we took a highly modular approach both in design and implementation. The p2 structure in Ganymede should be familiar to most Update Manager users but under the covers there is an enormous amount of flexibility.
Rapicault then added:
For me, update manager compatibility as well as cohabitation story (UM / p2 in the same system, which is the default in the SDK) was probably one of the most challenging as UM had provided, over its 7 years of existence, many ways to manage the system, especially some that were at odds with the p2 way of things. For this, we did some archeology in bugs, code and other document we could find to ensure that we were not leaving use-cases behind. Related to that is simply replacing a 7 years old piece in a year.
Another one was to get out of the "prototype" / "let's build a really flexible super cool platform" mindset and just focus on building something that ppl could use. For that we frequently asked ourselves "was that in UM"? That got us to ship.
The resolver was also a very challenging part, but from the pure complexity inherent to the problem. After several months of research and prototype, I finally found 2 research articles describing the usage of SAT solvers in very similar situations. It just felt like the right solution to the problem and I started prototyping using SAT4J. Once the PMC got convinced that putting a solver tackling NP-completeness in eclipse was the right way to go, we were set. We were also lucky that the SAT4J team, in the person of Daniel Le Berre and Anne Parrain, bought into our story as well and could provide an incredible amount of help. We have established there a true bilateral collaboration. Stepping back from it, I'm really glad that we made that choice as we already have plan to leverage more of the solver capability to advance the way sets of plug-ins and products are described in future releases.
Building up a community from the start was also challenging. In a sense, it was relatively new for us to have to work with the community right from the birth of the project but also on an ongoing basis. We were lucky to be able to meet most of the future contributors at the provisioning workshop and then at the equinox summit which allowed us to share our vision and gather the areas of expertise / interest. All that paid off, since there is already one company who has developed a commercial offerings based on p2, and other companies are writing p2-based installers for their commercial offering composed of other things than just an eclipse based application.
With respect to future plans for p2, McAffer indicated that the 1.0 release was intended to create the infrastructure which would allow for further development. Tooling, provisioning policies, performance improvements and an expansion to more runtime systems are all on the list, and p2 will be usable in a wide range of situations. Rapicault added that growing the community and providing actions to enable a better integration with the desktop are planned. McAffer also said that, just as the Eclipse team did not implement support for all of the languages that are supported by the Eclipse IDE, the p2 team will leave room for others from the community to generate solutions based on p2.
When asked how p2 would be integrated into or affected by e4, McAffer said:
From the beginning I have seen p2 as an e4 effort. It happens that it came along early and is available to be used in 3.x Eclipse but our design thinking included e4-like scenarios. E4 is slated to be more dynamic, more modular and more decoupled. These characteristics present great opportunities as well as great challenges. We want p2 to be extremely approachable to end-users and system integrators yet give people the power to manage e4's highly integrated systems. p2 will be an integral part of e4.