The goal of the new JClouds 1.0 release is to provide a common interface for managing compute nodes and storage nodes across many vendors, providers, frameworks and APIs from IaaS to PaaS, says JClouds founder Adrian Cole.
JClouds supports 30 different providers from all over the world. Developers and DevOps use JClouds from downstream tools like Apache Whirr (runs cloud services like Hadoop, HBase, elasticsearch, Voldemort and more) or Pallet(library for provisioning, configuration and management of cloud computing resources), or use JClouds directly as a library via their APIs and via their ant tasks.
InfoQ got a chance to catch up with Adrian Cole the founder of JClouds and an active community leader in the cloud interoperability and devops space.
InfoQ: What major items have been added in the JClouds 1.0 release?
There are quite a few new things in JClouds 1.0.0, too many to cover, but here are the highlights:
On the usability side, David Santiago and Mattias Holmqvist completely revamped our blobstore and compute clojure apis to feel more natural to the language. Tim Peierls revised our java api for BlobStore to be more intuitive to write.
As JClouds becomes embedded in management and application platforms, modularization has become a priority. Thankfully, Gustavo Morozowski and Ioannis Canellos took a lion's share effort wrapping up OSGi bundling and Karaf integration: OSGi support was one of our earliest issues!
We've also had a number of features geared at power users. For example, Tibor Kiss brought our BlobStore into Petabyte scale, honing and testing our internals inside of a NGS platform.
Jeremy Whitlock led our first version of ProviderMetadata, something you can use to query the available clouds and details such as their console URLs, friendly names, etc. Cloudsoft are using details like this to have jurisdiction-bound deployment rules, something folks outside the US are pretty keen on.
For those security conscious, we have a really nice tool called AdminAccess, which locks-down nodes and creates a personalized user for you in a single step.
Important to many is our continued breadth of support: Grid Dynamics and GigaSpaces donated support for the popular OpenStack Nova platform, and we've also added many new cloud services from providers as far apart as Ninefold in Australia (XEN and VMWare hosting) and Stratogen in the UK (VMWare hosting).
InfoQ: In the InfoQ interview dated Dec 23, 2010, you mentioned that JClouds might work with libvirt soon so one can use JClouds for virtualization control? Has there been any progress on that front?
Not too much. To a certain degree, we are leaning on cloud controllers such as OpenStack Nova and vCloud until this is together. Another option is our byon (Bring your own node) provider developed in collaboration with Twitter recently. BYON allows you to give JClouds a file or web url with machine details in yaml syntax. It is a nice workaround to platforms we don't yet support.
InfoQ: What Cloud APIs does JClouds support (vCloud, Amazon EC2, etc.)?
JClouds supports storage installations of EMC Atmos, OpenStack Swift, Eucalyptus Walrus (S3 clone), (other) derrivations of Amazon S3, and even a bunch of files on disk. On the compute side, we support installations of vCloud (initiative from VMWare), ElasticStack, Eucalyptus (Open Source Amazon EC2 clone), Deltacloud (some overlap with JClouds), many derrivations of Amazon EC2, and the byon format.
Cole went on to explain the feature roadmap for JClouds. There has been a lot of pressure recently to help with PaaS abstractions so support is on their roadmap. PaaS focus is new to JClouds which previously mainly focused on IaaS abstractions. This PaaS effort included working with Google App Engine and CloudBees Run@Cloud.
Another area of focus is throttling and scaling. There have been some issues with some providers when trying to provision 100+ nodes. JClouds is working on some added built-in strategies so soon you can ask for large pools of resources with a decent chance of success.
Cole went on to explain that load balancing and virtualization support is also being worked on and should be complete soon. The virtualization support will include alternative boot sources, ISO mounts, PXE, building your own images, and cloning images from other machines. It sounds like the virtualization support will support private clouds well.
InfoQ: How does JClouds compare with Jets3t , typica, Deltacloud and Dasein? Can you mention quality, performance, scope, maturity, etc.?
The most fundamental difference between JClouds and a lot of alternatives is that we spend a lot of time on equivalences of metadata. This is really helpful as you can expose your requirements directly, ex. I need an Ubuntu 10.04 server in Ireland. This is a pretty hard process to get right, and we constantly try to get better at it.
James (Murty) from Jets3t (Amazon S3 focused API) made a very solid library as has David from Typica (Amazon EC2 and other Amazon services focused API works with clones like Eucalyptus). They both have shared their experiences during JClouds development, notably so James, who was a founding dev and put several months of hard work in. Both jets3t and typica are mature libraries and were around before JClouds. That said, they don't focus on portability, so you can't really exchange either with JClouds. Generally speaking, we play catch-up with amazon features while jets3t or typica have near or complete parity faster. When we've measured performance against jets3t, we've fared better. Of course, we wrote the tests :)
Dasein is a closer match to JClouds. It was founded about the same time, and in fact uses JClouds components for a good portion of the cloud providers. That said Dasein has a few providers JClouds doesn't support and visa versa. Dasein focuses on single-server operations, while JClouds focuses on bootstrapping set of machines. What's really cool about Dasein is that it exposes a lot of gears such as firewall rules we haven't finished modeling yet. (Adrian Cole is also Dasein committer.)
Cole went on to discuss Deltacloud which has a very "beautiful REST API". Deltacloud's focus is on API portability, whilst JClouds focuses on portable ways to execute use cases. Deltacloud can discover the various means to customize a machine, if they are available. While JClouds allows you to provide a bootstrap script and it will sort out the details. It is no surprise that JClouds can integrate with Deltacloud.
InfoQ: Does JClouds support Eucalyptus? What are your thoughts on Eucalyptus?
Yes JClouds does support Eucalyptus. My opinion of Eucalyptus is positive. They take ownership of compatibility issues we raise and dedicate effort to seeing them through.
Eucalyptus is a popular open source Amazon EC2, S3 etc. clone. It gets used for private cloud computing.
InfoQ: If I were to start my own private cloud, which platforms would JClouds support? Now? Future?
Presuming you mean an on-prem (on premises) compute cloud, you could use Eucalyptus, OpenStack, or vCloud 1.0 today. Very shortly, we'll also support cloud.com CloudStack 2.2.8, which is a quite mature and robust product available under commercial and GPL license. By the end of the year, we'll support vCloud 1.5, and possibly even OpenShift Power. There are a few other cloud controllers out there, like OpenNebula and Abiquo, and they'll get woven in as demand justifies.
InfoQ: What are some big wins that JClouds has had in the last three months (developmentally, architecture enhancements, etc.)?
OSGi has been a quite positive progression of the last 3 months. The idea that you can take your connection to a cloud resource offline, or upgrade it while your app is still running is a very relevant concern. Like typical datacenter ops, clouds go into maintenance windows and so you may need to dynamically adjust to things like this. OSGi and other modular systems are the key to this, and our first step into OSGi was a big deal for us and our users.
Another big deal was Tibor Kiss' optimization of data transfer. We've recorded speeds of over 300MB/s between EC2 cluster nodes and S3, which makes Petabyte scale transfer very actionable in the cloud. Andrew Phillips also managed to resurrect our TweetStore demo, which transfers twitter updates (thanks twitter4j!) to BlobStores from within Google App Engine. He took this a bit further and now has it simultaneously deployable to CloudBees Run@Cloud PaaS. We're starting to see the subtitles of mashing together IaaS and PaaS, and hope to make some useful tools to others who encounter the need.
InfoQ: Do you have any thoughts on when someone might use Google App Engine over EC2 over Rackspace over Azure over an internal private cloud?
Well, there are a dozen choices in PaaS now, so the whole GAE, Azure, etc decision is evermore tricky.
PaaS or IaaS tends to be a decision of cost, service level, or congruence with your requierments. If you can get your requirements met with the limited knobs available in a PaaS offering, it could be worth it. You are paying for the luxury of less details. If you are looking into hosted PaaS offerings, pay close attention to the data layers. You'll want to know how import/export works, house rules, historical availability, and maintenance windows.
I would be generally careful about attempting to run a PaaS on your own. They are often composed of many pieces like Rabbit MQ, CouchDB, Nginx, Nagios, etc. If you want to roll your own PaaS, you should investigate the dependencies to ensure you have the ability to support all of them, or know where to go to get that support. Also, be suspicious of single-server quick start PaaS or IaaS demos. Realistic, multi-server deployments are often difficult, manual, and/or undocumented processes. It is these concerns that make hosted services all the more attractive.
If you need to launch hundreds of nodes at a time, you pretty much have to use EC2. I've found Rackspace are about as good as any other provider at smaller scales. I'd expect decisions about which public cloud providers to deploy to to become more centered on business value. For example, who do you want to be close to for performance reasons? Are they deploying in cloud X, if so which regions? Which clouds serve the jurisdictions you are interested in? Which of your technical partners have experience in cloud X? On-prem is a valid decision and can make a lot of sense especially if you already have expertise and infrastructure. This year, there's a lot more products to help with on-prem cloud, too.
InfoQ: Which provider of IaaS is the easiest to work with (integrate with)? Which is the most buggy?
Easiest providers are those who run ElasticStack. They pretty much always work, extremely consistent across providers, and are very simple from a programmatic perspective.
There's must be a competition for most buggy as it seems many are fighting for the title! Seriously, I won't name and shame. Suffice to say that those who write their own cloud controllers, and haven't a history in writing software as a core competency tend to be more buggy.
InfoQ: What are your thoughts on Amazon BeanStalk? Do you plan on abstracting more of the APIs around Amazon, i.e., Amazon Simple Notification Service, Elastic Load Balancing, and Auto-Scaling?
BeanStalk is interesting and had a disruptive impact on our perceptions of where Amazon starts and stops. I think it was a warning sign to some ISVs who are now somewhat competitive with BeanStalk. That said it is a fairly simple service and doesn't really go into config management, so until it does, the real impact is limited. With regard to other AWS services, we have some already using Elastic Load Balancing support in JClouds (for ex. OpenShift Flex). Other APIs are basically waiting for enough people to care that we don't yet support them.
JClouds 1.0 seems like an impressive release in a rapidly evolving space. It seems that JClouds is growing in features and purpose.