Hazelcast, maker of distributing computing technologies and tools, has released version 0.6 of Jet, their open-source stream processing engine. Jet has been publicly available since February 2017 and version 0.6 is a major release in Hazelcast's numbering scheme.
Jet aims to provide a capability that allows developers to build distributed, fault-tolerant data processing pipelines on top of a variety of data sources. It uses Hazelcast In-Memory Data Grid (IMDG) as an internal data management technology, and builds on Hazelcast's expertise in the in-memory computing space. Hazelcast describes Jet's goal as: "to enable fast big data as just another part of application infrastructure".
Jet is a Java-first technology like all of Hazelcast's offerings, and is squarely aimed at Java developers and teams. One of Jet's core APIs is a distributed version of java.util.stream that provides an obvious on-ramp to Java developers who are new to distributed processing.
New functionality and enhancements in Jet 0.6 includes:
- Improved and simplified high-level API (Pipeline)
- Improved job management capabilities
- Computation elasticity
- Updated connectors – Kafka v1.0 and HDFS v2.8.3
- Java 9 support (including modules)
- Spring integration
- Examples and demos repo
The announcement also indicates that Jet is now using v3.10 as its embedded Hazelcast IMDG. This version has not yet been released, but is expected soon. Jet's deployment topologies are similar to the options offered by Hazelcast IMDG: Jet can either be embedded in a JVM application and run in-process, or it can be run standalone in a more infrastructural mode.
Internally, Jet represents distributed computation jobs as directed acyclic graphs (DAGs) and uses co-operative multithreading (aka "green threads") and concurrent queues to provide the low latency capability. Hazelcast credit Martin Thompson and his "Mechanical Sympathy" movement and approaches to wait-free algorithms as being an inspiration for the design of Jet.
The team claims significant performance improvements in version 0.6 of Jet vs similar processing done with competing technologies, such as Apache Spark.
Jet version 0.6 is out now, and can be downloaded from the Hazelcast website and used under an Apache 2 license.