Apache Tribes, a Tomcat 6 module, supports group communication in the server cluster. Filip Hanik talked about the challenges in heterogeneous clusters and how Tribes helps with group communication requirements of Tomcat clusters. He did a presentation at SpringOne Americas conference about Tribes messaging framework.
Filip started off the presentation saying that there are several open source group communication project including Appia, Spread, Erlang, and JGroups. He talked about the uniform group model where all nodes in the cluster are identical and they process, send and receive messages in the same way. Many group communications modules are built for a uniform communication model. But in most heterogeneous cluster implementations this is often not the best solution to achieve the performance and scalability that is needed in the cluster.
A non-uniform group communication model is a better solution when processes on each node of the cluster are dynamic and are running in heterogeneous hardware environments. The messages are sent with different priorities and guarantee levels.
Tribes is a messaging framework with group communication capabilities which was created out of the cluster/session replication code from Tomcat 5 container. It is the communication framework for Tomcat's cluster implementation. One of its goals is to simplify peer-to-peer and peer-to-group communication for distributed applications. Tribes supports two types of message delivery, a concurrent message delivery that can be used even between two nodes and a parallel delivery which can be used to send messages to multiple nodes.
Other features in Tribes framework include:
- Guaranteed Message Delivery: The default implementation is based on TCP and uses java.io and java.nio packages.
- Guarantee Levels: Tribes supports 3 levels of message delivery guarantee (NO_ACK, ACK, and SYNC_ACK).
- Per message delivery semantics: These semantics allow for each message to be delivered differently as well as use a different guarantee level per message.
- Pluggable Interceptors: These can be used to intercept any events through defined methods and react on message attributes (flags). ChannelInterceptorBase class can be used to minimize redundant code for non intercepted methods.
- Delivery Feedback: Tribes aims to deliver feedback for each message and each delivery semantic (NO_ACK, ACK, SYNC_ACK). Message delivery can be either synchronous or asynchronous.
- Concurrent & Parallel Delivery: Concurrent delivery means more than one message can be sent or received at any point in time. There is no "message blocking" meaning a 10 MB message with SYNC_ACK guarantee level will not stop a 10 KB NO_ACK guarantee level. Parallel delivery allows to send a message to multiple destinations in parallel using one thread (NIO).
- Fixed Node Hierarchy: This feature supports the ability to determine cluster leadership, auto merge groups, and node discovery where multicast might not work.
- Failure Detection: This includes a simple interceptor TcpFailureDetector to provide feedback when a cluster member goes down. There is no need to wait for timeout and no risk of node pings getting stuck on a busy network.
Tribes also supports features like RPC messaging and a JNDI Channel to bind a channel into a JNDI tree. The framework architecture includes the following components:
- Channel: This is the first interceptor in the chain. It holds a list of one or more ChannelListeners and MembershipListeners. It serializes and deserializes messages and supports ByteMessage for transfer of pure byte[] data.
- Interceptors: Some examples of the interceptors are failure detection/static membership, total order or per member order, leadership election/message data encryption, message dispatch (asynchronous messaging), and all or none delivery guarantee.
- Coordinator: This is the last interceptor in the chain. It coordinates I/O components such as Sender, Receiver and Membership.
In the presentation, Filip also demonstrated a sample application on how to enable Tomcat's clustering and the configuration options for enabling a webapp for session and context attribute replication. The file server.xml includes configuration for cluster elements like Cluster, Session Manager (DeltaManager or BackupManager), Channel (Tribes), Membership (supports two ways of membership: Dynamic membership which uses multicasting to discover other nodes at run-time and Static membership where we define each node in server.xml), Messaging (happens over TCP; each node has a receiver and a sender), Receiver (receives cluster messages), Sender (sends cluster messages), Interceptors (similar to valves in terms of functionality), Valves (initiate session replication at the end of each request), and ClusterListener (supports custom messaging listeners for certain types of messages).