Getting Started with Shuttle Service Bus

Shuttle Service Bus is a free .NET open source software project that provides you with a new approach in developing message-oriented EDA systems.  Even though it is still in its infancy it is already used in production systems.

Here are some key points:

• Developed with C# on .NET 3.5
• The core has no dependencies on third-party products or projects
• Supports command messages as well as event messages (Pub/Sub)
• Has integrated message distribution
• Includes an administration shell to ease operational requirements
• Makes extensive use of interfaces to facilitate replacing or extending functionality
• Fault-tolerance via automatic retries

Why Would I Use a Service Bus?

Although using a service bus requires somewhat of a paradigm shift there are many advantages to going this route.  By designing your system to have a service bus perform very specific functionality on a specific endpoint you can focus on getting that bit of software working in isolation. 

Such an endpoint can then be independently versioned and maintained if you have decoupled it to the required degree.

This basically means that you will end up with messages being sent between various components and that these messages are processed asynchronously.  This results in a fire-and-forget scenario that requires some careful thought as to how the system works. 

The resulting UX can be quite different from traditional implementations where all actions requested by a user are processed immediately.

An example may be something as simple as sending an e-mail.  You may set up an endpoint that handles the relevant command message type.  So, when you need to send an e-mail you use something like this:

	Bus.Send(new SendEMailCommand
	              {
	                    From = "someone@from.com",
	                    To = "someone@to.com",
	                    Subject = "testing e-mail",
	                    Body = "Hello"
	              });

You may be wondering how this differs from simply sending the mail directly yourself:

• The message sending call is immediate and we do not wait for it to complete
• Since the requests are queued we avoid a bottleneck
• Should the e-mail sending fail it will be retried automatically
• Given correct data we can be sure that the e-mail will be sent eventually; or, at the very least, manual action can be taken in the event of a problem

So our client code does not care how the endpoint actually sends the data.  It may be using SMTP, or even some custom web-service.

How Does Shuttle Do Its Magic?

Shuttle Service Bus relies on two things:

• Messages, and
• Queues

The queuing infrastructure can be anything.  You would, typically, want to use a real queuing technology such as MSMQ.  Shuttle provides support for MSMQ and Sql Server table-based queues straight out-of-the-box.  Should you wish to use anything else, it is a matter of implementing the relevant interfaces and you will be good-to-go.  Shuttle makes use of a URI structure to represent the queues, e.g.:

• msmq://{machine}/{queue}
• sql://{connection-name}/{table}

To implement your own queue, you would simply pick a scheme and your queue implementation would decode the structure.

In order to host your endpoint, Shuttle makes use of a generic service host to simplify deployment.  Implementing a new endpoint is a matter of configuring which queues to use and then starting up your service as shown in the following config file and code snippet:

<?xml version="1.0"?>
<configuration>
  <configSections>
    <section
      name="serviceBus"
      type="Shuttle.ESB.Core.ServiceBusSection, Shuttle.ESB.Core"/>
  </configSections>

  <serviceBus>
    <inbox
      workQueueUri="msmq://./inbox-work"
      journalQueueUri="msmq://./inbox-journal"
      errorQueueUri="msmq://./shuttle-error"
      />
  </serviceBus>
</configuration>

    public class ServiceBusHost : IHost, IDisposable
    {
        private IServiceBus bus;

	public void Dispose()
	{
	    bus.Dispose();
	}

	public void Start()
	{
	    bus = ServiceBus
	             .Default()
	             .Start();
	}
    }

It is important to note that the generic host can run as either a console application or it can be installed as a service.  This makes it particularly easy to debug your endpoint since you can specify the generic host to be the startup application when debugging within Visual Studio.

Once a particular message is retrieved from the inbox queue, the service bus will attempt to locate a handler and pass the message to the handler for processing.  Messages that have no handler can either be moved to the error queue or discarded depending on the configuration specified.

Command vs. Event Messages

Since a command is an explicit request that a particular function be performed, it is sent to a single endpoint.  This means that in order to send a message you need to know that an endpoint implements a specific behaviour.  Therefore, commands result in a higher degree of behavioural coupling.  When sending a message, the endpoint that the message should be sent to is configurable so that it may be changed at any time.

As an example you may have commands such as:

• SendEMailCommand
• ConvertDocumentCommand
• DeleteFileCommand
• CancelOrderCommand

Events, on the other hand, may have zero or more subscribers.  Typically, an event would be defined since it is required in some business sense.  So it stands to reason that there would be at least one subscriber, unless the event is defined in light of future requirements.  When an event is published, each subscriber will receive a copy of the event message.

This differs from message distribution where a distributed message will be sent to only one worker inbox queue.

As an example you may have events such as:

• EMailSentEvent
• DocumentConvertedEvent
• FileDeletedEvent
• OrderCancelledEvent

Shuttle Service Bus allows you to specify headers for messages in order to add any necessary ad-hoc data to a message.  There is also a correlation ID that may be used to group together related messages.

Scalability

Shuttle buys you quite a bit in terms of scalability since messages are queued and no immediate bottlenecks can arise.  Even though endpoints may be configured to be multi-threaded, this does not mean that a particular endpoint cannot fall behind due to an excessive number of messages being received.  Each and every endpoint has the built-in ability to distribute messages to another endpoint if any other endpoint notifies the distributor that it has threads available to perform work.  To get this going is simply a matter of configuration.

Modules

Shuttle makes use of an observable pipeline structure.  Events are registered in the pipeline in a specific order and observers may be registered in the pipeline to respond to particular events.

In keeping with the extensibility you can register your own module implementations in Shuttle.  These modules typically plug into specific pipelines by adding observers that respond to particular pipeline events.  You can even add your own events into the pipeline upon its creation.

What about Dependency Injection and Logging?

Since Shuttle makes such extensive use of interfaces for decoupling you are free to plug in any DI or logging implementation you feel comfortable with.  The default implementations do not rely on any third-party components but there is currently a Castle Windsor implementation for DI and a Log4Net implementation for logging that may, optionally, be used.

An Example of Shuttle in a Production Environment

Shuttle has been implemented at a large South African short-term insurance company with great success to replace an aging document indexing system.

Clients send in claim-related documents via e-mail that are received by the Lotus Domino e-mail system.  These e-mails are then extracted by the FileNet Email Manager application and placed in the IBM FileNet Content Engine.  The content engine is configured to respond to any new documents with an e-mail classification that are committed. The content engine code that handles these arrivals writes out an XML file containing the relevant data.

From there, a Shuttle Content Engine endpoint picks up these XML files and an event is published indicating that new content has been placed in the content engine.  The structure of this endpoint is such that it is not specific to the indexing process.  It is therefore possible to re-use this endpoint to publish any new content.  It is up to the content engine to simply write out the relevant XML files.

The Shuttle Indexing endpoint subscribes to the new content messages and as soon as one arrives it starts tracking all the documents that arrive for a specific e-mail since each e-mail has a unique identifier that is attached to the content engine metadata of each document.  Once all the documents have been gathered for an e-mail, command messages are sent to the Shuttle Document Conversion endpoint to convert the HTML e-mail body and all JPG files to TIFF documents.

The document conversion endpoint is unaware of the indexing process and simply performs document conversions.  Once a document has been converted, an event is published informing that a document conversion has succeeded or failed.  Any system that requires document conversion would then subscribe to these event messages.  In order to establish whether or not a message is intended for a particular system, the system requesting the conversion can make use of the correlation ID on the conversion request command and/or it can make use of name/value pair headers added to the outgoing conversion request command message.  These headers are always attached to any related messages sent by Shuttle.

As soon as all the required document conversions are complete a command message would be sent to the Shuttle OvaFlo endpoint to create an indexing workflow instance in the IBM FileNet Process Engine. OvaFlo is a meta-workflow framework product developed by Ovations Group.

In order to perform the actual indexing, the users access the indexing web-based application.  This application pulls the next available indexing workflow instance from OvaFlo and the related documents are displayed for classification.  Each document is also linked to a specific claim and any other relevant indexing data is entered.  It is also possible for users of the system to request that certain documents be converted to TIFF format.  This is particularly useful when various documents that need to be individually classified have been placed in one file.  Once the user is happy with the data the task is submitted for completion. This is handled asynchronously by firing off a command message allowing the user to immediately continue with the next indexing task.

In some instances, it was noticed that the work arriving from the web front-end queues up behind background system work.  A particular example is that of document conversion where all the required conversions from e-mails arriving were being handled before the requests from the front-end.  We simply installed a separate Shuttle Document Conversion endpoint using the same compiled assemblies with the generic host but we changed the configuration for this endpoint to use its own queues.  The frond-end then sends conversion requests to this priority endpoint and the conversions are timely handled.  All the background conversions are sent to the existing endpoint.

So, in this instance Shuttle was used to glue together disparate systems.  The system is much more stable than the previous one having required fault tolerance built-in.  The performance is excellent with no backlog being created.

Conclusion

Shuttle provides you with another free option when implementing an ESB.  The project is hosted on CodePlex:

• Project site
• Wiki

Pros:

• New
• Highly extensible
• Free open-source software
• Administration shell

Cons:

• New
• Process state data has to be handled manually

In time, and with community support, the various implementations available can be extended to include more queue, DI, logging, and other options.

About The Author

Eben Roux has almost 20 years of experience in the professional arena as a developer, consultant, and architect within many industries and has provided strategies and solutions that have contributed to the successful implementation of various systems. He is owner of the free open-source Shuttle Service Bus project and believes firmly in the development of quality software that empowers users to get their job done.

Having come from a Visual Basic background, Eben first became a Microsoft Certified Professional in 1998 and has completed 3 Microsoft Certified Solution Developer certifications by 2003 (VB5, VB6, VB.NET). Since moving exclusively to C# development in 2007 he has focused on domain-driven design implemented within an event-driven architecture based on message-oriented middleware.