In 2002 David Luckham introduced the term Complex Event Processing (CEP). In papers several years later on the history of CEP, Luckham showed how although event processing has been a core part of distributed systems and simulations for decades, CEP is an evolution and not the same as, say, context switching. In fact CEP can best be summarised as:
Complex Event Processing is a technology for low-latency filtering, correlating, aggregating, and computing on real- world event data.
Over the intervening years, CEP was discussed within the context of SOA as a natural fit and at times called "a new physics of computing" or, as the "next big thing" in 2008 by WebSphere CTO Jerry Cuomo:
I've started to hear some people talk about this in the context of SOA, and I really believe it's the next big thing in SOA, and that's event processing. I don't know how much you're hearing about this, but certainly we're taking it very seriously. I see a spectrum of event processing. There are types of event processing that we are doing very well today on the simpler notion of event processing.
There have been several companies offering and developing CEP products or open source projects, such as Esper, Drools Fusion, Oracle and Sybase. At the same time we saw people talking about Event Driven Architectures (EDA) and Event Stream Processing (ESP), building on the concepts of events and CEP, with the likes of Gartner and Oracle even suggesting that the combination of CEP and SOA was the genesis of the next generation of SOA, i.e., SOA 2.0, though that didn't go down too well with the wider industry. However, as Eric Roch points out recently, often the term CEP can make it difficult to sell the underlying concepts:
The term CEP and the idea of its complexity is further exacerbated by the uses cases often associated with the technology. The most common use case stated is algorithmic trading. Can you imagine going up to your CEO and saying you want to try CEP, it’s the cool technology Wall Street is using for algorithmic trading? If you are lucky you might get, “Why in the heck do we need that in our business?” - Or maybe just an aggravated, dismissive stare.
Eric agrees with Luckham's original analysis that there's really not a lot new behind the concepts underlying CEP, such as Message Oriented Middleware and rules engines. But even here there can often be far too many complex terms to cloud the arguments:
But when you throw in all the buzz words– complex events, event channels, inference engine, Rete networks, forward chaining, state machines, tuples, algorithmic trading it does sound like you need a PhD to make it work.
Fortunately over the past decade implementations have improved and tooling is now often a critical component:
CEP tools, especially the most mature ones, are model driven and fairly easy to use. You really don’t need to know the internals of an inference engine to write business rules. They are in fact easier to write than a bunch of nested if statements.
However, over the past couple of years the amount of discussions and hype around CEP have reduced. This could be because of a similar reduction in that associated with SOA. It could be because CEP has not really delivered on the original promise. Or, more likely, it is simply because as Luckham pointed out, events have been a part of software systems for decades and now CEP is just a core or natural part of SOA implementations and people simply take it for granted.
Of course with the advent of Cloud and the push to take existing software investments there, we have seen discussions on moving SOA into the Cloud and of course that means all things associated with SOA, e.g., CEP. However, it seems that this may not be as simple as it may appear initially as Colin Clark reported last year:
Perhaps it’s because that the current incarnation of CEP products don’t lend themselves well to grid deployments. None of the current crop of products include a message bus or cache, something that’s necessary for grid deployment. And because this crucial component is missing, chances are that the products were never really architected for the “I don’t care where a message is coming from and I don’t care where it’s going” kind of world. The kind of world we are increasingly living in. The kind of world that readily lends itself to cloud computing.
As mentioned in a separate article, it is possible that some of the inherent assumptions behind CEP, such as low latency, or the "complexity" of current implementations is an impediment in using them in the Cloud. Or maybe CEP will be as critical to Cloud as it has been to distributed systems over the past 5 decades? So are you using CEP in your SOA applications? Is it a core part of your SOA infrastructure? Are you considering using it within the Cloud and if so, do existing implementations match your requirements or do we need something more Cloud specific?