Beyond SOA: A New Type of Framework for Dynamic Business Applications - Part II

Part II – Building Dynamic Business Applications in Practice – a Tale of Two Adaptive Systems

Productivity improvements are the building blocks for increases in the standard of living. The experience in the United States suggests that extended periods of strong productivity growth are characterized by innovations in technology that are accompanied by changes in organizational structure and in business financing arrangements and by investments in human capital. Underlying these determinants of productivity growth, however, is a more fundamental factor: the willingness of society to transform itself dramatically, with the confidence that technological progress and the economic opportunities attending that progress will enable people to improve their lives.

–Roger W. Ferguson Jr. and William L. Wascher, America Economics Association Distinguished Lecture on Economics in Government: Lessons from Past Productivity Booms, Journal of Economic Perspectives—Volume 18, Number 2—Spring 2004

“When it came to automating these [enterprise] processes, roughly 10% is in implementing the technology, the other 90% is change and process management”

– Mark Evans, Tesoro Petroleum’s CIO, Managing Automation, March 2004

Summary of Part I

In Part I of this article we introduced an enterprise framework for building a new type of IT systems called Dynamic Business Applications (DBA’s). These DBA’s are well suited to support enterprises with dynamic operations, a category containing most current businesses. Part I also describes how relying on requirements collected through use cases as the primary input to system architecture and design limits current enterprise architecture approaches. The proposed new framework uses the business event model as a starting point and views the enterprise as an adaptive system with a clear information architecture. Based on a new theories of adaptive systems and information, the framework captures both enterprise processes and business changes at the fundamental level. Use cases are introduced as a way to refine requirements only later in the design process. This framework-first approach was inspired by traditional engineering methodology but it was modified to address adaptive systems like the enterprise.

Introduction

It is time to push craftsmanship aside and put proper engineering into software engineering. This is the only way complex software applications, especially in the enterprise space, can be built to common standards to be highly reliable, easy to change, and easy to debug. Selecting and executing the correct system architecture is the most important factor for this to work. The solution we propose makes it possible to standardize on only one system architecture, regardless of the software application and to avoid costly re-architecting.

As mentioned in Part I of this article, there is a fundamental difference between architecting software applications and designing other engineered products. Because software works with information, and information is the “carrier” for change, then change has to be built into the system architecture at the most basic level. Additionally, the way business operations change and the way that change is introduced to a system by a technical team take very different paths. These two organizations, one business and one technical, each responding differently to change and having different operations can be treated as two different adaptive systems with different requirements during the architecting process. This is why we selected the sub-title a “a tale of two systems” for Part II of this article.

In designing enterprise applications there are two adaptive systems involved

As mentioned in the Part I of this article, the only approach that can successfully address the complexity of building DBAs is to use a framework-first methodology. Engineers have used this methodology for centuries - but always to design “static” architectures. When changes were applied, the system most likely had to be taken out of service and modified. For instance, changing the product to be assembled by an assembly line requires the line to be stopped, changes applied, and the line to then be restarted.

Today’s software engineers building IT systems for business are most likely to follow the same path. When business changes create the need for application modifications, it is very likely to trigger a costly upgrade. Developers must start from scratch with new business requirements and the old design often goes through an extensive re-architecture. This upgrade cycle could take months. If the enterprise application is built by an outside vendor with its own agenda, the entire process could take much longer and the results could fall short of business expectations. As Mark Evans said it so well, over 90% of this effort is directly related to applying changes alone.

With the help of a new theory for adaptive systems and their information architecture (see sideline), our proposed solution to enterprise application design introduces two complimentary but unique frameworks. These two frameworks allow us to handle and coordinate changes in business operations and technical team operations effectively and efficiently. For the design process, business operations and technical team operations can be considered two distinct adaptive systems, each with their own requirements.

Business operations can be represented by the generic concept of a Basic Dynamic Business Platform. Applying the theory of adaptive system, any enterprise business function has three types of fundamental processes. The main process type supports normal operations, while the other two are responsible for introducing changes: the internal decision-making process handles management and other enterprise decisions to change and the business environment change process handles customer decisions and other external sources of change. As a result, the proposed framework for business processes, the Basic Dynamic Business Platform, is built around business operations or value cycle, with two types of changes handled by two different change management platforms. From the information processing viewpoint, the system enabling these operations can be compared to an “assembly line for information” built around the event model (see Part I). The business operations for an enterprise can be identified by following a similar approach to the Value Stream Mapping methodology from Toyota’s lean manufacturing best practices. As a consequence, the lifecycle/event model becomes the main driver for the system design and architecture, eliminating the unreliable collection of use cases as the primary input for the system architecture.

Technical operations has its own challenges and distinct dynamics. The main goal of technical operations is to maximize how well various teams collaborate to introduce changes. The Basic Dynamic Business Application is an adaptive system with two type of processes: operations and operational changes. These two processes link business users with technical support (operations) and development (operational changes).

Fig 1. Frameworks for both Business Process and Business System change management are required build and maintain the Basic Business Dynamic Platform.

Both frameworks have to be merged into a unique system design that takes into consideration both the technical team operations and the business users activities. This system is built around a server-side architecture. This is the only approach that provides support for collaboration between different types of users while emulating their structure and control hierarchy.

Designing server-side infrastructure is orders of magnitude higher in complexity than for desktop applications

Every designer of enterprise software recognizes that implementing complete client-server applications for businesses is more difficult than building desktop applications. In the desktop case, the interface is built around one role, one main task, and a clear interface, with the entire application running on a single machine. The interface can be built around a few small steps that can be implemented as functions. In the server-side case, the implementation is considerably more complicated. One source of complication in the server-side case is the expectation that more than one role must be implemented including a manager. The manager is unlikely to play a passive role and will need an interface that will allow him to change operational parameters if needed. Additionally technical support must monitor the application for normal usage

To better understand the difference between implementing a desktop application and a client-server application, we can look at what would be required for a desktop app to become a client-server app. Let's imagine that we need to use MS Word implemented as a server-side app to write a letter. First, you would need a technical support rep to setup the fonts, the page, and all other settings. Whenever, you need a different setting, you have to call the technical support to make the change. While you are writing, a manager has to approve the fonts, the letter head, etc. Once it is completed, the same manager may need to approve the letter before it is sent. While this example looks simplistic, it helps to understand the challenges faced by a software architect when trying to extrapolate expertise gained in developing desktop applications to a client-server application

In fact, when multiple users are working with the same application, business or technical, it requires an entirely different approach. The architecture has to provide support for tasks or processes that are distributed (users could be in different locations and using different machines), stateful (a state is not anymore defined by the next Window that is hardcoded in the application), dynamic (either a manager or an external user may require an instance of a business entity to be changed on the fly, similar to the way a manager may request that a product or service can no longer be sold for the same price or a customer may request changes to an order), and hierarchical (all organizations have a control hierarchy, so various users have different control over processes). All these new attributes must be embedded in the foundation of an application or it will have to be rewritten every time a change is made.

But this is not the only difference when designing server-side applications. When a desktop application crashes, the user simply restarts it. Because it implements only simple tasks, it may always restart the work, even if it lost most of it. A user doesn't need a separate technical team to assist him in running the application. For client-server applications, the situation is entirely different. Because multiple users are working at the same time, when something goes wrong only a technical support team that has knowledge of the technology and has access to tools to monitor the application can make the right decision on how to restore the application to normal operation. And the technical team has their own operations blueprint that is quite different from the business operations blueprint. The technical team has their own controlled hierarchy of processes, their own distributed environment, their own way to implement changes, and their own states.

As a consequence, architecting a client-server application capable to run at the enterprise level is orders of magnitude more complex than a desktop application. The main complexity of the server-side application derives from the fact the it must support two adaptive systems, one for the technical team and one for the business team, each with their own operations and controlled hierarchy. There are laws applying to adaptive systems and their associated information that need to be followed when architecting applications supporting both the management and technical support adaptive systems.

In architecting these type of complex systems, an ideal architecture doesn't have to be redone every time a change in business or technology is introduced. The ideal situation is to have a set of standard components with a well-defined behavior implemented as event models. These components are linked in a controlled hierarchy by event models that are implemented at different levels of control. While on the desktop, there is a well defined set of components with their own event model that are linked together by a clear controlled hierarchy. The server-side is far less developed. J2EE, .NET, or any other existing proprietary architecture in use are missing basic components that provide support for an environment that is: distributed, dynamic, stateful, and hierarchical. Because Web standards are stateless, flat, static, and one-server only, they are not of much help when applied to one adaptive system let alone two.

From the laws of adaptive systems and their associated information, a controlled hierarchy always conforms to three patterns:

• the initialization process always has the same top-down sequence
• information travel between layers always follows certain rules: commands flow from the top down, and feedback flows from the bottom up
• each layer has its own normal operations that doesn't change until a feedback or a command is received. Processing commands or feedback always requires a change management capability. And because there are two types of interactions--commands and feedback--then there are two different types of change management

We will show how to implement a standard architecture for server-based applications using standard components, linked together in a controlled hierarchy that is dynamic, stateful, and distributed. But first, we present another event-centric platform that is very familiar to any computer user.

Microsoft built its fortune on a few components and a static event model for GUI

There are many reasons why Microsoft has been so successful, but one of them stands out. They built the Windows operating system into a desktop standard despite all its critics and competition from Apple or IBM. One explanation could be that Microsoft was the first to provide not only an operating environment like Apple did with Mac, but that they also had the easiest OS and GUI components implemented as a set of APIs with which to work. With this component model, software companies including Microsoft, were able to build databases, office applications, and development tools like Visual Basic. These applications were able to take advantage of the Windows embedded hierarchical event model built into structure and controls.

The advantage Microsoft had while developing Windows, was that most of the main elements of an OS and a GUI, together with their event model, were either already in place, or could be built easily from intuition. For any desktop application, the controlled hierarchy looks deceptively simple. At the top of the GUI stack there are windows that act like containers for the remaining GUI components. Open the window and the embedded controls are automatically initialized too. Close the window and every components is closed as well. Capture those events in APIs and the development of an Windows applications becomes a plug-in architecture.

Fig 2. Microsoft Windows controlled hierarchy model - One user, one application, one location, one static set of functions

The implementation of those APIs is the Windows operating system itself. It is the layer that controls the initialization process and the one that ultimately has to handle errors without crashing and losing data. On the server-side, the components, their events, and the controlled hierarchy is complicated by the need to provide native support for tasks that are stateful, dynamic, distributed, and hierarchical. This is the main goal for the Adaptive Enterprise Operating Platform (AEOP).

The proposed AEOP increases productivity by being designed around a dynamic structure and control

A few years ago IBM enlisted thousands of enterprise architects in an effort to reduce their number of solutions customized for various vertical industries. After more than a year, they were able to reduce their number from over sixty applications to less than twenty. The magnitude of the effort and the less than stellar results shows the magnitude of the problem. Despite their enormous resources, IBM had real difficulty creating a generic solution that would apply to all their customers, regardless of their vertical or their size. One thing that may have complicated their effort was that they started with a faulty set of legacy solutions.

Our goal was the same: to create a generic solution that is almost universally applicable to all industries. Having the freedom to start with fresh thinking, we focused on the fundamentals that we considered most important first. While most enterprise architecture relies on existing technology, components, or trends like the SOA, the AEOP architecture goes beyond that. It starts with an understanding of the fundamental role technology plays in a business, which is to increase productivity. This role is valid for all technical solutions, IT or not. In the IT case, because of the dynamics of the business and the complex relationship that exists between users of an enterprise system, the architecture needs to be built around the structure, control, and dynamics of enterprise processes.

This approach also provides a means to address the discussion generated by Nicholas Carr's " IT doesn't matter" article published by the Harvard Business Review in May 2003. Asking if IT is relevant for a business or not is like asking if the assembly line is relevant for a business or not. While it is essential for companies like GM or Toyota, it is part of almost any manufacturing company, including the stars of the high tech industries like Intel. The impact of the assembly line can be far less in industries where custom products or services are widespread, like in the food or healthcare fields. Despite its importance for companies like GM, the assembly line is nothing than an enabler of higher productivity. The same is true with IT, it may matter for a company that depends on information processing for their operations, but could be far less relevant for others. The only difference with IT is that it is far less mature a technology than the assembly line.

Analyzing how IT can increase productivity requires looking at the basic business mechanisms that drive how technology is delivered and used after it is developed. We must look at the fundamental roles associated with information processing and ask what are the core informational elements that drive the enterprise operations. In the engineering field, this process is quite well understood and can be seen in several examples. When an engineer starts to design a plane, he never looks at the plane as a collection of parts like nuts, bolts, wires, connectors, pumps, electrical motors, levers, panels, etc. He looks at what connects them and what role they play in delivering functionality. The structure of a plane is built around the wings, the engine, the cockpit, and the main body, and follows a controlled hierarchy that maximize the interaction between various participants. The same is true with an architect designing a house. The type of air conditioning, the appliances, or the wiring to be installed are not important until the architect knows the number of bedrooms needed and the number of families that will live in the house.

In today's approach, enterprise software is mostly architected / designed by focusing on technology components and standards. Requirements are an afterthought evaluating increased productivity or enterprise software structure, control, and dynamics. In fact, high level architecture recommendations revolve around only presentation, business, and persistence layers. No current architecture approaches provide a solution to support the reality of dynamic, distributed, stateful, and hierarchical business process architecture.

To use an even better analogy, look at how engineers design assembly lines. First of all, they are designed to be easy to maintain and repair. This is why assembly line architecture and design are built around interchangeable, modular, and standardized components. Because the assembly line doesn't change dynamically, then the second criteria is to be easy to be reconfigured for a new model. This is why robots are so popular on the assembly line. They can be easily reprogrammed for a wide variety of current and future tasks regardless of how complex they are. Only after these two criteria are met, the design engineer assess and tries to optimize each particular task.

To architect/design a generic AEOP, we identified three fundamental factors that need to be accounted for in the design and which have a big impact on improving productivity. They are almost the same for all businesses, regardless of their field or size:

1) Increase productivity for technical team operations by providing support for collaborative events between technical support, developers, and business users

The biggest impact on the overall architecture is the relationship between the different technical teams and the business users and how changes to requirements are introduced. Because the IT team is the one that spends the most time with an application, increasing its productivity should be a top criteria for the architecture.

In this context, the structure and the control for normal operations of a technical team is simple. A business user opens an application which triggers business events. These events make the application process information. The technical support team monitors the application. While the application is in production, the development team usually plans for the next upgrade and to implement new requirements changes.

The control of this entire process follows the same hierarchy: at the top of the control pyramid is the team of developers. They have all the control over the implementation as they are the ones hard-coding the future functionality. The technical support team is the next in line, as they can stop and restart the servers or change some of the system configuration. The team that has the least control over an application includes the business users, managers or workers. They can only follow the application script.

Because business is dynamic, requests to change system functionality may come very early, even before the application is installed and operational. In fact, requirements are valid only until the next meeting happens. As a result, how changes are translated into code is an important factor when we look at what drives productivity. In the AEOP, the same groups of users are found when changes are introduced. The development team is the one that receives the request for changes, the team that monitors the applications installs it and gets trained to support it, and then the business users gets trained to make use of the new functionality.

As a result of this structure and controlled hierarchy, the top level for the enterprise application is always built around three platforms: one that implements functionality for the development team, one that implements it for technical support team, and one that supports the business users. Each one has its own main lifecycle and event model that drives the design.

Because the AEOP is built to support primarily the dynamic of technical team operations, the three-platform structure, their components, and the event model are built around the lifecycle of a request for change.

2) Increase productivity for business operations by providing support for collaborative events between the three fundamental types of business users

Collaboration between the fundamental types of business users has the second biggest impact on productivity. In any enterprise process there are at most three fundamental user types, representing the three areas of an adaptive system: workers representing operations, managers, representing decision-making process, and customers, representing the economic environment that closes the value cycle. Other business participants to the value cycle, like suppliers and government agencies, all play a secondary role and their processes can be identified with one of the three basic processes -- operations, management, and environment.

Business is dynamic. One of the biggest productivity levers in a business is the efficiency of the process by which changes are introduced. There are two types of changes in a business: internal changes introduced by managers and external changes introduced by customers. These changes are at some point introduced as updates into enterprise applications. An efficient architecture/design must account for this dynamic. An ideal architecture for enterprise applications is able to adjust to most changes with minimal code and configuration change. Because businesses are increasingly dependent on technology for their operations, a quick and efficient way to update applications is important to overall productivity.

Based on this, the AEOP defines only three types of enterprise architectures for high level collaboration in business operations:

• STATIC ARCHITECTURE: is purely operational -- all current enterprise software fits in this category. Changes to business processes beyond the most trivial configuration changes are done by making a request to the IT department that usually leads to at least stopping the application. Static architecture behavior is similar to the assembly line that is stopped at a car manufacturer to be prepared when a new model is introduced.
• INTERNAL-ONLY (INTERNAL decisions) DRIVEN DYNAMIC ARCHITECTURE: provides support for dynamic operations and implements the relationship between a manager and a worker. In this case, manager decisions are automatically accounted for during normal operations and changes are applied on the fly to all current instances. Because the manager and the worker are operating on different timelines, they need a change management platform that handles internal decisions to bridge their activities. Processes that fit in this category are those in which customer doesn't play a direct role.
• EXTENDED (INTERNAL and EXTERNAL decisions) DYNAMIC ARCHITECTURE: provides support for dynamic operations and automates the collaboration between all three fundamental types of users, workers, managers, and customers. This is the most complex architecture for business operations. It has two different change management platforms, one for the way internal decisions are applied to normal operations, and one for customers that want to change how their orders are executed during normal operations.

Note that in any enterprise, a business process, regardless of its type, is always "connected" to the management "command and control" structure, and to the customer feedback. As a result, each application fits into the FULL DYNAMIC category, regardless of its targeted business process.

The dynamics of the operational platform architecture revolve around business change lifecycles. They are driven by internal decisions (managers makes internal decisions) and external changes (customers could decide to apply changes to existing orders and services in progress).

3) Increase productivity for the development team by providing an architecture that is entirely decoupled at the lifecycle/event level

The previous productivity factor was related to the impact business changes have on architecture / design. Ideally business changes can be accommodated in the architecture / design by interfaces to change management modules. The interfaces give managers and customers access to automatically apply changes to all operations in progress.

Sometimes changes require new functionality to be implemented by the development team . The application architecture / design should support high productivity when new functionality is implemented.

The current approach is to have a few components grouped together under a label called "business layer" that have access to a database. There is a fundamental mistake in this design. Databases used in this design are very likely to be relational databases with very poor capability to store historical or stateful information. They mostly rely on caching mechanisms, huge consumers of computing resources that continuously update massive amounts of data in the background.

All business processes in the enterprise are linked in one way or another to the main business entity, which is a product or a service. For a business process linked to the main business entity, the most important aspect is the existing history. This is true even for the simplest transactions. Going to a store to buy something assumes that the previous history of your financial resources will give you enough financial "resources" to buy the product while the store has the product "prepared" to be sold. The history of an entity is composed of events that eventually describe its entire lifecycle. Because the events that may change the state of the entity could be triggered everywhere within the business structure and at multiple control levels, the AEOP architecture built around the dynamic of this business entity has to be stateful, hierarchical, and distributed.

Because the lifecycle is built around events, the entire AEOP can be built around them. Any event implementation can be done by following the same generic approach: the entire lifecycle can be considered an "assembly line" for information that mirrors the main business entity transformation. As a result, the event can be implemented with the same structure, assembling informational elements 1) retrieved from various locations (distributed) that need to 2) exist at a certain moment in time (for instance retrieving the account information for a credit card transaction has to be done in real time) and 3) can be implemented by a certain business process and 4) has applied certain business rules.

Another AEOP advantage is the ability to design both types of change management around the same event model. In the end, the entire implementation of the operational platform can be decoupled at the event level. The operational platform can be built as a plug-in API infrastructure, similar to the way Windows API is built around the desktop integrated event model.

It is important to note that the EDA (Event Driven Architecture) model found in the literature is different than the AEOP event model. The EDA is built around a stream of unstructured events, while the AEOP is built around a stream of structured events that are linked together by a clear set of lifecycle templates.

Fig 3. The AEOP controlled hierarchy model - Many users, many client applications, many distributed locations, many static functions grouped dynamically by change type

The AEOP hierarchy of components and their event model contains 5 levels:

• OS Level—this is where all the OS components are found, like the kernel, file system, networking, and I/O. OS may have also GUI components which can be used to run graphical tools to monitor various applications and their activities.
• Technology Level—this is where all traditional server-based components, like application servers, BPM engines, Business Rules Engine, Databases can be found. Also here are components that implement standards for Web and Web Services, Messaging, etc. They have the same role, more or less, standardized parts play in plane design. Nuts, bolts, electric motors, hydraulic pumps, electrical wires, connectors, chairs, electrical panels, actuators are generic to many other industrial equipments and are not specific to a plane. Nevertheless, they play a key role in simplifying and standardizing as much as possible the design.
• AEOP Technical Operations Level—this is the top level that has components specific to the AEOP. There are three components—install/start/persist/restore platform, system platform, and operational platform. They represent the three groups collaborating to run a client-server application: business users team, technical support team, and developers team. Each component implements specific events like the ability of the technical support team to monitor for errors and restart the application. For applications required to work with external systems, there is the fourth type of component called External System Supervisor Platform. This component manages the initialization, normal operations, and error translation for interactions with external systems.
• AEOP Business Operations Level—this is the top level for business implementation and the next level down for the operational platform. It also has three main components: normal operations, change management for internal decisions, and change management for the external environment interactions (i.e. customer decision). There are many additional components including operational translators for external systems, system map, licensing manager, etc.
• AEOP Event Processing Level—this is the plug-in level for implementation. The entire normal operations for the enabled business process is implemented at this level. Because every functionality is broken down to functions at the event level, the entire architecture is decoupled around the event model. This can include operational errors, operational changes, and even security access.

Obviously, there are many components and events that are not covered in this short description of the AEOP. In the next section we will take three components and analyze them to the next level of detail.

There is a difference between the AEOP event model, and Event-Driven Architecture (EDA). EDA is a software architecture pattern promoting the production, detection, consumption of, and reaction to events (from Wikipedia). What EDA is missing is how to capture the structure and control of business processes. Also, the AEOP distinguishes between the three types of events, one for normal operations, one that drives internal changes, and one for external changes. They land on different modules to be processed versus EDA which uses the same approach to process all of them. The AEOP uses the lifecycle to group events in a structure that is distributed, hierarchical, stateful, and dynamic.

The “Assembly Line” for Information in the AEOP is Built Around The Multi-shell Container for Dynamic Lifecycle Pattern

All modern software platforms are built around some form of container pattern. The AEOP is no different. Application servers based on J2EE even have two types of containers, the Servlet and the EJB. Operating System can be considered a form of container for applications. The concept of containers has been associated with the idea that computer resources are limited. Containers are used differently in the AEOP architecture:

• Objects created and managed by the container are a direct representation of business entity resources. For instance, an AEOP operational container manages an entire order lifecycle and not only objects that reflect existing computer resources.
• A container is able to differentiate between calls that will change the state of a managed resource as part of a normal sequence of events versus calls that will trigger a change outside of normal operations. Because there are two types of changes, there will be two types of change management module associated with each container. The AEOP container supports dynamic lifecycles for managed resources to support the reality of enterprise dynamics better than existing static architectures.
• A container is not a standalone structure like in J2EE application servers. Containers are linked according to the controlled hierarchy structure. The integrated event model drives how events generated outside the system are received and processed. In this case, a container that is neither at the top of the hierarchy or at the bottom plays a dual role, one as a resource lifecycle manager, and one as a managed resource. This is what gives the multi-shell structure to the system containers.

The entire AEOP architecture and implementation is built around the pattern that represents a container that manages resources representing real business entities, that makes a distinction between normal operations and changes, and is part of a structure that reflects the business controlled hierarchy.

Fig 4. The AEOP Multi-Shell Container Architecture places Technology Operations at the top, Business Operations in the middle, and Event Processing at the bottom

Each AEOP container may process three types of events: (1) normal operations events identified by the “Event Model” module; (2) events from internal changes sent by a container higher in the controlled hierarchy; or (3) events from external changes sent by a container lower in the controlled hierarchy.

Other than the main lifecycle and its associated event model, there are business entities that can be considered “static” for normal operations. For instance, prices for items or a physical location are not expected to change during normal operations. They are captured by a module called the “Static Model.” That doesn’t mean that they are fixed and isolated from business dynamics. The Static Model is updated with internal and external changes only through the change management process.

The AEOP controlled hierarchy can be used to set the container mode: initialized, operational, and shutdown. Events can be processed only if the container is in operational mode.

In the AEOP architecture, there are three relavent elements: 1) the high-level technical structure of components and control, 2) the operational platform, and 3) the event processing platform.

The high level AEOP technology represents the entire application, which can be identified with an adaptive system. It has three main platforms and each one of them can be identified with an adaptive subsystem as well, with the platform above playing the management role, and the one below the environment. Each platform has its own well defined event model that implements its own functionality and also the interaction with the other two. On each platform we find a repository, which can be a cache database or a more traditional relational database.

Integration with external systems follows the same event model. There are two integration strategies that can be used based on the types of external system. If the AEOP has access only to a persistent repository, it can use an ongoing scheduled task to look for data changes into the database records. The second approach relies on API calls and can be implemented when the external system implementation can be changed to make calls when the state of main business entity changes.

Fig 5. The high-level AEOP multi-shell container architecture

Each of the three platforms is also built around specific lifecycle elements. The Operational platform is built around the main business entities lifecycle, the System Runtime platform is built around user sessions, and the Install/Persistent platform is built around the version control lifecycle.

While data must be processed only according to the strict event model blueprints, the same data can be viewed by anyone with access to the system and having the right credentials. There are, however, few data viewer utilities targeted to platform users. For instance, the System Runtime platform has a viewer that monitors the users and the distributed systems for runtime errors. The same tool can be used to restart various subsystems when they fail.

The AEOP technology level also implements a complete initialization process. When started, an application must go through three steps before a user can run the first task:

• Start Application – a system admin starts the main application. This is done by starting a container like Tomcat (the J2EE servlet container) together with Axis (the Web service platform). The application could load the Spring framework into a lightweight container and use Spring bean configuration file to load the initial “Start” module. The “Start” module initializes the “System” module. This module is responsible with the hardcoded configuration for the main application and it may include hardwired links to existing remote data sources, Web service endpoints for reading deployed remote applications versions, ways to test and open connectivity to various distributed systems, functions to provide data related to normal operations, performance, and logging. The main users for data generated are developers and technical support.
• Install/Start/Persistent/Restore “System” module – this module implements the all the system-related functionality, including checking on system configuration. Its main function is to load into the cache a system map with all the resources (remote data sources, users, versions, servers in the cluster, connections, security data, etc) and their status. This is the module that provides the application monitor. The main users for this module are the technical support. The steps are:
  • At this stage the first step is to verify which distributed systems are up and running. If a remote system is not active, an operational change is scheduled to be sent to the operational system that will invalidate those tasks dependent on them. It will also flag their cached data as “stale.” It will send a signal to app dashboard and it will log it.
  • Once that is done, it verifies that the software version for those remote data locations is correct. If not it will disable them, and it will send a signal to the app dashboard, together with a logging action.
  • Last step is to check the data definition for those remote distributed systems. If they have changed, the same steps as the previous activity will take place
• Start the main application operationally – this is the last step before a user can access the application. It loads into the cache the operational context, together with all the active user instances and their status. Part of the operational context are pointers to distributed systems data and their status, together with the distributed systems operational context. The main users for this module are the business users.

Fig 6. Status of the three main containers are determined by the system’s controlled hierarchy

After initialization, the same framework is used to control the operational readiness of the main application. There are three scenarios for this:

• An error is generated when a user runs/creates a query – this is the most common scenario. When an error is detected by the operational model sub-platform, the system is set in the “operation error” mode, a message is sent to the system model sub-platform, and the new status is displayed/alerts are sent to the tech support team that monitors the main application. The same error will be sent to the install/persist/recover model sub-platform for logging (this information is for the development team). The user will receive also a message that displays the status of the current event, together with available options. One option could be to recover the instance from the previous step. This is possible because the entire operations is built around the event model that guarantees that the previous event was completed successfully.
• A manager or a data source admin makes changes to the operational context – in this case the main application is put into a special state of “operational change” mode. All current users running tasks that are affected by the changes are notified. They have two options: either run through a process to update the instances or simply ignore it.
• An environment system error is generated – in this case the tech support team that monitors the IT environment is notified. This is done by having tools that monitors the cluster, or the J2EE/NET application server. In this case the main application will try to log the latest activities or the debugging application server stack.

The next to detail is the AEOP operational platform. The event model is built around the types of business entities. Each service of product type has its own lifecycle represented by its own event blueprint. An event blueprint is a well defined set of ordered events. A set of types of changes is associated to each blueprint. Internal or external events can be part of the normal operations or can represent different types of changes. Each event has to be clearly labeled because it needs to be processed by a certain module. This approach is different than the Event-Driven Architecture approach that treats events as a “notable thing that happens inside or outside of your business.” Also, EDA is focused on an events structure, like it needs to have a header, a body, a timestamp, etc. The AEOP doesn’t have any of these restrictions because the entire structure is determined by its type.

Fig. 7. The AEOP’s “Assembly Line for Information” is built around the Container for Business Operations

The operational platform automates the interaction between the three fundamental types of users: workers (they support normal operations), managers (they trigger internal operational changes), and external business parties (i.e. customers).

The last platform to describe in detail is the AEOP event processing. One of the main reasons why the “assembly line” concept proved so popular for over a century was because each step could be easily decoupled from the rest. Not only this, but at each “station” a worker would follow the same steps, regardless of the work he needs to perform. When the main instance of a product arrives to the “station,” it is like an empty “shell” that needs to be “filled” with various parts. Those parts are of two types: specific parts to be assembled at the current “station” and parts that only contribute to the assembly process, parts that we call “kits.” For instance, a “kit” can be some glue that will help the assembly process. The three elements to be assembled have their own warehouse, they have their own “arrival” timing, and they follow a specific logic for the assembly process. Once assembled, nothing happens to that product instance until the next “assembly station.”

The “information assembly line” is very similar to the real assembly line concept. We encounter the same types of information during an event processing: a “shell” that represents an instance of the main business entity, the information specific to an event and instance that needs to be added, and information that is only specific to an event that needs to be added called “kit.” The main characteristic for a “kit” type of information is that it is finite, it can be used multiple times for the same event, and you may “run out” of it in the middle of an event processing, similar in a way to the “glue” used to hold parts during the assembly process. An example of “kit” information in order processing is a credit line that a business gives to a customer. He may decide to use the entire available credit to pay for the product or he may decide to use only a portion of it, he may use it to pay for multiple purchases, and he may “run out” of it in the middle of a purchase.

Another similarity is with the “warehouse concept.” Information to be “assembled” has certain rules when it comes to the way it is retrieved. For instance, when a customer pays for a purchase with a credit card, the account information is retrieved in real time from the bank. A value stored in any other data “warehouse” is not a valid one.

During the processing of an event, getting the information from various systems can be accomplished by the event “location” layer, and by the event “scheduler” layer. Only after these two steps are completed can the “information” be “assembled” by following the logic. The “location” layer uses a message-centric component, the “schedule” layer uses a schedule-centric component, and the “logic” layer uses a BPM/Business Rules engine as the main component. The order is always the same.

The scheduler role in the architecture/design goes beyond timing different tasks. The other role is to help design a multi-threaded architecture in which the control over a state change is done at the event level. This is accomplished by using the scheduler as an event module that translates processes that may fork during a state change into separate “swimlanes.” This way we can implement the state change logic at the event level in the same way a scheduler handles each event sub-task. This way, a failure in a process “swimlane” will leave the entire system in a state that can be clearly identified.

Fig. 8. The AEOP Event Processing Container – Location, Schedule, and Logic

By combining all these three elements, the “assembly line of information” can be built to be generic for any enterprise system, regardless of the field or size. This is similar to the way assembly line, that started in the car manufacturing, was extended to all other manufacturing industries.

The AEOP Event Processing Container has two change management modules for internal and external changes. When an event is identified as a change, a decision table links the correct type of change to the specific event. The architecture is designed as a set of plug-ins that are called based on the state of the existing instance. For instance, in an order process, there will be a different way to process a price change for pre-paid and unpaid orders.

Conclusion – Architecture in days, design in weeks, implementation in months—building the integrated server-side for DBA infrastructure the easy way

The AEOP approach has many advantages. The primary advantage is that it standardizes the architecture for server-side applications, something that is missing from current IT. It transforms the entire server-side implementation into event plug-in code writing, similar to the way Microsoft Windows applications are written.

Fig 9. The Adaptive Operational Platform Supports a business that is dynamic, has a controlled hierarchy, is distributed, and is stateful

The three steps to implement an application using the AEOP include:

1. Get the object flows and change types
2. Get the components for the three platforms
3. Build the five models for each event processing

Technologies and architecture concepts like SOA play a minor role in this architecture. Components like BPM engines, schedulers, messaging have a clear role in this architecture, however they have very little impact on the design. This is similar in a way to the role various parts play in the design of a plane. Electric motors, wires, nuts and bolts are essential to the overall plane functionality, but they are not essential to the design process.

Because the approach is not dependent on the business, tools similar to the Microsoft Visual C++ wizard or Visual Basic can be built that will automate further the implementation process. This not only increases productivity of the development team, but it also helps developers to focus on the real aspect of implementation instead of “fighting” with the wrong architecture.

Coming soon Part III -- Case study and sideline
The article will continue in Part III with a case study of a real implementation and a short summary of the new theory of adaptive systems.