PushToTest, makers of the TestMaker open source test tool, announced the results of a research study that evaluates and compares the total cost of ownership of SOA development and deployment solutions from IBM, Oracle and TIBCO in a paper titled "The Composition Approach for Large-Scale SOA", which is part of the revised soa knowledge kit. TIBCO was declared the leading solution on multiple aspects including total cost of ownership. InfoQ spoke with Frank Cohen, CEO and Founder of PushToTest, on the underlying mechanics of the study.
InfoQ: What is the motivation to perform a TCO study on SOA stacks and offer support on a kit of collaterals for the community?
As a community leader in the IT industry I keep looking for an expert source that provides software architects and developers with a way to understand SOA development platforms for interoperability, developer productivity and performance. PushToTest is working for medium and large organizations to define a standard SOA application blueprint to surface interoperability, performance and deliver problems. Adoption of these practices will help PushToTest profitably deliver the knowledge, test tools like TestMaker, and support services to organizations. The SOA Knowledge Kit is the defacto standard at Best Buy, PepsiCo, Deloitte, and 30 other companies.
By urging the standards bodies (OMG, OASIS, W3C, IETF) to adopt these practices I hope to help the IT industry to deliver a more reliable world. PushToTest's work on the Kit appears in the OASIS SOA Blueprints project. http://tinyurl.com/85tsbal. We maintain that work and others.
We initially published the SOA Knowledge Kit in 2008. I chose the Oracle, IBM, and TIBCO stacks since they provide SOA development and deployment platforms. In 2009 we added JBoss to the Kit. We hope to add other SOA platforms, including Mule, in the near future. The current work updates the Kit to the latest versions: IBM WebSphere Integration Developer V7.0, TIBCO ActiveMatrix SOA Product Suite 3.13, and Oracle SOA Suite 11gR1 (11.1.1.5.0).
InfoQ: What was the team composition on this project? Can you tell us the average engineer's prior experience with each of the stacks and with web services development and orchestration in general?
PushToTest used 2 engineering teams to implement the SOA Knowledge Kit. Each team has 1 architect with 5-8 years of experience building SOA, Rich Internet Applications (RIA, using Ajax, Flex, Flash,) and Web applications, 2 engineers with 5 years of Java coding experience, and 1 project manager. The engineers have experience writing EJB's, SOAP and REST-based service interfaces, mediation, orchestration, and workflow. Team A implemented the Oracle and TIBCO projects. Team B implemented the IBM project. It took approximately 3-4 weeks to implement the use case for each stack.
InfoQ: Were the same service patterns used to design and implement services on all the stacks? Can you describe some challenges that were faced with implementing certain service patterns on the individual stacks?
PushToTest defined a use case for a typical manufacturing organization. We implemented the use case on TIBCO, Oracle, and IBM stacks. We then made changes to the implementation: added HTTPS/SSL security, changed the message schema, and changed to an asynchronous message delivery transport. We ran a functional test and a performance and scalability test of the finished work. We document the use case and the developer's experiences and publish everything under a free open source software GPL v2 license.
The manufacturing scenario use case implements a 3-step business flow.
1) The Allocate Purchase Order (PO) process opens a new PO. The implementation provides a method through a SOAP interface accessed over HTTP protocols.
2) Reserve Parts using a service for warehouse just-in-time inventory control to reserve portions of the inventory to meet the delivery needs defined in the purchase order. This is a Spring and Data Access Object (DAO) service that receives a REST-encoded request over a JMS service interface. The message conforms to the inventory control service's XML message schema. We used the OAGIS organization's Business Object Document (BOD) schema. A Web page with Ajax elements provides a human interface to request the service.
3) Price Purchase Order - uses a service to assign a price point to the purchase order based on the current price catalog. The system prices products by simulating business functions on an SAP installation and accessed through a simulation of SAP NetWeaver's SOAP Web Service interface and security credential system. We built mock SOAP services using soapUI.
The engineers kept a Developer Journal of their experience on each stack. Each Journal is 60-80 pages long. The Journal contains our software engineer's instructions, comments, and opinions to accomplish the use case implementation step-by-step. The Developer Journals describe significant challenges on each stack and workarounds/solutions.
At a high level we found the following challenges implementing certain service patterns on the individual stacks:
Oracle
Oracle's stack provides quick solution creation for engineers. Everything in the Oracle stack requires a code dive and manual adjustments to deployment descriptors, classpaths, and general configuration. Oracle's tools are for developers - not for architects and business analysts. There is little or no model driven approach to designing, building, and deploying SOA applications in Oracle.
For engineers identifying the tools for the task using only the Oracle web site was easy. Oracle's Quick Start Guide is straight forward and useful.
Nothing worked out-of-the-box. Everything required community support (especially of the BEA components.) Most solutions came from workarounds found by third parties. For example, we found many tutorials on Web Service creation on the Oracle site. The one we chose failed. http://st-curriculum.oracle.com/obe/jdev/obe11jdev/ps1/webservices/ws.html#t5. We found soapUI web service mocks can not be called from Oracle. JDeveloper throws a WebServiceException: Error creating model from WSDL. We document this in the Developer Journal.
Other times we found multiple tutorials, some worked and others did not. For example, we found a tutorial on building asynchronous services at http://download.oracle.com/docs/cd/E17904_01/web.1111/e15184/asynch.htm#CBHECBFG. We used annotations that are not available on the app server: @AsyncWebService and @PortableWebService tags.
IBM
The IBM stack is several big platforms integrated together: WebSphere Application Server, WebSphere Integration Developer (WID,) Rational Application Developer (RAD,) and WebSphere Process Server (WPS.) IBM RAD is a 6 G download alone. The platforms have version issues: WID 7.0 requires WAS 7.0 and RAD 8.0.3 requires WAS 8.0.3. You wind-up having to install multiple versions of WAS to make their SOA stack work. The parts are enormous and often don't play well together. For example, WebSphere Integration Developer (WID) gave us broken WSDL document output.
We encountered many situations where we could not get 'there' from 'here.' For example, we could not create a client for a service we built in WPS. WID refactoring could not handle simple changes to namespace, name, and schema values. WID created corrupt WSDLS. RAD does not have the capability to develop business processes. And, integration and process development happen in different tools: WID and WPS.
TIBCO
The TIBCO SOA stack consists of modeling tools, component development tools, and service grid deployment: ActiveMatrix Service Grid 3.13, BusinessWorks 5.9.2, Enterprise Message Service EMS 6.0.1, Rendezvous RV 8.1, and BusinessWorks Studio (BS.)
The tools are model oriented. They keep a better control of code and abstract away the underlying technologies. There is no need to look at code, just models with SOA Concepts. This is great news for developers because they now have common tools to work with business managers and software architects to model the service interfaces and workflow.
Code oriented developers without SOA knowledge will have a steep learning curve. The documentation is abundant and quite good. Finding the right guide is hard. The surface-level documentation on the TIBCO Web site often describes the functionality rather than showing how to do things. The modeling language is a proprietary one, reducing intuitiveness. BW Studio is an Eclipse Based IDE that will be familiar to many developers. Some concepts will be new, such as Aysnchronous Services using JMS where some code oriented developers would expect to use Message Driven Beans (MDB.) And, mediation flow was required to implement asynchronous services.
InfoQ: Can you shed some light on the functional and performance testing methodology for the SOA app?
We applied Agile software development practices when building the Kit. We paired our developers with testers and created unit tests as we built the services. Some unit tests were Java JUnit tests that made class/method calls to object interfaces. Other tests were soapUI TestSuites that made SOAP and REST calls to services. We used PushToTest TestMaker to repurpose the tests as functional tests and load and performance tests. TestMaker deploys the tests to a grid of test servers in a QA lab and also to a cloud computing environment (Amazon EC2, GoGrid, Collabnet, and Rackspace.) TestMaker produces a set of reports showing the root cause to functional issues and performance bottlenecks.
The SOA Knowledge Kit comes with the tests we implemented and a copy of PushToTest TestMaker. Rather than publish the performance results based on our available hardware, we deliver the ability for anyone to run the tests in their own environment to prove out the performance differences between each of the SOA stacks in their own data centers.
InfoQ: Can you share some details on the TCO model that was computed from development effort?
did a time/motion analysis of each step of the services lifecycle to reveal the amount of time and effort required to build, integrate, deploy and manage a range of services needed to assemble a composite application. The goal of the study is to compare the cost savings that can be achieved through greater developer productivity resulting in significantly reduced TCO. The TCO model is a Open Office/MS Excel spreadsheet that assigns costs to each step of the Kits development. The model comes with the Kit.
In side-by-side product implementations, TIBCO ActiveMatrix and BusinessWorks provided the greatest productivity savings. TIBCO took 29 percent less time and development costs compared to Oracle. TIBCO took 22 percent less compared to IBM.
InfoQ: Did you utilize SOA infrastructure capability such as repositories, ESBs and other intermediaries from individual vendors as prescribed? Can you describe some of the high level details?
We followed the published best practices from each of the stack vendors, including use of their repositories, ESBs, and mediation services. For IBM that meant using: WebSphere Application Server v8.0.3, IBM Rational Application Developer V8.0.3 Multiplatform Multilingual, IBM Websphere Integration Developer V7.0, IBM Websphere Application Server V7.0 Multiplatform Multilingual, and IBM Websphere test environment. For Oracle that meant using: Oracle SOA Suite 11gR1 (11.1.1.5.0), Oracle Database XE 10g 10.2.0.1, Oracle WebLogic Server 10.3.5, Coherence, OEPE, Repository Creation Utility 11.1.1.5.0, SOA Suite 11.1.1.5.0 (2 parts), JDeveloper 11.1.1.5, and Oracle Service Bus 11.1.1.5.0. And for TIBCO that meant: ActiveMatrix Service Grid 3.13, BusinessWorks 5.9.2, Enterprise Message Service EMS 6.0.1, Rendezvous RV 8.1, TIBCO Runtime Agent 5.7.1, ActiveMatrix Sample Examples BWSE 5.9.2, Business Studio (BS) VS, and Designer Design Time DABS 1.3.1.
InfoQ: What is the support plan for future versions of SOA stack solutions from the three vendors: IBM, TIBCO and Oracle?
We plan to refresh the Kit for the Oracle, TIBCO and IBM stacks in 2012. That will give them some more time to make major new releases. We are looking for the developer community feedback to identify the additional stacks to add to the Kit. For example, we would like to add Microsoft stacks (maybe BizTalk Server, ASP.net,) and more open source software projects. Please let us know.
The SOA Knowledge Kit is available for free download at http://soakit.pushtotest.com