Current SOA processes and guidance generally encourage a phase-based approach to SOA implementation, fully understanding the problem and defining the solution before implementation begins. Digital Focus, an east-coast firm specializing in Agile software development and integration, is convinced that Agile development practices are equally suited to implementing SOA. In August, Digital Focus published an experience report, "SOA, Meet Agile. Adopting SOA with Agile Teams" describing how SOA was successfully deployed using agile methods at Federal Home Loan Banks' Office of Finance (FHLB-OF).
In the following article, InfoQ editor Deborah Hartmann interviewed two people close to the project, to understand how it came about and how it worked out. First, Geoff Henton, CIO of FHLB-OF answered some general questions about using Agile practices on this SOA development effort, an approach they had formerly used only on software projects. Then Tom Stiehm, co-author of the report, filled us in on how the project unfolded.
InfoQ: Geoff, how significantly do you think Agile coaching contributed to the success of your first Agile projects?
Geoff Henton, CIO of FHLB-OF: Agile coaching was integral to our early success with agile software development. It helped to have an experienced practitioner to advise us as we adopted agile techniques. It was not unusual for our developers to push back on a particular agile tenet and having someone experienced providing the pros and cons regarding that aspect of agile was invaluable. Since we were a standard SDLC shop before adopting agile, there was a constant temptation to over-predict and deliver too many forward looking design and analysis artifacts. Our agile coaches were able to redirect our thinking to focus on what is important now rather than trying to design the entire system and predict a rigid project plan. If we didn't have the coaching assistance, I believe we would have given up on agile before actually experiencing all aspects of the methodology.
InfoQ: How are SOA and your agile software development practices complementary for business success?
GH: We are implementing SOA because our stand alone systems have increased in complexity as the systems matured and it is not unusual in the fixed-income capital markets to have to react quickly so as to realize a market opportunity. Being able to arrange our major systems functions in a more component-based manner without regard to systems boundaries helps us react quickly to change. Agile development practices provide for business success because it puts the business in control of their systems development effort allowing them to change direction mid-project while seeing the full effect of their decision on the project goals.
InfoQ: Both Agile and SOA talk about developing business-IT alignment. Can you share an example that illustrates movement toward this alignment within your organization?
HG: We found that agile provided the ability for the business to adapt quickly to the Fed's regulatory changes and SOA enabled us to deliver functionality that simultaneously served both internal and external demands. The Office of Finance (OF) provides issuance and servicing functions to the 12 regional FHLBs. We began our SOA project, designed to re-engineer our servicing system, using agile in late 2005. In early 2005, the business realized a new challenge as it related to a Federal Reserve regulation targeting daylight overdrafts. This required the OF to add new features to the release plan and move these new features up to the top of the development backlog. As work progressed on the new features, we saw a need to expose some of the underlying features as services to be consumed by our 12 Federal Home Loan Banks. These services allowed the FHLBs to easily import cash management data for use in their shops. Both agile and SOA contributed to us being able to deliver the required functionality on time while demonstrating flexibilty that served to increase business confidence in IT for the remainder of the project.
InfoQ: Thanks, Geoff. Thomas, you also talk about business-IT alignment in your article. Would you talk a bit about the benefits of this alignment?
Tom Stiehm, Principal, Digital Focus: Business and IT alignment is about getting the best person in the company or project to make a specific decision. This means the person with the most subject matter expertise and experience, around the concerns of a given decision, should make the decision. To me this boils down to: the business gets to decide what is built and development gets to decide how it is built.
The benefits of Business and IT alignment is that applications are built that support the business process so they will be used by the business because they actually increase their productivity. Before joining Digital Focus I had the unfortunate experience of building what I considered great applications that did all kinds of wonderful things but just weren't what the business users needed. They lived the life of all shelf-ware: installed once, deleted in a server upgrade, never to be missed.
This scenario happens a lot when you have rigid requirements that have a high cost to change, and it is a real loss to all involved. The business doesn't get the software they need and often can't get a second try funded for some time. The developers feel their time and talents are wasted because they spent time building an application that is never used. The business pays for the software and don't get the benefit of increased productivity.
One the other hand when you do align business and IT and deliver solutions that are used, every one wins. The users get applications they can use, the business gets a return on investment and the people who worked on the project get the satisfaction of putting a useful application into production.
InfoQ: Thomas, can you give us more details about the project? Which groups within the organisation were involved?
TS: The primary business client was FHLB-OF's Debt Servicing Group and we worked with the client's IT department. The development team was a mixture of Digital Focus and client developers. We also worked with the Systems team to get releases into production. The vertical is Mortgage-related Financial Services.
The project's goal was to replace FHLB-OF's Debt Servicing Sytem (DSS). The Debt Services team uses DSS on a daily basis to service Bonds and other Debt. Debt Service is a critical business function for all 12 FHL Banks that must be actuate and timely.
InfoQ: Was kind of work were you doing with FHLB-OF before the SOA project?
TS: It was a typical agile coaching engagement for Digital Focus. I wasn't part of that one, another coach, Tony Batucan, worked on that initial project.
InfoQ: So, FHLB-OF liked your Agile approach to software projects sufficiently to try it on something different. How did you come up with your Agile approach to SOA? Did you create it yourselves?
TS: Yes, we had a team of four people that created the Agile SOA approach. We had a good mix of people with expertise in both Agile and SOA. For example, I have used agile methodologies to deliver projects for about five years. Jeff Nielsen, Digital Focus's Chief Scientist and Chief Agile Coach, also worked with us to help incorporated agile values into Agile SOA. I have delivered a good number of service based systems (systems using technologies older than the current SOA tool set) and some SOA systems using the current SOA technology toolset. One of the other team members had experience with distributed enterprise architectures.
After getting the team together, we locked ourselves up in a conference room at the client site for a month and hammered out the details of the Agile SOA approach. We even took an agile approach to writing the Agile SOA approach. Given that we had a month to develop an approach and implement it with our client on a project immediately afterward, we decided to work in daily iterations. We would have a standup each morning where we would discuss the work from yesterday and the work for the current day. We would use this time to plan out the next steps and plan our late afternoon deliverable. Each day in the late afternoon we would meet with the client, discuss our progress to date, emphasizing the work done since the last meeting. After discussing our progress, we would discuss the work we planned on delivering the next day with the client to make sure all of their requirements were being met. We also used this meeting to reflect on the project and discuss any issues or concerns. After the meeting with the client the team would meet to discuss any feedback from the client and any new requirements. This final meeting wrapped up the iteration and set us for the next day.
One of the difficult aspects of this project was the short time frame, made even harder by the already busy schedules of most of the team members. Using agile practices and extremely short iterations allowed us to best leverage the team members, given their schedules. For example, Jeff couldn't be at the client site every day. But he wanted to present the topics of the approach he primarily worked on to the clients directly. We were able to schedule topic reviews around the needs of the team so that Jeff could get direct feedback from the client.
At the end of the initial one month project we delivered an 80 page Agile SOA migration document that was used to introduce the client's staff to SOA and Agile SOA practices. The next day we began work implementing an initial SOA application for the client, leveraging the Agile SOA methodology.
InfoQ: How long were your iterations? How did you pick this length?
TS: For writing the Agile SOA methodology we used one-day iterations. We picked this length because it allowed us to maximize the amount of feedback from our client given the amount of time we had for the project. It worked out well and I would recommend it for writing technical documents but not for writing software. For the actual software development we used two week iterations. We picked two weeks because it is the default iteration length at Digital Focus.
InfoQ: Would this approach translate well to another project, or is there a need to develop such a process "in situ" to account for each project's differences?
TS: There are some client-specific aspects to the approach, but that is less than 10% of the overall approach. Just as with any methodology, certain aspects of the approach require an "in situ" resolution that will vary from project to project.
That said, the approach we developed would translate well to other projects. We developed the approach with general Agile SOA development in mind. Our client wanted an approach that could be applied to any project they have for the next ten years so the practices in the approach are solid software engineering practices that any IT shop could follow and get benefit from.
InfoQ: You talk in the article about frequent feedback cycles. How did you track the effectiveness of your new hybrid approach during the project?
TS: We use standard agile practices to track our work and feedback cycles. This includes release planning, iteration planning, iteration kickoff, requirements gathering, UAT at the end of an iteration and project retrospectives at the end of each iteration. We also have project steering meetings that allow the client to give the project leadership feedback, away from the team.
Effectiveness is measured in client satisfaction, by the value of the application and services. This is a subjective measure but some of the things that go into it include: time to market, flexibility of design, decreased time to make changes to the application or business process, decreased time to add business processes and applications.
InfoQ: Can you give us an example of using the "keep it simple" principle on the SOA implementation project?
TS: A good example of "keep it simple" with Agile SOA is: don't finalize your WSDL until you deploy it into production (and then it is only a version of that WSDL). I have seen SOA projects set their WSDL in stone after a design phase. This can lead to WSDL that doesn't make sense vis-à-vis requirements changes. Since WSDL is the interface you are using between service providers and service consumers it is hard to get it "right" before any code it written. Once it is "final" some application will use that interface and expect you to support it for a long time to come. A fixed and stable interface is reasonable for a service that is in production (i.e. vetted by actual working code that has made it through QA) but a fixed, unproven interface is unreasonable during development of that service.
Another good keep it simple rule: you shouldn't make any functionality a service unless you have at least two consumers for that service. With SOA there is a tendency to have a good amount of speculative service creation, a kind of "build it and they will come" mentality. While services are good and can make your applications more flexible, they also increase the complexity of your code base and testing strategy.
InfoQ: How long did this project take? How long did you estimate it would have taken in the old paradigm? On what do you base this estimate?
TS: The writing of the approach took a month but I don't think that is what you are talking about. The implementation of the initial project using that approach is still ongoing. I believe it will take in total a year and half to complete.
The old paradigm is an interesting question, do you mean the old client/server paradigm the client used or a more waterfall SOA paradigm or a strictly agile paradigm (removing the SOA aspects)?
The client is replacing their client/server applications and they won't build new projects using that technology. That said, I think they could re-implement that application using their client/server development environment in two years. This is based on my understanding of client/server development practices and discussion with their developers experienced in that environment.
I would estimate, using what I understand of IBM's published SOA methodology that the project would take at least two years and maybe longer to complete using a waterfall-like SOA approach. I think it would take at least a year before the client would be to see any output of the effort in order to provide feedback. This introduces the risk that the business requirements would change before the application is put into production and that speculative services would be built and not used.
Using just an agile approach to re-implement the client/server application as a Web Application would take between a year and a year and a half. This is based on my experience estimating agile projects. In the end I believe the agile SOA approach will give you about a 25-35% increase in time but the trade off for that increase in time will be an SOA with vetted and usable services.
InfoQ: Were you actually able to iteratively deliver working services, i.e. that were used before the project's end?
TS: Yes, we have services in production that are being used and the project is ongoing. It is certainly easier to grow services that haven't been pushed in to production but if you version your services and account for backward compatibility you can deliver usable services incrementally.
InfoQ: Thank-you for your time, Tom. Best of luck with the rest of this project!
The Digital Focus case study points out 4 Keys to SOA Success:
- understanding the business processes the client needs to implement;
- understanding SOA architectures and patterns;
- adopting an SOA platform that can grow as the SOA application portfolio grows; and
- getting frequent feedback from the customers and development team in order to fix issues as they arise.
It also outlines the 6 steps they developed for their own Agile SOA project:
Step 1: SOA Introduction and Education,
Step 2: Marriage of SOA and Agile Methods,
Step 3: Define Long-Term Business Goals,
Step 5: Define an Initial Software Platform,
Step 6: Define and Implement a Self-Contained Application or Application Sub-System to be Your First Release.
You can read the case study: SOA, Meet Agile - Adopting SOA with Agile Teams, on the Digital Focus website.