Transitioning to SOA brings many new challenges to the software development lifecycle: these challenges can only be met if organisations grow an explicitly service-oriented development capability in response.
This article offers some practical suggestions for improving an organisation's service development capability. After summarising some of the challenges SOA brings to the software development lifecycle, it describes how consumer-driven contracts, in the form of "stories for services" and unit tests exchanged between service development streams, can strengthen the service-oriented development lifecycle.
Development Challenges for SOA
Service orientation requires much more than just an adopting a new architecture. SOA initiatives founder when organisations revise their architecture to make it more service-oriented, but don't change their development techniques to match.
Some of the challenges around initiating, building and operating services include:
- During the initiation phase, service functionality must be specified such that it is amenable to being reused in multiple contexts: not so coarse as to be usable in just one particular context, not so granular that it can't be reused in different contexts without considerable supplementary effort.
- When building services, we must ensure that providers and consumers can evolve independently of one another. Services aren't really loosely coupled if all parties to a piece of service functionality must change at the same time.
- When operating services, we need to understand the relationships between services so that we can diagnose issues, assess the impact - often at some remove - of variations in service availability, and plan for evolving individual services in response to new or changed business requirements.
From the point of view of the estate as a whole, each service's specific development lifecycle must be reconciled with that of its peers, but without unduly locking all parties into a monolithic, glacially slow, ultimately unworkable timeline.
The challenges posed by SOA to the software development lifecycle stem from the federated nature of the service estate. Valuable business benefits are no longer realised by walled-off applications with single-track timelines and discrete ownership, budgetary and operational boundaries; rather, they are satisfied by the interactions between services, each of which inhabits its own quite distinct service development lifecycle.Reconciling these several lifecycles however, is not a one-off accomplishment. Instead, as we'll see later, we are led to identify a set of collaborative activities and artefacts that represent a commitment to delivering a shared outcome. These activities and artefacts provide the basis for joining streams up - if only briefly - every now and then.
Development challenges are all the more acute when organisations attempt to implement SOA in an agile fashion, delivering high-value business benefits early and often through frequent releases. Despite its name, many see agile software delivery as being inimical to the organisational agility benefits associated with SOA. The agile practice of making frequent releases is often curtailed because of the operational risk of deploying new versions of a service into an environment where there are many existing relationships between services. And whilst agile encourages the close and timely collaboration of all parties responsible for delivering a business benefit, such activities can be difficult to coordinate across several different services and their lifecycles.
Federated Expectations
Traditional stovepipe development brackets off peer delivery streams behind business ownership, budgetary and operational boundaries. Service-oriented development differs from traditional stovepipe development in that it emphasises and incorporates external dependencies and obligations into each phase of the development lifecycle. This is the province of SOA governance: managing delivery interrelationships and aligning streams around a common representation of the service estate and the business activities, processes and outcomes supported by the estate. SOA governance depends on insight and feedback: insight into the current state of the service inventory, feedback on the impact of evolving service providers and consumers. We can increase insight and feedback by capturing federated expectations and obligations as artefacts. These artefacts can then be transitioned across phases within a particular service development stream and/or exchanged between different development streams.
One practical way to capture and exchange expectations and obligations is to use what are called consumer-driven contracts. Consumer-driven contracts are part of a closely linked trio of service contracts - provider contracts, consumer contracts, and the consumer-driven contracts themselves - that between them describe the relationship between service providers and consumers in terms of expectations and obligations:
- Provider contracts - Provider contracts are the kind of service contract with which we are most familiar -think WSDL plus XML Schema plus WS-Policy. Provider contracts are, as the name implies, provider-centric. Providers mandate what they have to offer; each consumer then binds itself to this one-size-fits-alloffering. By adopting a provider contract, a consumercouples itself to the entirety of the provider's functionality, irrespective of how much of this functionalityit really needs.
- Consumer contracts - A consumer contract, on the other hand,is a more exact description of an individual consumer's needs. Consumer contracts represent the specific pieces of provider functionality required by that consumer in the context of a particular interaction. Consumer contracts can be used to annotate an existing provider contract, or they can help discover an as-yet unspecified provider contract.
- Consumer-driven contracts - A consumer-driven contract is a representation of a service provider's obligations to all its current consumers. Consumer-driven contracts are created when each consumercommunicates to the provider its specific expectations. The obligations created on the provider's side establish a consumer-driven contract. As part of adopting a consumer-driven contract, a provider is free to evolve and innovate its service offering just so long as it continues to satisfy its existing obligations.
The Service-Oriented Development Lifecycle
How do consumer-driven contracts map back onto the service-oriented development lifecycle? How do they help address the challenges posed by a federated service estate?
Initiate
In the first stages of service development, consumer-driven contracts look very much like user stories for services. Services are useful only insofar as they are consumed, and so the best way of specifying their usefulness is to describe what consumers expect of them. User stories help us frame the delivery of key pieces of service functionality by describing high priority business outcomes in terms of roles, features and benefits (see Dan North's What‘s in a story and Mike Cohn's User Stories Applied). Consumer-driven contracts implemented as user stories describe a service consumer role, the outcome or benefit the consumer is seeking to realise, and the features or behaviour it requires of a provider in order to achieve that outcome.
This approach to specifying business-meaningful service behaviour is decidedly outside-in, cutting across business, organisational and technology boundaries to capture dependencies, expectations and obligations from other work streams and development lifecycles. In practice it is similar to Behaviour-Driven Development (BDD), which also emphasises an outside-in view of a business' goals, the benefits attached to those goals, and the activities and behaviours different actors need to exhibit in order to satisfy those goals. By identifying the behaviours and interactions that contribute towards an overall business outcome, consumer-driven contracts, like BDD exemplars, specify "just enough" functionality for a service to contribute significant value to the business.
The key thing about these externalised interactions and behaviours is that they represent the business-meaningful stuff that must happen for some part of a business activity to proceed or complete. The value inherent in the system comes to the surface where services meet, in the business events, document exchanges, and conversations that take place between parties. Consumer-driven contracts capture what one business group, function or capability expects of another in order to do its job. Taken together, these contracts provide a high-level view of the business activities that contribute most significantly to business outcomes.
Consumer-driven contracts address one of the more difficult questions that is often asked of SOA; namely: why service-enable an activity or process when the business' immediate needs can be met just as well, and perhaps more simply, in a traditional, stovepipe fashion? In a well-factored service estate, where really interesting and useful outcomes are satisfied by the interactions between several peer services, it's not always easy to tie an individual service back to one discrete set of business goals and benefits. Whereas stovepipe applications can often be identified directly with a concrete benefit or outcome, the "process agnosticism" of many services puts them at one remove from the value they deliver to the business. To recognise the specific benefits and outcomes supported by a service, we need to understand that service in its collaborative context. This is where consumer-driven contracts help: a consumer-driven contract that represents the collaborative expectations of the service community effectively mediates the indirectly accessible value of the ensemble by way of its more accessible pair relations.
Build
During the build phase, consumer-driven stories can be turned into executable expectations. Because they describe acceptable outcomes, stories can be translated into acceptance criteria, and ultimately into programmatic assertions, or tests. The least disruptive way of incorporating consumer-driven contracts into the development process is for the provider team to translate consumer-driven stories into programmatic contracts. This certainly provides a way of transitioning artefacts between phases in a service's development lifecycle, but it doesn't really do justice to the federated nature of the service estate. What's far more preferable is for the teams responsible for delivering consumer applications and services to write their own consumer tests, and then give these tests to the service provider. Each consumer gives the service provider a test-based version of its consumer contract: the set of contracts received by the provider constitute that provider's consumer-driven contract. Consumers can then develop against their own contracts, assured that the provider is being developed against the same set of expectations. Doing so bakes the contracts into both development streams.
To preserve the properly federated nature of the service estate we need to recognise that consumer expectations and contracts are just that: the property of the consumer. If we take away a consumer's ownership of its expectations, we diminish the collaborative force of the organisational and architectural aspects of SOA. By assigning ownership of consumer contracts to consumer development streams, we ensure that a provider's consumer-driven contract is derived from first-hand consumer artefacts, rather than the provider's interpretation of consumer expectations. Exchanging tests in this way links development and build processes and reduces the risk of provider-consumer drift.
Occasionally, the contracts required by different consumers will conflict with one another.Making consumer contracts explicit helps us identify such conflicts early. If the conflict is relatively minor, we can discuss with the relevant teams how we might best accommodate all parties - perhaps by relaxing some rule in one consumer or another. If the conflict is more serious, it may indicate that we need to revisit our design, with the aim of factoring out some of the operations and messages to target more narrowly focussedconcerns. It's partly a matter of judgement: consumer-driven contracts can guide us to produce the minimum set of reusable, process agnostic behaviour, but we still need to ensure that the results arebusiness meaningful and achieve a useful separation of concerns.
The simplest consumer contracts make assertions about the messages exchanged between service providers and consumers. For XML-based messages, consumers can express their expectations using languages such as XPath or Schematron, both of which work well in a heterogeneous technology environment. Consumers can also use XML Schema and RELAX NG, but because validation with these languages tends to be an all-or-nothing affair, consumer contracts written in XML Schema or RELAX NG must provide extension points to account for the parts of the provider contract not currently of interest to the consumer, which somewhat defeats the purpose.
Consumer contracts have a direct impact on the kinds of provider contract exposed by a service. If a service uses XML Schema to describe the structure of the messages it sends and receives, then the schemas it publishes must satisfy its consumers' expectations. For a new service, this may involve development teams from both the provider and consumer camps jointly designing the consumer assertions and the provider schemas that satisfy those assertions. For existing services, consumers might choose to annotate a copy of the schema with custom assertions. For providers, consumer-driven contracts provide a clear direction for development. Much as any with other unit test, consumer tests can be used to drive development activities, with developers creating service behaviours to satisfy the tests' expectations. Tests can also be added to a continuous integration environment and asserted against service functionality with each build. If consumers supply contracts in the form of raw XPath orSchematronexpressions, the provider can execute the underlying tests against the messages it produces using a unit testing framework such as JUnit,NUnit or XMLUnit.
Providers can even incorporate consumer-driven contracts into their runtime activities so as to provide early feedback when real-world messages fail to satisfy one or more consumer expectations. If strict conformance to all consumer contracts is an absolute must, the provider may have to validate the messages it sends by asserting its consumers' expectations in the send pipeline. Depending on the size and number of contracts to be asserted, this can impact latency and throughput. As an alternative, the provider can implement a Wire Tap or some other additional subscription to sent messages, and assert its obligations outside the send pipeline.
On the consumer side, consumer contracts encourage some very particular development practices. If providers are obliged to adhere to consumer-driven contracts - nothing more, nothing less - then consumers must ensure that they too stick to their side of the bargain. Rather than importing the entirety of a provider contract, consumers should consume only what they've told the provider they need. This way, consumers protect themselves against changes to the parts of the provider contract that don't interest them. Consuming only what is absolutely necessary means not doing schema validation on the consumer side: assume instead that the provider is adhering to the expectations it's been passed and is sending a sufficiently valid message in line with those expectations. Depending on the degree to which any deserialisation mechanisms on the consumer side can tolerate additional, missing or out-of-order fields, consumers may also wish to avoid deserialising received messages into strongly-typed representations; choosing instead to XPath out the parts of the message in which they're interested. This strategy is often called WS-DuckTyping.
Test-based consumer contracts are also usefulfor evaluating vendor packagesand COTS applications. Taking a consumer-driven approach, we describe our requirements for these packages in terms of the collaborations in which they will participate and the expectations that other parts of the system will have of them. By cataloguing consumer expectations, we specify what a package implementation ought to look like from the point of view of our externalized collaboration points; by instantiating these expectations as consumer-driven contracts, perhaps in the form of a suite of automated tests, we can decide whether the application is likely to be a "good citizen" of our estate.
Operate
The tests that have been exchanged between service development teams identify the more significant external collaboration points in the service estate. These tests and the interrelationships they represent provide useful executable documentation for those responsible for operating services. Consumer-driven contracts identify who depends on a service, and what the nature of that dependency is, increasing operations' insight into the day-to-day behaviour of the estate.
Besides making the relationships in the service estate more transparent, a suite of consumer-driven contracts implemented as shared tests also provides increased feedback on the effects of evolving services. Consumer-driven contracts comprise a foundation of asserted behaviour against which we can assess the likely impact of changes to a service contract. Consumer tests form the basis of a regression test suite that spans service development streams, and in accordance with which providers can evolve, just so long as they continue to satisfy existing obligations.
To the extent that consumer-driven contracts are used to plan changes to the service estate, they contribute to a delivery organisation's versioning strategy. Whilst the principles of backwards and forwards compatibility remain paramount in versioning services, consumer-driven contracts help contextualise compatibility issues based on extant obligations and relationships. A change to a mandatory element need not always be considered a breaking change; rather, changes that require some form of versioning can be identified with reference to the consumer contracts they break. If existing consumers aren't using a mandatory message element, then changing or removing that element need no longer be considered a breaking change.
Summary
Adopting an SOA brings new challenges to the development lifecycle resulting from the federated nature of the service estate. A successful service development capability depends on gaining insight into and gathering feedback on multiple development streams. We can address some of these goals by using consumer-driven contracts in the form of "stories for services" and tests exchanged between development streams to link activities within a development lifecycle and coordinate activities between development streams. Consumer-driven contracts support the development and testing of service-oriented systems, and encourage collaboration between all parties responsible for the service lifecycle.
Further commentary on consumer-driven contracts can be found in The ThoughtWorks Anthology and on the author's blog.
About the Author
Ian Robinson is a Principal Consultant with ThoughtWorks, where he specializes in the design and delivery of service-oriented and distributed systems. He has written guidance for Microsoft on implementing integration patterns with Microsoft technologies, and has published articles on business-oriented development methodologies and distributed systems design – most recently in The ThoughtWorks Anthology (Pragmatic Programmers, 2008). He is currently co-authoring a book on Web-friendly enterprise integration.
