InfoQ's Hartmut Wilms had a chance to talk to Jezz Santos, a Senior Product Development Consultant with Microsoft Consulting Services, who specializes in Enterprise Product Development, Guidance Automation and Software Factories.
Jezz has been emerged in product development since the mid 1990's, in a multitude of development roles for various types of companies including research, ISV, GSI and now as a Microsoft consultant. He is more than 6 years old at Microsoft as a field consultant, and as the result of this work, he explores the means and methods of developing and packaging reusable assets and automated guidance, making it actionable and customizable for customers to leverage, in the form of Software Factories.
As a trusted expert advisor for the Web Service Software Factory and the creator of one of the world's first implementations of a software factory (the 'EFx Factory'), which demonstrates some of the advanced features of a future generation of software factories to come from Microsoft, we questioned him on the Microsoft Software Factory Initiative.
InfoQ: What's the Software Factory Initiative all about and what's your take on software development supported by Software Factories and Domain-Specific languages?
Jezz Santos (JES): In my view, the Software Factories initiative is all about driving the software development industry towards higher productivity, cheaper software development and higher quality software engineering. This has to happen today because software solutions today simply cost too much to build and maintain for the end customer, and we now live in a market where the supply of quality software cannot meet the demands.
In order to achieve that, several key pragmatic things must happen, compromises if you like, to meet that demand. Primarily, most software requirements need to start being standardised, and custom software requirements become the exception to the rule - not the norm. Custom requirements will incur today's development efforts and expenses, but that is expected for customizations to products from any industry. Secondly, software solutions need to be mostly assembled from reusable standard parts that satisfy these standard requirements. The standard parts need to be available in a form that can be reused easily. The standard parts need to be built and packaged in a form that can be easily integrated with, and configured (and customized) to suit, a variety of requirements of an overall solution. We need specialist people to build custom configurable tools that allow us to collate, configure and orchestrate the assembly of those parts into a specific architecture of specific families of solutions. These configurable tools will need to use simplified descriptions and specific, custom languages tailored for that domain so that the solution implementers can focus their energies satisfying the overall solution requirements.
Software factories are these custom, configurable orchestration tools which provide the capability to assemble these variable configurable parts into specific families of products. But they are not the answer to all software development solutions. Software factories only apply as the solution to projects which target specific well known solution domains, where the people building the factories possess the expert experience and specialist solution domain skills to capture within these tools.
The real value captured by software factories is that the component parts are orchestrated by a tool (the factory) which associates the component parts with representations (languages) in terms of the problem domain, and with specific configurable tools to configure or customize the parts specifically for the domain that the factory is addressing. In plain terms, the factory not only tells you what parts to use, and how they fit together, but how and when to use them in languages you would understand for the domain the factory builds solutions for.
To clarify, because the factory understands the architecture of the product it is assembling, and the component parts it uses in that architecture, and it is represented in higher-level languages pertinent to the problem domain; the factory can guide you on exactly how to build the specific solution to your specific requirements, with what parts, and the activities you need to perform on these parts to complete a specific solution. If you were using a factory, you would only have to understand the languages the factory uses to represent its architectures and its component parts, and how those parts can be configured to satisfy your specific requirements.
InfoQ: What is a software factory in concrete terms?
JES: There are really three major ingredients to putting a software factory together. The standard parts themselves, how they are represented, configured and cooperate together, and the factory to assemble them all together. The standard parts are nothing new to most software development organisations, especially those who already employ a product line mentality. The process of building a factory is taking these parts (assets) and applying abstractions and automation to them, and integrating them into one tool.
InfoQ: So, how do you build one from these parts?
JES: I'll use this story to illustrate this. I'll assume you build custom software solutions today for your customers. Let's see how we get from these standard parts to building a software factory. I'll also introduce the factory jargon as we go.
Let's say that a customer, wants a custom built motorcycle, and they ask you to build it for them. Something a little different from what you can buy off the shelf, not factory-made (excuse the pun, I mean not a standard model), but custom made - but not from scratch - a standard type of bike model like a 'cruiser', with a few personal modifications.
Now clearly, the customer is asking you to do this, because (a) they can't buy this bike off the shelf from a store, because the store only sell standard models, (b) they don't want to specify a bike from scratch, a specific known classification of bike will suit their needs for now (the 'cruiser' type), (c) they want some personal special modifications done to this bike, and finally, (d) clearly they would rather pay someone to deliver it than take the risk of doing it themselves.
Typically, as a bike builder, your simplified options are:
- Build it from scratch - very risky, and expensive and low profit for you
- Pay someone else to make one for you (outsource) - expensive and low profit for you
- Buy a ready-made kit of one (a 'cruiser bike' kit), and build it, and customize it yourself - cheaper more profit.
Now, this bike you want to build would be a very specific, known type of motorcycle (a 'cruiser'). The special kit would include most of the pre-made parts (engine block, carburettors, cylinders, wheels etc), some raw materials (fabric, tubing, sheet metal, plastic etc) and specialised tools to construct the bike with the step by step instructions on what to do and what pieces you need at each step. You still need to build it all yourself, and wield the tools in the right sequence of steps, against the right materials and parts. But most of the really detailed hard parts like carburetion, power transmission, fuel injection etc, are already solved and come in ready-made components. You just have to hook them all together to work on a frame you can sit on.
InfoQ: How is this different from how we build software today?
JES: For a lot of successful solution providers, this is the experience of building software today. But, for others it's quite a different story when you have to start with only raw materials like plate steel and tubing with screws, spanners, welding torches and milling machines, with a rough sketch or notion of how these things should hang together to create the end result. (A certain TV series comes to mind for where they do that very well for custom bikes). Then actually going ahead and building a highly customized, quality, unique one-off bike for your customer and their particular needs. Not that it can't be done of course, many will try; a few will even succeed, and some become specialists at it. (As we witness in that TV series).
The cost of doing this custom one-off development though is enormous to the end customer, and few businesses can afford to buy into that market. Few customers today can afford to invest in that type of luxury just to get a nice looking functional bike that gets them from A to B in style. The story of software factories starts with a requirement to want to reach out to larger markets, where the solutions don't need to be so customized and the customers are more willing to make a 'reasonable' investment in a high quality product that is completely functional.
The reason the customer came to you in the first place, was (hopefully) you already possess the capability to build them such a custom solution - you know the domain. You can do it at a reasonable cost, and you've probably done it before, and could do it again - you own domain experience and solution domain knowledge. It is likely that you probably already did all the work of creating a custom motorbike solution, so you know what to do. Probably, you did that multiple times before and in that process, understood deeply what all the nuances of building a custom motorbike are. All the tips and tricks (practices, and patterns) and more importantly, how it should fit together properly to satisfy larger requirements like: road safety, performance, fuel efficiency, noise control, comfort etc. (domain specific quality attributes) - in essence you are a domain expert. In the process, as an outcome, you probably developed some of your own custom tools to help building the bike (Assets). You probably also harvested and refined certain parts and architecture you found in common with all your previous attempts at building these types of solutions. No doubt, in the process of building these types of solutions, you would have built several one-off bikes somewhere along the line, certainly some failures, and learned (usually the hard way) how to do it better, more efficiently and with a solution meeting better quality attributes.
In short, you have become a domain expert with domain specific assets for building these types of vehicles.
InfoQ: Sounds familiar. What should happen next?
JES: You may have then realised that, with a little extra work, you could create a whole family of bikes that were essentially similar using the same basic design and materials and tools. Perhaps these bikes could vary by having a custom fender design or a paint job, different length forks, or engine size and tyres. But the heart of all these bikes (the architecture), are all the same. You probably identified a group of other people (end-customers) who would be interested in having such bikes (products) of this particular design (product line). Then you identified a group of people capable of building these products for the end-customer (stakeholders) using a lot less effort than you did, with specific instructions and custom tools, because now you have a demand to make more than just you can handle.
You would have then had to generalise the model of a bike, then defined some of the core parts of it in general terms (work products) and how they fit together (a logical product architecture). Along with that, you would have had to also specify what could be tweaked on each of the logical parts to produce a slightly different variant of it that would suit different end-customers requirements (variability). And then, what was left as common to the particular parts you factored out into reusable assets.
Then you probably found a way to describe these parts in a less technical detailed, simplified custom language that anyone (i.e. bike enthusiasts, hobbyists, customers) could easily understand. This language describes the important aspects of the part, and hides the rest (abstraction). It defines what is needed to be tweaked (variability) to create other variants of the bike in your new product line. This custom language is then used to communicate ideas and requirements (domain specific languages). To communicate the complete description of a bike, you would have needed to provide various separate, but related descriptions of different aspects of the bike (view points), some perhaps spanning all parts of the bike: an electrical wiring diagram, the fuel and lubrication system, the cooling system schematic, the physical structural frame design (all abstractions, and models or different views of a bike).
Finally, you would have to communicate how to assemble each of these parts together to complete the overall architecture of the bike - into a solution. The series of steps to take, to assemble and configure the parts, including what parts need to be fit first and in what order (Activities). With that, you also include which raw materials to use (artefacts), and what tools to use (Assets) and further, what settings the tools should use to assemble those parts together.
As you can see, (if you are following along the hidden analogy to software design), this is not unlike the outcome of what most solution providers (people who solve the same problems over and over again for different customer requirements) get as an outcome from evolving their software development of a particular product from scratch today.
This process is quite natural and logical if you care about saving yourself a lot of work the next time you may have do it. This is what we can a product line mentality.
InfoQ: So how do software factories fit in?
JES: The real crux of the transition to software factories from this - is what happens next? Traditionally, in today's practices, the domain knowledge and experience is in the people's heads, and in the solutions that they build. And there is little means to package up their assets (intellectual and physical) into a reusable form for others to use on the next project. That's even assuming the next project seeks to re-use any of it! There is a human attitude adjustment that comes to play in this too. But more importantly is that these assets are probably not in a form of easy reuse in time for the next project. Typically, the instructions and languages are in detailed written (textual) form and the physical assets are in (low-level) undocumented binary form. Even then, the quality and clarity of those assets can be highly variable. Typically, consuming those assets is often perceived more work than it's worth, resulting in the fallback position of the 'not built here syndrome' we see so often.
Creativity is seriously underrated as a powerful driver for software reuse.
To turn these specifically focused assets into something highly reusable, configurable and customizable, and hence capitalise on the value of investment of learning this domain (and the value of those learning's and assets) - special, intentional steps need to be taken. In software development today it is often these steps that are typically not accounted for, and therefore receive no resourcing at the end of a software project, in preparation for the next one.
The missing piece that Software Factories brings to the table here is a means to systematically capture that value and those assets and package them into an automated form to be re-used by others. Those others, by the way, are not expected to necessarily possess the detailed expert domain experience, but they are expected to be able to define and assemble the same products quickly and easily using the factory.
The process of building the software factory is taking these assets, define configurable abstractions of them (which describe the various aspects of the bike), and then applying automation to translate those abstractions into physical solutions.
InfoQ: Who uses this software factory, and how does that experience look?
JES: The primary users of a software factory are software professionals building the solutions - these are the factory users that the factory builders design their factories for. Using the example above, the software factory displays various abstractions (models, designers, tool windows etc) to these factory users in familiar languages (to this domain) that describe the various features of the bike, and its configuration from different viewpoints. Each viewpoint may only be of interest to particular factory users in the development team. Maybe your team has different skilled resources. Perhaps wiring specialists, engine specialists, safety specialists or branding specialists, they can focus on one particular aspect of the bike, while others complete the others. The factory provides them with the right tools, and the right view of the bike to do that.
The factory provides instructions to the factory users in a form that guides them through each step of constructing each part, using the relevant abstractions - much like a task list. Continuously presenting the user with tasks that can be worked upon or completed, based on the state of the entire product at this moment in time. With each step the user takes, the factory presents the right tool to them (pre-configured with the right settings) to assemble each part. The user simply 'turns a few knobs, and switches', and provides/refines the configuration for the part that suits the particular needs of that instance.
InfoQ: What is the value of the software factory to the users?
JES: The value here, using the example above, is that a software factory built for this kind of bike, would enable a hobbyist or enthusiast biker, (someone with considerably much less technical knowledge of the inner workings of the bikes engine, superstructure, and plumbing) assemble their own custom bike.
They do this by selecting the various parts from the toolkit for the factory provides to them, place them in the right place in one of several diagrams (designers) provided to describe the bike, and tweak their various settings to suit their own custom needs. The software factory then orchestrates the automation of the assembly of the bike (usually in stages), and outputs an almost complete bike, which can then be fine tuned or customized by enthusiast.
The added bonus of course, is that the person using the factory learns from it, during the experience, and the experience encapsulated within the factory from the original builders of the factory starts to rub off on them. In a way, the factory teaches you the domain, without initially committing you to investing in learning it from personal practical experience.
InfoQ: You say 'outputs an almost complete bike'. Please explain.
JES: We say 'almost complete bike' in the previous statement, because the basic expectation, even today, is that very few factories that deliver whole completed software solutions to satisfy, ultimately, all end-user scenarios. This limitation is really a side-effect of the complexity of modelling an entire solution domain to the level of detail that would be needed to provide products to address 100% of all end user scenarios, and therefore a complete set of product variants. Initially, in earlier incarnations of a factory, the models are more simplified (abstractions are higher), and they will be fleshed out over time to accommodate more exception cases of variability to satisfy more refined requirements. Initially, for most factories built today we are seeing a target closer to 60-80% complete product, followed with some manual customization. However, that automation gap is expected to slowly rise over time, as that particular factory is iteratively refined, and the general factory building processes, tools and technologies are refined.
InfoQ: In order to provide a toolkit that allows for "simply" assembling the parts with some additional customizations a Software Factory needs to be based on a fixed architecture, doesn't it? What if I want to tweak some parts to suit my architectural needs? How flexible or composable is a Software Factory?
JES: Indeed, a software factory defines a product architecture, since it's this product architecture that provides the model upon which the factory operates. We call this the 'factory schema', and it tells us what we are building, with what, and how. It is part of the guidance the factory builder conveys for their model of the solution in this domain. How the factory builder presents the variability of this product architecture, is an attribute of the particular factory. Some factories will want to intentionally fix this architecture; some may allow it to be extended. (Architecture can also be a point of variability).
There is nothing to limit the scope of the product architecture. Typically, when we talk about a factory, we assume that a factory will build a whole solution, but this is not going to be the case for a lot of factories. A factory might build only a part of a solution, and then this part is given to another factory for assembly into a larger part, for assembly by another factory into another part - etc. You might do this with multiple separate factories, or more likely, you will have a grand factory, that orchestrates the component factories (or their products), and manages the assembly for you. Either way, composability is a very important goal of factories. In the latter case, we will be sharing or extending parts of a factory schema to achieve this. As demonstrated by the 'EFx Factory', and now the next version of the 'Web Service Factory' (from patterns &practices), your factory can provide 'extensibility points' for other factories to plug-into to provide certain components or aspects of your product or architecture for your factory.
This is also a major requirement to take what we call a 'horizontal factory', such as a web service factory (WSSF) (that creates a type of solution - any web service), and tailor it (verticalize it) towards a factory for a particular industry or technology vertical. For example, you could customize the WSSF and build a financial payments web service (FPSSF). This kind of 'vertical factory', then fixes or extends certain points of the web service product architecture either to constrain the variability of those parts, or specialise them for a more specialised domain. For example, the financial payments web service factory (FPSSF) could fix or limit the data contracts of the messages you could exchange, with industry standard payment schemas. In the former case, the FPSSF would effectively fix the point of variability of Data Contracts in the WSSF with its own fixed Data Contract, preventing the user configuring a data contract at all. In the later case, it would extend this point of variability, and only allow the user to select an industry standard data contract schema.
InfoQ: What is this "factory schema" in concrete terms?
JES: The heart of a software factory is the factory schema (a specification, itself a domain specific language), which tells us above all the architecture of a solution the factory is creating (the product). It tells us what the various logical parts of product are (the Work Products), how they are related to each other, how to view them (View Points), how each one is constructed (Activities), and with what tools and resources (Assets) it should be constructed with.
The instructions for product assembly - the Activities - are presented to the user as a list of things that can be worked upon at that particular moment in time. These are associated to the things you work on (Work Products) and the tools available to manipulate them (Assets). Activities are conditionally-based on the state of product itself; and driven by what the factory users have already done, and what is left to be done. Using the example from earlier, it does not make sense to mount the engine of the bike to the frame of the bike before the pistons and carburettor are in place yet. But you could start painting the frame of the bike, whilst someone else is building the engine. So these activities appear when it makes sense. In this way, instead of presenting you with a long list of unrelated activities you need to perform from A to Z serially to complete the product, the activities become available to you as you (or others) complete work on the parts they need as pre-conditions for further activities. This makes team development with a factory much easier, and provides contextual guidance to the users on what things need to be done now, and what things can't be done and why (what other things need to be done).
There are several ways to view a factory product (View Points). Like the electrical wiring diagram, cooling system, and lubrication system from our example earlier. The work products, assets and activities are organised by these view points. Each View (instance of a view point) is typically of interest to only certain people working on the product (Stakeholders), and each view makes available different assets with different activities to work within that view. If you were painting the frame of the bike, you wouldn't expect to be asked to 'connect the fuel lines' as an activity, and you wouldn't need the torque wrenches handy. Similarly, if you were building the engine, you wouldn't want the airbrush, and be told you could now 'prime coat the surfaces of the frame'. The view associates the right activities to work with the right assets to complete the work products you are working on now for your role. Clearly, you don't need to have mechanic's skills, and specialised tools just to paint a bike.
InfoQ: You've talked about domain-specific languages and explained their role in the software factory initiative. For those unfamiliar with this concept could you provide an example of a domain-specific language illustrating their benefit?
JES: You can think of a DSL as a means to communicate an idea or a concept in its own specific terms. You are not forced to describe it in a formal way that conforms to any rigid structure or process. Part of the problem with general modelling languages like UML in practice, is that you have to fit your concept into strict terms and structure (classes, operations, attributes etc), and by trying to do this you might spend too much time figuring out how to fit that formal description and notation and likely lose the fidelity or uniqueness of your concept and its behaviour.
A DSL is much more like giving you a large whiteboard, and saying "go draw your idea, label the parts, describe their attributes and show us how they interact!" Do that in whatever language, and vocabulary that make sense to the thing you are describing, and how people think about that thing. Now build a logical model of that idea, and figure out how you want to users to program it and display it. Perhaps as a text based language or more commonly as a graphical diagram.
The power here is that you are authoring your own language to describe this thing, not trying to describe it with another more granular, detailed language and its own ridged syntax.
If you are used to modelling your software in code, you would do this typically by crafting some programming interface and a whole object model to describe it. But look at the level of detail and time it would take to capture all your concepts (by the necessity of that programming language), and what about changing and refining that idea, or the meaning of its concepts? Is a class diagram the best way to represent what your software does to your customer?
Because a DSL is in much higher level terms than a general language, it becomes an abstraction of the thing it describes, and therefore a simpler way of thinking and interacting with it. This is very powerful in software engineering, where we need to clearly communicate the software we are building in terms more generally understood by those who collaborate in this specific domain (i.e. development teams, management, customers, marketplace etc).
As an example, in the domain of building layered applications and services, it is easier to convey the architecture of such an application, and the pieces and interactions of that architecture, in a language that focuses upon these parts only. This language is composed of common terms and vocabulary that is important to those who design such systems, giving them a means to program and configure this application at that level.
This DSL would be a good graphical example of such a language to describe this domain.
InfoQ: What are some advantages/disadvantages of using a domain-specific language in comparison to general purpose languages?
JES: For me, as a seasoned professional developer, it's clear from practice, that a general purpose language (pick C#/VB/C++/Java) is harder to drive than a constrained specific one (pick SQL, XAML, SOAP) to define the same thing (i.e. a SOAP message, a user interface or a database query). But I would certainly want to use a specific language like SQL to interact with a database, instead of writing the C# or C++ code to do the same. I am also certain that the database itself would prefer you to use that constrained language to do that too! That's because the general purpose language (like C#) does not tell you how to use it or what it should be used to describe, it has no bounds. The domain of things it can describe is so vast. This power is also its Achilles Heel, when it comes to applying it to building specific things. It's so flexible that you can use it in any way that you want to describe almost anything you want. Which means it's also hard to verify if you have at the end what you intended to have at the start.
For example, if I want to build Smart Client application and ensure I had addressed all the concerns of smart clients (i.e. offline, connectivity, auto-updating etc.), can you verify that I built it right using C#? But, you can certainly tell me if I described a database query wrong if I was using SQL.
A domain specific language immediately constrains you to defining the things it knows about, in the way it knows about them. It's much quicker and easy to configure these things because your options are limited (constrained). Therefore, it's easy to verify your use of the language in any given context. Furthermore it's much easier to use this language since it teaches you how to use it, and about the thing you are using it to describe. Furthermore, you typically don't have to be a seasoned programmer to use one, because the language and notations it uses should be familiar to the domain it addresses.
A specific language provides a simplification of the things it describes; it's also an abstraction. It presents you with only the aspects that are variable about the thing it describes, and hides all non-variable aspects and complexities of that. It also tells you how to configure those variable aspects.
There are downsides of course with using domain specific languages. If you want to describe or model something slightly different than the domain model you already have, then it's probably fruitless to do that with this domain model, unless the thing you want to describe is a near generalisation or specialisation of the thing the language currently describes. Because there are by nature very specific, they are naturally hard to generalise. In that respect they are generally not very flexible for describing other classes of domains. For example, if I had a specific language for describing Smart Client applications, it's probably worthless to adapt that for describing Web Clients. However a more general modelling language like UML can be easily adapted to describe both.
InfoQ: Thanks a lot for your time!