The publication of “The Essence of Software Engineering” gives visibility for the Software Engineering Method and Theory (SEMAT) initiative to a larger audience. The actionable kernel described in this book is something that is useful in software development organizations to understand and improve their way of working.
The kernel consists of things we always have and things we always do when teams develop software. Teams can use the kernel elements to discuss how they will be doing what they need to do, and explore different ways of doing it depending on their needs. The kernel helps team to focus on results rather than on documentation or on activities. The descriptions of the elements on cards can be used by team members in their daily work, enabling teams to self assess their way of working and continuously improve themselves.
This first SEMAT book gives much attention to the agile principles and ways of software development, thereby confirming that agile is a usable and relevant approach to develop software. This book can be very helpful for organizations that are adopting agile and lean, as it actively support the agile concept of self-organization with the software engineering kernel.
InfoQ did an interview with Ivar Jacobson, one of the authors of the book The Essence of Software Engineering. He is also one of the leaders of SEMAT, the objective of which is to refound software engineering as a rigorous discipline.
InfoQ: Thank Ivar for doing this interview with InfoQ, Why do you now focus your interest in SEMAT?
Ivar: Since around Y2000 my interest moved from the technical practices of software development to the human practices. I realized that all the wonderful techniques that we had developed were only adopted by a few percent of the developer community. The adoption hasn’t increased and today the whole community is closer to 20 million people. At the same time the agile movement grew very rapidly. I agreed with the manifesto, I supported many of the new practices coming up in the agile community. In my company we transformed many of the existing practices such as use cases and architecture to become superlight, agile and lean, learning from the rest of the world.
Our focus became scaling agile in larger companies. We developed the three imperatives for scaling agile, which we now work with our customers. In working with this we realized that the software community has some more fundamental problems to deal with. These problems were not solved by any existing method, and if we could solve them, they would be applicable to all existing methods – agile or not. This is how SEMAT was born. We realized it was a gigantic task we had in front of us, it would take many years to realize it, and the road ahead would be bumpy with no rewards to be expected in a long time. And, we also realized that to become successful we needed a fundamentally different way of working with methods than we had had before.
InfoQ: For those readers that do not know what Software Engineering Method And Theory (SEMAT) is, can you give an overview?
Ivar: SEMAT was founded in 2009 with the objective to “refound software engineering based on a solid theory, proven principles and best practices that include a kernel of widely-agreed elements, extensible for specific uses”. I just quoted our Call for Action. We had identified a number of critical problems in the software community such as “the prevalence of fads more typical of fashion industry than of an engineering discipline”, “the huge number of methods and method variants, with differences little understood and artificially magnified”, and “the split between industry practice and academic research”.
More than 50 people around the world are now actively working in SEMAT, all on a volunteer basis, and more than 2000 people support it. SEMAT has taken its first step, namely to get its kernel called Essence adopted as a standard. Essence represents a paradigm shift in the way we work with methods. Historically, all methods have been manifest as descriptions (papers, books, handbooks), but working with SEMAT, the focus will move from talking (describing) to doing. Developers will get support in a lean and agile fashion when they actually work using their chosen method.
InfoQ: The SEMAT book describes a kernel of software engineering elements. What are these elements?
Ivar: The kernel includes things we always have and things we always do when we develop software. It is universal to all software development endeavours – agile or pre-agile. SEMAT has adopted many fundamentally new ideas. One of them being what we call the alpha concept. Over more than seven years this concept has been used and its value has been proven by many clients of my company. This concept allows team to focus on the doing instead on the describing. Teams can measure the progress and health of their endeavors in a practice or method independent way. At any moment in time the team can identify where it is and where it is going next. It makes the team result-focused instead of document-driven or activity-centric. The alpha concept also supports several other ideas, for example the idea of a method being a composition of practices built on top of the universal kernel. Thanks to the alpha concept we have been able to create a robust, extensible, intuitive kernel.
InfoQ: You already mentioned one of the new ideas adopted by SEMAT – the alpha concept. Can you mention some more?
Ivar: SEMAT relies on two important principles: 1) Agile while working with methods, and 2) Separation of Concerns. These principles have guided us in almost everything we have done when working on Essence.
InfoQ: What is it that makes agile working with methods so important for software engineering?
Ivar: The principle ‘Agile while working with methods’ requires a change in mindset of similar nature as that required in moving from traditional software development to agile development, for example a) the full team owns their method, rather than a select few methodologists, b) focus on method use rather than comprehensive method description, c) evolve your team’s method, rather than keeping your method fixed.
InfoQ: What do you mean by separation of concerns?
Ivar: Separation of Concerns can have many interpretations, but the one we have applied can be simply described as “keep the basics simple but allow it to be extended without changing the basics”. Usually we humans complicate things as we evolve them. For instance, usually the first release of a piece of software is simple because it has only the minimum functionality. If the software is successful people want to add features, and to do that they feel they have to change what is there to extend it with new features. As a result, the software becomes more complicated. The good news is that we really don’t need to do that. We can actually extend without changing what already works.
InfoQ: The book mentions that the kernel is actionable, with the aim to "supporting software professionals as they carry out their work". Can you elaborate on that?
Ivar: We are saying that the kernel is actionable and by that we mean that when you run an endeavor, the kernel comes alive by you using it. The essential elements always prevalent in every endeavor such as Requirements, Software System, Team, and Work move from state to state as you progress the work and apply practices, some of which may be described and others may be just tacit knowledge. For instance Software System will move over the following states Architecture Selected, Demonstrable, Usable, Ready, Operational, Retired.
InfoQ: The kernel uses what is called "alphas" and "states" to describe the way that software can be developed. Can you explain how they are used?
Ivar: There are seven alphas in the kernel: Requirements, Software System, Work, Way of Working, Team, etc. They represents different dimensions which you need to progress in an endeavor. Each one of them has a set of states, which you need to move to as the endeavor progress.
The first thing you do when you start working with Essence, is assess which states have already been achieved. For each alpha and each state there is a set of checkpoints, which describe a number of conditions to be fulfilled to reach that state. Once you have established which states you are in, you decide what your next move is going to be, that is which new states you want to achieve in your next iteration or sprint. For each such move there are a number of tasks to do. Which tasks to do comes from the practices you apply. If these practices are tacit, you apply them based on common knowledge in the team. If they are described, you get them from your practice descriptions.
InfoQ: When a team decides that they want to change their way of working, they can choose the practices that they need? When can they start using those practices?
Ivar: Smaller competent teams can use the kernel as it is. They have to understand the kernel alphas and their states, then they can, based on their existing experience and knowledge, agree within the team how to make state changes. Less competent teams need guidance on how to make state changes. They will then be able to select practices from a practice library, and these practices are added on top of the kernel. Developing this library is now the top priority of SEMAT. Today just a few practices, for instance a Scrum practice and a use case slicing practice, have been developed to verify that that the whole idea works. The practices will at their base be developed for people less interested in details. Most developers in the community are happy with just a description of the essentials of a practice. However, some people, for instance people developing life-critical systems, need more detailed guiding to comply with regulations in their field. Both these user types, and several others, will be satisfied by the library.
InfoQ: The Essence makes a distinction between alphas and work products. What is the difference?
Ivar: First we use the term work product to stand for documents and similar artifacts.
As I have already explained, in Essence the key elements are alphas. Alphas have states and definitions of the states using checkpoints. The state of an alpha is not directly determined by the production of work products, but by its progress and health (defined by the checkpoints). And the kernel has seven alphas only.
On top of the kernel we can compose practices. With every practice you can 'add' new alphas specific for the practice. Work products also come with practices; thus they are practice specific. Every work product is attached to an alpha. Some work products are attached to the kernel alphas, some to the practice-specific alphas. Here we have an interesting feature of Essence. The number of work products for a product is n times the number of alphas, where n may be as high as 10. Thus emphasizing alphas instead of work products we focus on say ten elements instead of 100, which is quite an improvement.
Moreover, in Essence each work product has an attribute 'level of detail', which can take several different values such as tacit, outlined, bulleted,...fully described. Agile projects usually have a low level of detail. Life-critical systems usually require a high level of detail. And, as we know, the fact that you have a high level of detail of a work product is no guarantee that its alpha has progressed to a new state.
In my own experience from really large telecom products, most documents were never read, just written. Maybe I should more correctly have said that most words in the documents were never read; they were not essential for understanding - they were there to fill in a template. With the Essence approach the documentation of a product will be reduced dramatically to what really is essential. The alphas measure progress, and progress is not measured in massive documentation.
InfoQ: Can you give us some concrete example of how SEMAT has applied separation of concern when Essence was designed?
Ivar: The kernel is very simple, but it can be extended with practices to build more advanced methods without changing the kernel. Here some more examples: a) separating the kernel from the practices, b) separating alphas from work products, c) separating what is important from the details.
Being able to separate what is important from details allow us to have different descriptions for different user types. Most practitioners in the world of 20 million developers are not interested in comprehensive descriptions; they just want what is important. However, there are as we already have discussed some developers of for instance life-critical systems who are obliged to define their practices in high detail. They need more but that more must not complicate what is needed by developers of less critical systems.
InfoQ: Earlier you mentioned that the road the realize SEMAT would be bumpy, and that it would take time before there would be rewards. This sounds like something where agile thinking could help you, by taking on risks early and delivering early results with the aim to get quick feedback from the community?
Ivar: Thanks for asking this question. Of course, we developed Essence iteratively. Essence includes both the kernel consisting of universal elements and a language. The language was designed to describe the kernel elements, to describe practices built on top of the kernel and to describe entire methods, which are compositions of practices.
The work started with three architectural spikes (sprints, iterations), and in each spike we progressed the kernel, the language and a few practices on top of the kernel. We tried out several sets of kernel elements, language constructs and practices before we converged on what now is in Essence. Intentionally we made the language very simple for practitioners not very interested in formal methods. By using the Separation of Concerns principle we then added language constructs to express more details for people asking for more.
InfoQ: How does SEMAT help to bridge the gap between the research on software engineering and application with industry practices?
Ivar: This still remains to be seen of course, but there is some evidence that this may happen. Within SEMAT we have a great group of people working on what we call the Theory Area. It is looking for a general theory in software engineering. At the end of May this year it had a very successful workshop at ICSE13 in San Francisco with 25 participants and 10 published papers. This far the work is at its beginning, establishing the need for a theory and identifying requirements on such a theory.
The Essence kernel is expected to serve an important role in being a definition of what software engineering (or software development for that matter) really is, which in my mind most likely is needed to create a practical theory. Moreover, the ability to measure progress should also be useful to evaluate any theory. Having said that, we are not pushing the theoreticians to adopt the Essence in their work on a theory.
Clearly the objective is to get a practical theory, eventually assisting us when developing software. Some people including me think that this theory will be socio-technical recognizing that developing software is a human activity applying technical practices.
InfoQ: If an organization wants to use SEMAT, are there specific training classes available on the kernel and on how to apply it?
Ivar: Training in the kernel and its application is done at many different levels. A three-hour tutorial on Essence was given in SFO in connection with the GTSE workshop and it will be given June 7 in Moscow and in Berlin June 20. At the university level, it is being taught at Florida Atlantic University, at Universidad Nacional de Colombia, Royal Institute of Technology in Sweden, Oslo University in Norway, and many other places. My company has been teaching a kernel since 2007. We are also teaching ‘Agile & SEMAT – Perfect Partners’ in Europe, USA and China. I know that many other companies are interested in providing training in SEMAT and its outcome.
InfoQ: Are there any costs involved if an organization wants to use SEMAT, for instance for licensing or certification?
Ivar: There are no licensing costs to use SEMAT. There are also some tools available at no cost. Certification will be done at several different levels and that incurs some costs.
InfoQ: Where can people go if they want to learn more about SEMAT, have questions, or need any support?
Ivar: People can go to the SEMAT website where they can find news and what is going on around the world. If they really want to dive deep, they could read the Essence specification.
If people are interested in participating in the work moving forward they can become a supporter. They can contact me or any other of the other people working with SEMAT, and we will find something interesting to create.
About the Interviewee
Dr. Ivar Jacobson is a father of components and component architecture, use cases, the Unified Modeling Language and the Rational Unified Process. He has contributed to modern business modeling and aspect-oriented software development. Lately, Ivar has been working on how to deal with methods and tools in an agile and lean way.
In 2004, Ivar received the Gustaf Dalen medal from Chalmers Institute Of Technology, Gothenburg Sweden. He is an international honorary advisor at Peking University, Beijing, and he is an honorary doctor at San Martin de Porres University, Peru. He is the principal author of six influential and best-selling books. He co-authored two UML books with Grady Booch and James Rumbaugh. Ivar is also a founder of Ivar Jacobson International, operating in seven countries around the world. Information about SEMAT is provided on the IJI SEMAT Support Page.
