This paper tells the story of the emergence of the agile approach, focusing on the role of the originators of the Agile Manifesto in this paradigm shift. Based on in-depth interviews with twelve of the seventeen originators of the Agile Manifesto, we describe how technology-driven forces led to the cultural change introduced by the agile approach. We describe the originators' background as children, their main professional activities prior to the conceptualization of the Agile Manifesto, and the process that led to the formulation of the Agile Manifesto. One of the main messages we wish to convey in this paper is that while the originators of the Agile Manifesto were led by the technology, the software community was in fact captured by the human aspects of the Agile Manifesto. This message implies what human aspects and methods, practices and tools should be emphasized in adoption processes of agile software development.
Introduction
Agile software development introduces a new paradigm into software engineering. This paradigm legitimizes new methods, practices and tools, new values, new behaviors, new perceptions and new perspectives with respect to software development processes. Furthermore, the change introduced into software development environments by the agile approach is not restricted to the developers; it encompasses management, customers and other project stakeholders.
Clearly, the agile approach has become mainstream in the software development community [1]. From managerial and technological perspectives, examining the emergence of agile software development yields questions such as: How did this paradigm shift begin? What were the main forces behind its evolvement? How did it evolve? Was it envisioned?
In order to answer some of these questions, we interviewed twelve of the seventeen software practitioners who, in February 2001, formulated the Agile Manifesto (http://www.agilemanifesto.org/) during a three-day meeting that took place at a ski resort in the Wasatch Mountains of Utah and was dedicated to a discussion of the common aspects of the practitioners’ perception of software development. The main outcomes of that meeting were the decision to capture this commonality by the term agile and the formulation of the Agile Manifesto.
The formulation of the Agile Manifesto hints at people-centered ideas and a cultural change. Based on the interview analysis, however, it was revealed that the originators of the Agile Manifesto were driven, in fact, by technological forces, and a cultural change was not their explicit intention; it was, rather, the software industry that, at that time, was captured by the human, organizational and cultural aspects of the Agile Manifesto. In other words, the cultural change introduced by agile software development was the end of a story that was driven by technology-focused intentions. This paper tells this story as told by its key players – the originators of the Agile Manifesto. The story is presented using anonymous quotes taken from the interviews, with no mention of personal details; the originators are denoted by serial numbers.
Data Gathering and Analysis
Data was gathered from interviews with the Agile Manifesto originators (hereinafter, the Originators). The Appendix presents the interview questions. Each interview lasted about 1.5 hours.
Ten Originators were interviewed in August 2007 (nine were interviewed at the Agile 2007 conference, one was interviewed elsewhere in North America), two Originators were interviewed by email. The remaining five Originators either refused to be interviewed or could not be contacted.
Since the data is based on twelve interviews, statistical analysis is irrelevant in such a case. Rather, a qualitative approach was applied, using the content analysis method, which is a common analysis approach in qualitative research. The analysis process took advantage of the different professional disciplines of the authors of this paper, which enabled to examine the topic from two perspectives: the behavioral and psychological perspective on the one hand, and the technological perspective, on the other.
Guided by the qualitative approach, the following story is based on themes that emerged from the interview analysis. Therefore, not all answers to the interview questions are addressed in this paper. Only relevant major themes are presented and are demonstrated by a few illustrative quotes.
The Story of the Originators of the Agile Manifesto: From Technological Leaders to Community Leaders of a Cultural Change
This section presents the story of how the originators of the Agile Manifesto turned from technological leaders to leaders of a cultural change. The story is divided into three stages of the Originators’ life: prior to the formulation of the Agile Manifesto as children and software practitioners (Section 3.1), the turning point – the creation of the Agile Manifesto (Section 3.2), and today, as leaders of a cultural change (Section 3.3).
Prior to the Creation of the Agile Manifesto: Technology-Driven Forces
We divide the period in the Originators’ life prior to the creation of the Agile Manifesto into two phases: childhood (Section 3.1.1) and professional lives (Section 3.1.2).
Childhood
Based on developmental psychology, childhood experiences shape future behavior. We therefore focused, among other things, on the Originators' childhood, looking for similarities in the way in which they were raised in order to discover whether specific experiences in their early childhood influenced their future activities.
We found that most of the Originators grew up in homes in which the fathers were liberal professionals and the mothers were usually housewives, who had a fondness for reading, and other art- or culture-related hobbies. Most of the Originators had moved many times during their childhood and lived in several places; thus, they had to cope with changes from earlier stages of their life. Also, they were inspired at home by very similar values. We now elaborate and illustrate these similarities in the Originators’ childhoods.
The phenomenon of moving from one place to another (without mentioning specific names of places to maintain ethical norms) was expressed, for example, in terms such as:
[1]: I was born in Washington DC.
Q.: Have you left since then?
Hundreds of times.
[4]: I was born in [a city name], NC. … It is one of the biggest cities in NC. …Well, actually, I lived there for a while and then, from early childhood, I lived in [a city name] and [a city name] for a couple of years, I lived in [a state name] for a while, and then [from age] 5 or 6 to 11, I lived on a farm.
[5] We tried to count it [how many times we moved] but it is difficult. [Mentioned names of about ten places he had moved to] … The globe has different colors, different people [and different] cultures.
Parents' occupations seem to be very similar. Nine out of ten Originators, for whom we have information about their mother, had a creative housewife mother who encouraged education. Their fathers were liberal professionals, engaged in engineering (4), business (3), science (2), and medicine (1). As they indicate in the following quotes, this dual perspective influenced their future tendencies.
[8]: My mother was a mother, homemaker.… I think my mother influenced me as far as my love for reading. Both of my parents had respect for education.
[4]: I think I got some of my analytical part from my dad who was an engineer, and part of the social organizational development stuff from my mother.
[5]: Well, my father was very theoretical in his thinking and practical in his daily life…. My mother … was very articulate, I got that from her, she would make me go to art galleries, dances and theaters…. And that was connected to some of the stuff that I did, that I am comfortable being with people.
They were educated to do things correctly:
[1]: We were expected to behave well.
[8]: There was a certain amount of "clean up your room", and there was "you are going to school to do well", and "the teachers are the authority, so pay attention to the teacher."
[10]: Ingenuity, [that is,] do what needs to be done with what's on hand. Self-reliance. Perseverance. Education - in the sense of learning how things work, not necessarily in a formal setting.
All had a technical tendency from an early age, as the following quote illustrates:
[6]: The kind of hobbies I had, I was really interested in electronics, video, stereo, stuff like that.
Technology professionals
In college, most of the Agile Manifesto Originators studied computer science or electrical engineering. Their childhood and studentship both gave the impression of technology-oriented people. This impression is further illustrated in this sub-section, in which we specify the technological topics they worked on prior to the conceptualization of the Agile Manifesto. This illustration further strengthens the argument that technology-driven forces led to the creation of the Agile Manifesto, as the following quote indicates:
[8] [A software practitioner] once referred to the agile methods as methods developed by people who like to program and I think that's true. So it is very much bottom-up, programmer driven.
Specifically, prior to formulation of the Agile Manifesto, the Originators' work focused on three related topics – Smalltalk, object-oriented development and patterns. In what follows, we describe how their work on these topics led to the emergence of some agile practices and to the creation of a community that shares similar ideas about software development processes.
In general, Smalltalk, object-oriented development and patterns inspired the feeling that a different software development method, than the common approach applied at that time, was needed. For example, working with Smalltalk fostered a different development style, as the following quotes indicate:
[8]: One of the things that is striking about the people who wrote the Agile Manifesto is how many of them knew Smalltalk, the programming language … I thing that the roots of the style of agile development go back to the very flexible programming environment, particularly Smalltalk, which dates from the late 70s, early 80s, and also the style of developing, those characteristics of the programming language Lisp, which dates, I believe, from 1957. So those are, I think, the earliest things I can point to.
[11]: The lifecycle of Smalltalk programs was iterative, piecemeal, and organic. I felt much more comfortable trying ideas and seeing where they lead rather than trying to think everything out up front.
[9]: Smalltalk seemed like a clean, simple and powerful language. It was the computer language that was just made for expressing what I wanted to express. … Smalltalk was so interactive that we just said, well, let's just program it and see. So instead of just working on a white board and arguing about what is the right idea before we program, we just said, let's program and see. … We want to do that thing, let's just start it, just jump into it, and that was exactly the opposite of what everybody said the way it should be.… So you asked, where did it start? It was really the roots. It was a love of programming… Smalltalk really played to that love… and we said, well, there is something going on here and it is not a technique.
While Smalltalk was accepted as a laboratory programming language, object-oriented development was a programming paradigm that was starting to be used by the software industry. When object-oriented development began being used extensively, a suitable development methodology, different than the one used for procedural programming, was sought after.
[6]: I was … asked to build a development tool that would take their company to the right level, … leverage them to an object-oriented software environment in a process that could accelerate the development process.
[3]: I think some of the influence [on agile development] was object-oriented design, when people [used] … incremental development, rather than trying to figure out [all the requirements a priori]. … We didn’t call it agile, we didn't know it was agile. We were just working in close collaboration with customers. That was probably in the mid 90s.
[9]: Well, there are a couple of things [about object-oriented development]. One, scheduling was very simple because I did not worry about dependencies ... All these dependencies that everybody worried about just melted away because we could [do it in object-oriented development] in any order … Even today, people don't believe that.
The Originators’ interest in object-oriented development led some of them to meet at the OOPSLA Conference and to start forming a community.
[4]: A lot of the XP people came out of the object-oriented movement so there was a center for them, their networking was at OOPSLA – the object-oriented conference.
Patterns were also recognized as a topic that shaped the Originators’ orientation and ways of thinking, as the following quotes indicate:
[6]: So, the whole movement in software in the 80s and 90s developed around pattern thinking….
The following quote connects object-oriented development and patterns:
[9] At the second OOPSLA, … I think, the second or the third, we took over a workshop, which was a workshop on object design. We just walked in and said: Here is how it is done with patterns, and we just showed off and … we were confident. Everybody at the workshop, maybe thirty people, they just listened and said: That sounds about right. And so, with that confidence, at that workshop at OOPSlA, we started to build a community to talk about how patterns could work. … We had the first Pattern Languages of Programming conference, PLoP conference. … Patterns and code go together, and we were still focusing on the code … We said: We are doing things differently. …The design patterns book came out at the same time and design pattern was a bigger success than any of the agile books. … So there were patterns, there were objects first and people started to understand patters.
The actual application of Smalltalk, object-oriented development and patterns in software projects yielded new development habits and human interaction styles. Some of these new concepts were later to be grouped under the umbrella of agile development, as the following quotes indicate:
[4]: The first thing I remember was not called agile at the time, in the early 1990s. I worked on a project called rapid development at that time. … One week iteration … with the client and … we published the first article on that in 1993.
[5]: We did not have the word agile at the time, not even in anybody's mind. But there were patterns of behavior of successful projects … It was 94, 95, 96, so at about that time I … would talk about trying to use a lightweight process, lightening the process, and I had the term surviving object-oriented projects.
[9]: We were doing programs in cycles or something …without even knowing how to do that … But we felt it was very important … using our respective knowledge and sewing it together in a way that was constructive as opposed to argumentative. … So we had this sort of cycle, we did an idea, and we explored it for a day, and if we were making progress, we put another day into it, and on about the third day, we actually did refactoring, and cleaned out the experiments that we did, to look really clean. … I came up with that entire methodology, you just write notes on cards. And the nice thing about the cards is that instead of having these complex diagrams that hold everything still, you can grab a thing and tell a story. … And by consciously not planning ahead, it gave us the ability to think freely about whatever we felt at that moment.
The Turning Point: From Technology-Driven to Cultural-Driven Leaders
OOPSLA, PLoP as well as other technology-focused meetings, were among the gatherings in which the originators of the Agile Manifesto started forming a community. As the following quote indicates, the peak of this sequence of meetings was the 2001 meeting in which the word agile was coined and the Agile Manifesto formulated:
[5]: The 2001 meeting was the end of a decade of a work by all of us, that was not the beginning of something or the end of something, it was the middle of something. … It is the end of the story; not the beginning of the story.
This section focuses on the 2001 meeting. It examines the importance and influence of capturing the newly emerged ideas presented in the previous section with one single term, i.e., agile. We also suggest why the Agile Manifesto was formulated using human-orientated terminology and why it was the human-related aspect of the Agile Manifesto that was attractive to the software community at that time.
The importance of the 2001 meeting is derived from the fact that it established a community with a name, a manifesto and leaders, as the following quotes show:
[4]: We got together in February 2001 in Snowbird. We were representatives from [different methods] and we knew that we were all these individuals, but we wanted an umbrella terminology.
[11]: The mission of the thing, besides giving it a name, was to answer the question: What do all these different approaches have in common?
[2]: I said, let's have this meeting, but I didn't want to have a meeting where we all get together and talk, but that should be a work-out, and that work-out turned out to be a Manifesto.
In practice, a name other than lightweight was sought for that commonality:
[8]: Lightweight in American English has a connotation of not serious, stupid, not capable of doing real things, and it was thought that it just was not a good brand, and so Agile was explicitly chosen to be more of an appealing brand than Lightweight.
[4] Till that point, the word that was used quite often was Lightweight Methodologies, and we didn't like that, so we put a bunch of words on the board, … there were 15-20 different words and we sort of systematically analyzed these words and decided that Agile was the most appropriate.
[6] I can't remember the specific words, … but Agile was one of then and we settled down that Agile is the word. This is how the Agile word came out.
Though the term agile was chosen in a democratic process, it had some drawbacks:
[11]: I think Agile is a weak name. No one doesn't want to be agile, so there is no cost for people claiming agility and acting however they want to act. There is no accountability in the term. My prediction at the time was that the term would become so watered down that it would cease to mean anything. I expected that everyone would say they were agile and that they would behave in many ways exactly as they always had. Some people and organizations have made dramatic changes, but in general the term agile does not hold a lot of meaning. When you visit an "agile" company, their values and/or practices can be all over the map.
After the word agile was coined and the Agile Manifesto was published, software practitioners could feel part of a community that was searching for a new development method and explicitly indicate, using a specific name, what their development process was. Furthermore, people could sign in to the Agile Manifesto website (http://www.agilemanifesto.org/sign/display.cgi) and express their support; this option further enlarged the evolving community, as can be observed from the following quotes:
[8]: What the Manifesto did was it gave a name that lots of people who stylistically tended to that direction … it gave a document, which is only page long, that people could look at and say: Yes, this is what I mean and that they can sign their name to… I think that the signing has a lot of affect.
[6] Now, what happened in 2001, by getting the 17 project leaders together and publishing [the manifesto], that all of a sudden, there were hundreds of paper and thousands of book and so all of a sudden now we had a marketing machine for agile in general that exploded it out much faster.
[5] What we were describing was what so many people were doing or wanted to do anyway. So they came to the website and signed their name on the website, so there were hundreds of people coming and signing up. It is not that we pushed anybody, so many people wanted it, the moment it came out they said: Yes – that's what I wanted to say all along.
While the Agile Manifesto reflected a feeling of some entity, practitioners desired to join a real community. This need led to the establishment of the Agile Alliance and then to the organization of the first agile conference.
[7]: In 2002 … we were thinking: All these people want to push the Agile Manifesto, they want to join us … join what? There is nothing to join. And I … said, why don't we start something, we can call it something like the Agile Alliance, and that will be something that people can join and feel they are part of. … And the first agile conference [was organized].
Interestingly, the Originators did not anticipate the huge response to the Manifesto:
[5]: We predicted much less than what we got.… The way in which people came, leaping out to join, was a big surprise and I don't think that any of us would have predicted it would be so common five years later.
How can this gap between the modest expectations of the Originators with respect to the impact of the Agile Manifesto and its actual attractiveness be explained? We hypothesize that this gap results from the different perspectives regarding the Agile Manifesto taken by the Originators and the software community. Specifically, while the Originators were driven by technological forces (as described in Section 3.1) and therefore addressed the Agile Manifesto mainly from a technological perspective, the software industry was captured by the cultural message conveyed by the Agile Manifesto that fitted the industry’s needs at that time. We should recall that the Agile Manifesto was published just after several meaningful events had taken place in the software industry – the Millennium bug, the hi-tech bubble burst in 2001 and the dissatisfied customers, as the following quote indicates:
[6]: By 2000, the military had shown that out of a sample of 35 billion dollars that were in projects, 75% of the money was a total, complete waste, absolute failure that have never used. Those are the kind of failure rates that you get with traditional project planning, on big projects.
Therefore, it makes sense to assume that the software industry just needed some new message to hang onto, and the message delivered by the Agile Manifesto fitted very well.
Indeed, what remains to be explained is the gap between the (technological) intention of the Originators and the actual people-oriented formulation of the Agile Manifesto. This gap can be explained in several ways. First, human-oriented terminology could encapsulate in just a few sentences the very spirit of what was common to all of the Originators, a commonality that presumably could not be captured using technological terms. Second, when the application of agile practices began (even before the term was coined, as showed in Section 3.1), the awareness to the human aspects of software development started to evolve and eventually influenced the formulation of the Agile Manifesto as well, as the following quote indicates.
[6] There is a huge background there … organizational theories, psychology, motivation and … all this stuff all coming into computer science.
The rapid spread of the term agile, as well as the fact that the originators of the Agile Manifesto did not plan to lead a cultural change, can explain why no follow-up meetings took place, but one:
[12]: It was a manifesto, a set of thoughts and philosophies. It was 17 people sitting in a room for a couple of days sharing experience. And, after it was published, that was that.
[8]: There was a second meeting of the agile, of this group of people, minus some people, plus some other people, a year and a half later … The agreement was made that this was it, we can not do anything more, we raised the sail and we should just let it go.
After the Creation of the Agile Manifesto: Cultural-Driven Forces
The originators of the Agile Manifesto became leaders of the agile community (there are, of course, new leaders and some of the Originators are not part of the leading group any more). However, as described above, while their original intention was technological-driven, today they are leaders of cultural change, as they themselves admit:
[5] I am in the middle, actually all of us are in the middle, of doing organizational development, because we are trying to do agile development, but it requires changes in the team structuring, reporting, conversational habits, process, any of these things. And so all of us are sort of ex-programmers … found ourselves in the middle as organizational consultants.
[11] We have found the combination of technical and social skills very effective in workshops and consulting.
Conclusion
It is usually assumed that agile software development evolved as a response to the poor results of software projects and that a realistic and conscious decision was taken to change the paradigm of software project management. The story told in this paper reveals that this assertion might encapsulate only part of the story of the emergence of agile software development. The paper highlights another aspect of this story, according to which, technological forces fostered cultural changes. Indeed, a quick look at the program of any agile conference will reveal that agile software development is a mix of cultural and technological ideas.
Thus, the originators of the Agile Manifesto can be viewed as change agents who triggered a process of cultural change. It should be kept in mind though, that had the software community not been starving at that time for change in its values and norms and in need for both technological and behavioral change in its development processes, the agile paradigm shift could never have taken place.
In this context, in conclusion, we pose several questions that highlight the essence of this paper: What kind of agile values and principles are required to initiate a cultural change? Should they be technological-oriented or cultural-oriented? Specifically, if we feel that a cultural change is needed in our organization, is a technological change a good strategy by which to initiate the change? And in the context of large organizations with a strong culture of planning and centralized power, what agile methods, practices and tools are required to launch a major change?
Though one answer cannot be given for all organizations, we propose that the two orientations (technological and cultural ones) should be considered and examined when a change is considered; the exact dose of each of them, however, should be determined according to the needs, status and culture of the said organization. One option is to start with a technology-oriented change (with practices such as short iterations and measures), and in parallel, to start increasing the attention to cultural issues, for example, by the facilitation of periodical reflective processes in which the human and cultural aspects will surface with respect to the technological changes. One main implication of this perspective is that no matter what orientation (technological or cultural) an organization chooses as the first stage for its agile adoption process, it will not be able to neglect the other aspect, since the two are tightly connected. The mere awareness to this fact, we suggest, should guide agile assimilation processes. However, as mentioned before, the exact dose of each orientation along the adoption process should be examined carefully based on the analysis of the characteristics of the said organization in each stage of the agile implementation process.
Acknowledgment
We are deeply thankful to Kent Beck, Alistair Cockburn, Ron Jeffries, Brian Marick, Bob Martin, Jeff Sutherland and the other originators of the Agile Manifesto who shared with us their thoughts and reflections on the emergence of agile software development.
About the Authors
Dr. Orit Hazzan is an associate professor at the Department of Education in Technology and Science of the Technion – Israel Institute of Technology. In May 2004 she published her book Human Aspects of Software Engineering, co-authored with the late Jim Tomayko. Her second book – Agile Software Engineering – co-authored with Yael Dubinsky, was published by Springer in 2008. In parallel to her research work, she is a consultant for several software projects in the Israeli software industry. She presents her research at computer science and software engineering education conferences (e.g., SIGCSE), as well as at conferences on software engineering in general (such as, ICSE) and agile software development in particular (e.g., the XP and the Agile Conferences).
Dr. Tali Seger is the Head of Management and Organizational Development Program, in the MBA Studies at the Business School of Management, Ruppin Academic Center Israel, Hemek Hefer Israel. In the past 1989-1998 she worked as Human Resources Manager, Member Corporate Mgmt in Communication Companies. Her teaching and Research areas: Organizational Psychology, Emotions in Organizations, Group and teams psychodynamics, Strategic Human Resource Management.
Dr. Gil Luria is a lecturer at the University of Haifa, the faculty of Social Welfare and Health Sciences at the department of Human Services. He conducts research on organizational climate and culture. He received his PhD from the Faculty of Management at the Technion- Israel Institute of Technology. Gil has conducted research projects in a variety of organization, from leading manufacturing and military organizations to small factories. These studies were published in scientific and professional journals, and were adopted by practitioners in several organizations as applied tools for safety management. His recent work focuses on the relationship between leadership and climate, and on a new model of culture as leverage for change.
Appendix – The Interview with the Originators of the Agile Manifesto
1. Breaking a paradigm in software development
1.1. What was the first stage you recall with respect to agile software development? What was especially important for you?
1.2. How was the term "agile" conceived?
1.3. Please describe the group work process. How many times did you meet? What were the meetings like? Was there a formal manager for the process? Please describe the atmosphere at the meetings.
1.4. How long did it take, from the day the idea was conceived till it was first implemented?
1.5. Who was the first person to implement it (other than the 17 original originators/people)? Who did you approach? What characterized the organizations that cooperated with you?
1.6. What points of resistance were raised by organizations? And who in the organizations raised them? How did you deal with such resistance?
1.7. Were you assisted by organizational/managerial consultancy during the thinking phase in order to outline the path and implement the approach throughout the world?
1.8. What were your expectations regarding the direction that agile development would take? If you had been asked in the year 2000 about the extent of acceptance of the agile approach, what would you have predicted?
1.9. What crises did you face during the process? Did you think about possible failures? If you did, what contributed to your positive coping?
2. Professional perception
2.1. What highlights and pitfalls did you experience at the beginning of your professional career (as far as you can remember)?
2.2. Can you describe the development of your networking? What is your professional networking like? Meetings? Correspondences? In your opinion, whom is it important to stay in contact with?
2.3. Assuming that you worked with organizational consultants, what is your opinion with respect to this profession? Have you implemented work and management methods following your contact with organizational consultants?
3. Student life
3.1. Did you know, a priori, what professional domain you would study at university or did you have doubts? If you had – what were they? If you could not have studied a software engineering-related subject, what would you have chosen to study?
3.2. What characterized you as a student? What courses did you especially enjoy?
3.3. In your opinion, how did your fellow students describe you? What was your style back then? What was unique in your behavior as a student?
3.4. How did you prefer to study? Alone? In a team?
4. Family background
4.1. Where (country) were you born? City/town/suburb/village? Urban/rural/ periphery/central?
4.2. What did you like to play with as a child? What were your hobbies? What was your favorite sport? In what framework, if at all, did you play (e.g., sports club etc.)?
4.3. What were your parents' occupations when you were a child?
4.4. In what ways are you similar to your parents? Who, in your opinion, influenced you as a person? What did you like about them? What characteristic was the most appealing to you?
4.5. What values do you remember being emphasized in your home as a child?
[1] Source: http://www.versionone.com/pdf/AgileMyths_BetterSoftware.pdf
