Right, first of all let me define my terms because I am not sure all of the audience will be aware. So what we mean by component teams is groups of programmers that just specialize in one software subsystem. For example, a really simple example, a group of programmers that just do the UI layer, a group of programmers that do some middle tier, a group of programmers that do database stuff, or in my world, the larger scale when you got millions and millions of lines of code and lots and lots of subsystems, programmers in this subsystem, programmers in this subsystem, and then when you want to do a customer feature it needs to be split up between all of these component teams. So that’s what we mean by a component team.
What a lot of people don’t understand about component teams is what it leads to if you have a component team, the organizational implications so to paint a picture of it with words, if you have a requirement F1 and it has to cut across five components and therefor across five component teams, the first question you need to ask is who is going to do the requirements analysis for it. And if you have got a hundred component teams it’s not at all clear which of those hundred will do the requirement analysis, and they’re programmers anyways, so we give it to somebody else, call it the requirements analyst or the business analyst or the specification group. And they do some kind of requirements document.
Now, if you have a hundred components and a hundred component teams the next question you need to answer is which of these hundred components is touched by F1 and what are the interfaces between the components involved. Now which of the hundred component teams is going to answer that question? It’s also not at all clear and so we’ll create an upfront group to answer that question usually called systems engineering or the architecture group. And they’ll take as input something created by the requirements group and then they will create some kind of an output describing work to do in each component, the components needed, perhaps the interfaces between the components and so forth. Then we are at a step that the programming can finally happen and so then the work is given to the five different component teams and they do their bit of the work.
But it’s important to understand that the way I described the story almost sounds like everyone is just sitting there waiting for F1 to come to them but in fact there’s F2, F3, F4, F5, and all of the work has been chopped up and it’s either in queues before each different work group, or they are doing work out of order, or they are partially allocated across multiple features at the same time. So the work is being done out of order and if you’ve got for example five component teams working on F1 one of those teams might do their part in January and another team might do their part in June, because it’s all out of order. Finally all of the programmers have finished their work on the components and we need to do the end to end testing for F1. Who is going to do that? Will it be the first component team? The forth component team? It’s kind of arbitrary and not at all clear so we’ll give it to a separate group called testing, or systems testing.
So what do we have here? We have a group that does requirement analysis, that hands off their work to a group that does architecture and design, that hands off their work to a group that does programming that hands off their work to a group that does the final testing. In other words a sequential life cycle, so it’s important to understand that if you have component teams it inevitably generates a sequential life cycle with all kinds of levels of hand off, delay, multi-tasking, and the myriad wastes in Lean Thinking of over production and so forth. Like for example, one of the wastes in Lean is work in progress and over production of features or inventory and so forth.
So, if you are a group of just systems engineers or a group of just requirements analysts it’s not like you are just going to sit there and not do things, you have to keep busy, that’s from this local optimization view of the important thing. So, when you’ve got these single specialist groups that are driven out of having component teams everyone is busy creating documents and content, or work in progress, for the next group as well. So you get very high levels of WIP. Also, let’s suppose you have got a hundred component teams, component team five has to keep busy doing component team five work irrespective of the actual value for the customer prospective. So what happens is that you optimize the choice of requirements not around value to customer but around keeping people busy.
To put it in Scrum terminology, you may take some very low priority item and start working on it because otherwise component team five wouldn’t be busy. So, you don’t have the case then when everyone is working on the high value stuff, that’s another negative that comes out of this. Furthermore, you’ve got F1 chopped up across all these different groups so you could drop the ball on it because it’s not being done by one group end-to-end and consequently we need to solve that problem somehow which is usually solved by adding an overhead role called the project manager or feature manager. Then this role is responsible for keeping their eye across this feature as it goes across everyone so we’ve added more unnecessary overhead, more coordination problems, sub-optimization on keeping people busy, more WIP and a very slow organizational design that all arises out of the assumption that programmers could only possibly work on one little chunk of code that it wouldn’t possibly be useful or skillful or even possible for programmers to work on two different things.
And to me it’s fascinating how this extraordinary large, inefficient, organizational design, essentially drives from that key assumption, the mental model that programmers can only work on one little piece of code, which is fallacious, it’s not at all true. So the alternative is to move to Agile teams, Scrum teams and in Scrum or in any Agile team, the idea is that the team is what was traditionally called a feature team, that this team is cross-functional, cross-component, and they contain all the skills within them to do everything end to end. When we are talking about changing the organizational design from say a traditional sequential life-cycle with component teams, to a large scale Scrum model, what it means then is that the people in this different single function departments, requirements, systems engineering, programming teams, test - all of them will leave those groups and they will form new groups, team blue, team red, with people from these different single specialist groups and ideally with programmers in a team across different components.
So that then within the team you’ve got people with skills essentially across this stack so that then there is cross functional and cross-component feature team that can be given end to end work, they can be given the goal F1 and they can do the requirements analysis, the design, the programming and the end to end testing as well. Now there is a scaling question, which is that that kind of transformation works relatively simply and straightforwardly with little groups of let’s say five or ten component teams moving to cross-component teams. But in the world that I work in which is a hundred or two hundred teams your’re not going to be able to create teams of seven plus or minus two people, who know the code across the entire stack of all fifty million lines of code. So how can we address that problem?
Well, a solution from large scale Scrum framework 2 is to have what are called requirement areas, which are areas of requirements organized from the customer prospective like transaction printing, or color printing, or protocol support, or options trading, or whatever - it’s a requirement from the customer area. And then have groups of feature teams that just specialize in that area. Now that area is not necessarily isolated to a particular subset of code, but odds are that a requirements area, which is a requirement from a customer prospective, nevertheless tends to work in a particular subset of the total body of code so this tends to reduce the learning burden so that then a set of feature teams working in one requirement area, are more likely to still be able to work an end to end feature with enough knowledge within it.
Then, if they don’t, then they’re just going to have to face the learning challenge of starting to work in components that they know less about. And if that’s the situation, than we need to address the problem of the team essentially working on code they might not know very well and then we can use other techniques to address that potential risk, such as a component guardian, someone from another team who knows that component well, and looks at their code and participates in a design workshop with them in order to help them.
Another related transition technique for moving from an organization with a lot of components to cross-component feature teams is a baby step where instead of moving to truly cross-component teams you start by making teams that just work across a larger set of the stack, but not across the entire thing.
So suppose for example you had fifty component teams and the fifty component teams were organized into ten teams in layer one, ten teams in layer two, all the way through layer five. Then, in layer five your original design could be that one of the teams just worked in that little bit and this team worked in this little bit. What you can do in layer five is that you can mix up those component teams to create teams with cross component members but they only work in layer five, but you can mix them up so at least each team can do anything in layer five, and so too with layer four, and layer three, and layer two. And then when you have to do a feature like F1 that cuts across layer one to layer five, you’ll still have the problem of hand off and multiple programming teams that are only doing part of the work. But you’ll have reduced the number of hand off from maybe twenty different teams to now only five different teams, which has benefits as well.
Then maybe over time you can go from having teams that are just specialized into a layer where for example you take teams that are just in layer five and teams that are in layer four, and re-mix them so now that you’ve got a team that knows, some of the people know stuff in layer four and some of the people know stuff in layer five, and now you’ve got teams that are a little bit more flexible. And in some of my customers that are very large scale, we take that kind of baby step as well. I think that’s kind of an introduction to the component - that’s the short answer to the component team issue.
Yes, I am going to generalize the answer first of all. First specifically the usual mistake with component guardians is that they have check-in authority related to the code that other people are adding in other words they are a committer role, to use open source terminology. And you don’t normally want that unless it’s like a safety critical issue, where you want to optimize the system on life safety and you are willing to sub optimize the delivery of value. My context is always that we are optimizing on the delivery of value to the customer that is higher priority than life safety and other things, so my answer is in that context. So in that context, if delivering value fast is your optimizing goal, then if you have any kind of a process where a person acts as a gate, you will introduce all kinds of delays and problems, and the generalizing comment that I want to make is that if you have any kind of processes where somebody creates something, then there is a delay of time, and then at the end there is a gate and a quality inspection before something can go through that gate, we have a problem.
In some quality communities you’ll sometimes hear the phrase: "Quality gates imply a failure of quality". And whenever you install quality gates, inspection steps and so forth, you’re not actually addressing the route cause, you’re just putting a Band-Aid on the situation, because the idea in Lean is to build quality in so that you wouldn’t need a quality gate to try to inspect and correct defects out. In Lean one of the wastes is defects and then all of the knock-on effects from defects like inspecting and correcting. So reviewing a document and then handing it back, testing code, and then it failing the test and handing it back, all of those are part of the wastes in Lean Thinking, the wastes of defects - better to build quality in.
So instead of a component guardian acting as a committer who waits for someone to send them the final code, so that they can review it and then check it in, better to build quality in with that component guardian as a component teacher and they go proactively ahead of time and work with the team that’s going to be working in the component they know best about, participate in the design workshop, or pair programming, or in that way help educate someone on the team, to know as much as they know, so that their job as a component guardian is no longer needed.
In other words, they are always trying to work themselves out of a role, through continually teaching. It reminds me of a story. Some years ago I was working in Paris at a bank called Society General and they were trying to do a Scrum adoption in a particular sub-group. And one of the mangers told me a story: he told me that when the new CIO of Society General joined the company, on his first day, he asked his direct reports "Please bring me the name of one person in each of your groups which you consider absolutely critical for delivering your systems into production". And on the second day his direct reporting managers gave him the names of these super critical people. And then the CIO said: "These people will not be allowed to work anymore". And this is such a rare example of a leader with real systems thinking insight, he understood that those people aren’t your strength they’re your weakness.
In Canada we would call it the truck number of your project - how many people have to be hit by a truck before you are in trouble. In London it’s called the key man problem. And so the CIO said that after this point, these people, all they can do is teach other people. I want them and only working with others and acting in roles as teachers. So it’s the same thing with good component guardians versus dysfunctional component guardians.
Yes, absolutely. I think, first of all, I’ll answer the question in the context of what I call the Fourier programmer problem in outsourcing to off-shore locations. I haven’t worked in every country in the world but I’ve seen this for example in India and China. And I’m not really trying to pick on the country per se, these dynamics could happen in Canada where I come from, they just didn’t happen to. So in India, for example, it’s quite common that university professors, smart as they are, and well educated as they are, know close to zero about the craftsmanship of programming. They really don’t know how to work as professional programmers and so, "so called" computer science students coming out of the Indian or Chinese universities they learn very little, or close to nothing, about how to be good professional programmers in those countries. And it’s not really much better in Canada where I come from. But maybe a little bit better.
So they come out of the university say in India knowing squat about how to be a programmer then their goal is to stop being a programmer as fast as possible and they call this the four year programmer problem. So in like China or India "Ok, well I am willing to be a programmer for four years but then what I really want to do is become part of the overhead management" because that sounds better and I can make more money and so forth. That’s perfectly rational, you know if that’s the only way you can make more money. But what it leads to then is organizations in these countries, outsourcing organizations full of people with quite low programming skill. And furthermore, it isn’t a culture that values clean code or good code and it’s not a culture of master programmers.
This can happen in any country but it tends to be even exaggerated worse in these offshore outsourcing contexts. And so the role of a component teacher, say to play it out with a concrete scenario - we’ve got a bank in London they’ve got developers inside of the bank who really know the code well, they’ve been there for five - ten years, they know their stuff. And then they’re required by some HR policy or some CIO policy "You must work with outsourcers because we will save money that way" which in itself is really highly fallacious and you can unpack that whole assumption and it’s often not true that the value for money it’s not there, but then leaving that aside. So they are forced to take on working with an outsourcing group in the probably mistaken belief that it’s actually going to save people money.
Then, in that case the programmers they are working with are un-likely to produce great code. The first thing that I recommend when you are choosing an outsourcing partner for an Agile engagement or any engagement is pay no attention to what the sales people in the outsourcing group say about the quality of the people. Pay no attention to their claims about the CMMI certification level, that is all just bla-bla-bla. Look at their code. I have told these story many times before, Valtech where I used to serve as Chief Scientist, was considering buying a company that had CMM level four certification, and we decided to actually do a project with them and then rather than listen to their claims actually look at their code. So we did a project with them, where essentially we were the customer, (I am not even sure they even knew we were considering of buying them at that time, I don’t think they were) and then we looked at the code they produced and it was utter crap.
So we didn’t buy them, but the point is that you have to go see - as they say in Lean thinking - go see, look with your own eyes, spend time sitting with the real development teams there, spend weeks working with them, see what they are really like, and look with your own eyes in depth of the quality of code they create, and choose your programming partners based on this direct insight rather than any indirect claims from say sales people or assessments or ISO levels, or what have you. And then thereafter, when you find a group of great talented programmers try to keep them forever. Pay them whatever you need to pay them to keep them as programmers for twenty or thirty years. a culture of paying people a lot of money to be master programmers. And try to develop a culture in these offshore development organizations where they start to value master programmers themselves rather than establishing a culture of promoting people out of the value work into overhead management roles.
While you are working with the programmers, have your local component guardian and your local on-shore programmers establish a role of coaching and mentoring with them. Now I’ve noticed that sometimes this leads to turf problems because the on-shore people, let’s take the case of London, they will feel probably threatened by potential loss of job security and so forth by these new people coming on and they might not want to help them as teachers because they might feel like they are working themselves out of their job. One of the key Lean ideas is job safety but not role safety so the idea in Lean is that there is no way that we can deeply and dramatically improve the organizational system unless we can deeply change the organizational design which implies that roles may disappear for example like in Scrum we no longer need the overhead role of project managers and program managers.
But all of these good people in project management or lead technical people, out of respect for them and in order to get real improvement we need to have job safety for them. So we need to somehow communicate and have the policies that while you are acting as a teacher to these people you won’t be working yourself out of the job other value adding roles will arise or exist for you but what we value is teaching. It reminds me some years ago when I was in Japan I went there with my co-author Bas Vodde and did some interviews at Toyota in Nagoya and Toyota City. And we interviewed some Toyota people about their HR policies and one of the things that we found is that in Toyota they actually measure people upon the question "How much time are you teaching other people?" and they especially measure managers upon this. So there is this culture in Toyota of manager as teachers.
The saying in Toyota is "My manager can do my job better than me" and a key job of managers is continue to act as teachers both of Lean thinking skills as well as the hands-on work whatever it is. And so the Lean culture is a culture that really values people, senior people managers who are acting as teachers and I would like to encourage that in when we start working with off-shore teams as well.
So, first of all to state the obvious, you’ve just described the situation that typically exists and the mentality is "Oh, we are going to have the important people, the management overhead and so forth in Rome and then we’ll have the unimportant people, the coders, in Shanghai", it’s that kind of mentality. Before I answer your question directly, I want to go to a geo-political level. You know you can outsource the management too. And in fact as groups that master Scrum and Lean and Agile practices well know, you need almost no overhead management, it’s discovered to be truly a wasteful job, you have self managing teams which work quite effectively.
And here is another point: inevitably if you set up a site somewhere that does the hands on value work, and I’ve seen this all around the world, inevitably management moves there, because the bottom line is that there is where the value is being added and one way or another that’s what happens. So, let’s play this out into the future: we’re in Italy, I am painting a picture, and we have the mindset that the important work is the overhead management role and we create organizations where nobody has the skills to do the hands-on value work and all we can do is do Microsoft project and so on and so forth.
Contract negotiation and all that stuff. So, play this out ten, twenty, thirty years in the future, what’s going to happen? People aren’t going to have any jobs in these countries anymore because the bottom line is where the value is being created. And simply what will happen is that all of the value work will shift to China, India and so on, and there won’t be people in first world countries who are acting as hands-on value creators, hands-on engineers, designers, scientists, what have you. And I don’t mean to imply that we shouldn’t encourage success in developing nations, it’s a big pie, potentially growing pie. And I’ve got kids that have grown up now and if ever you go to a poor country and you see little kids on the street who are not advantaged, as a parent you naturally hope to see them have a better life. And I want to see developing nations succeed economically.
But I don’t think that needs to be at the expense of existing developed first world countries, but it will be at the expense of developing first world countries if we promote this culture of overhead management and people who do hand-off in these countries and engineering work as being in other countries. We need to establish a culture of hands-on great engineers, scientist, designers in all countries. And the world is becoming flat. And in this highly competitive world where we have to shave off all the waste and all of the inefficiencies, we can no longer stand, in this highly competitive world, a world that can afford a lot of overhead management. Self managing teams and self management, which is the eleventh Agile principle, is not just a nice to have idea in Agile methods, it’s going to be a critical requirement to stay competitive in the future as well.
I know I’ve kind of taken a bit of a tangent on your answer but I think it’s the larger picture that needs to be communicated. So your original question was how do you deal with organizations where I am seeing that kind of problem? And one of the things I first do is I talk about this very point. I play it out over time and then I tell stories of some companies I’ve worked with where I’ve seen the shift. And then the gradual reduction in the first world country sites, because I work with companies over multiple years, and so I see the shift over time. When companies first start this they have the fantasy that "Oh, we’ll be able to have lots of big expensive important jobs here in London forever".
But I look out over five or ten years with my customers who are doing this and it’s very easy to see the trend that that’s not what’s going to happen. And so I normally start by sharing that and then ask the question, how can we do things different, how can we move to a culture where everyone is doing hands-on work? And are you part of the next generation of becoming a skilled value worker? Or are your only skills in overhead management?
I often have a filter. I get a fair number of requests for work and if it’s less than five hundred people per product group, I don’t usually take the work anymore. Just to manage the work.
I am not sure I can exactly rank them but the big ones that come up repeatedly are number one what in Lean is called "The waste of working to job title". In large organizations the whole mindset is you’re a systems engineer that’s all you do, you’re a programmer in component three, that’s all you do, you’re a project manager that’s all you do. And the mistake or the challenge in these large-scale adoptions is breaking that "waste of working the job title" mentality. And it’s not just an individual person, the whole organizational structure is wrapped around that. You have the business analysis department, you’ve got the systems engineering department, you’ve got the testing department and so dissolving all of those groups have implications on the existing management structure and job careers. And breaking people out of that mentality is a real challenge.
Even if you start to form in these organizations an ostensibly cross-functional, cross-component team of seven plus or minus two people, usually the mistake that happens, just focusing now on the team level, is a requirements comes in and the person previously known as the tester says "Well there is nothing I can do for the next seven days because there is no testing tasks yet". Or the person previously known as the business analyst says "Well I’ve finished doing the business analysis so there is nothing more for me to do". And so then a very common mistake is that they increase their WIP level, their work in progress levels, by then the business analyst going on to working on the analysis for the second item while the other people are working on the programming for the first item, and so on and so forth, instead of whole team together with multi-skilling taking on secondary or tertiary skills and learning to do one thing at the time and working outside of their comfort zone.
It reminds me of an interesting experiment that we’ve talked about in one of our books, there was a group in the United States that was applying an Agile development method, I think it was XP I don’t think it was Scrum, not that it really matters in the context of this story, and they measured their velocity over a series of iterations where they applied the policy of most qualified person where the person most qualified for a task always chose it and they measured their velocity. Then after X number of iterations, they flipped the policy to least qualified person where on purpose the person least qualified to take on a task always chose it. After three iterations their velocity was higher than it had ever been before, which always strikes me as a fascinating story. And I don’t mean to imply that other groups’ results are going to be the same but it’s a very telling story about the assumptions that people have.
It’s related to the assumption that the important thing is optimizing how busy a person is or optimizing how fast one person works which is not the Lean view. The Lean view is how fast can we deliver an entire thing from the system, how fast can we deliver value to the customer? And from that prospective, hand off delay or bottle necking with single specialization is often a dominant issue, and even if people can do things slower but they can do three things, you’ll deliver value faster and from a Lean prospective that’s what’s important. Another big, I’ll say mistake or challenge, in these large organizational adoptions is that they’ll typically have a large amount of management overhead: project managers, program managers and they even sometimes have a project management office.
They mistakenly believe that you can adopt Scrum and still have all of this overhead management, which of course doesn’t work at all. If you write down a list of all of the project management responsibilities, just make a list, and then identify the three roles of Scrum: Scrum Master, Team, Product Owner, and you take that long list of all of the project management responsibilities, and say which of the three roles they going to. Fact, they will find a natural place into those three roles, most of them will go into product owner and into team. So the point is in Scrum, virtually all project management responsibilities in proper Scrum are allocated to the product owner from business or to the self managing team. Which therefore means that if you still have around the old traditional project or program manager role, it’s going to gum up the works, because what are they going to do?
They are going to interfere with the proper Scrum model and we won’t have self managing teams. And what are they going to do? What do managers do? They have to manage and so it will create various kinds of problems - they’ll be taking on the things that the Scrum team should have been taking on. And this is a delicate political issue, because these are often very bright people, talented, they’ve put in years of hard working time in the companies and it’s not like I’m trying to imply that I don’t value their native intelligence or their contribution, but it’s back to this Lean principle of job safety versus role safety. The role of project management and program management is simply no longer needed in large scale Scrum organizations.
The leadership that’s planning this transition needs to address this and find new work for these people. Which, by the way, is why for large scale adoptions bottom up adoptions don’t work because we are talking about deep changes in the organizational design and that’s going to require the decision making by leadership in changing of roles, new work for people in traditional project management roles and so forth.
The very quick answer is I don’t know. That’s painting it a little black and white, but I’ve not thought about this question very formally so I can only kind of answer it anecdotally, rather than with a lot of careful thought and reflection. The first thing that comes to my mind is that the people in my life that have had the greatest influence on me, was primarily from their behavior. And I’ve been fortunate to work in my life sometimes with some people that I consider just freaking great people in product development and software and so on and so forth. And then a corollary from that is that if we want to create teams that demonstrate learning, than we the people that want to inculcate that kind of behavior I think will need to demonstrate that in ourselves and in our own behavior.
So, if there is some core set of what I’ll call organic leaders in organizations, Scrum masters, natural leaders who have been around for a long time, managers in the organization, and so forth, have a discussion at this level that we all need to be demonstrating multi-learning which is a key Scrum idea, rather than just a single specialization. That we’re taking time during the days to read books, that we’ve got piles of books on our desks, that we’re going to courses regularly we’re going to conferences, we’re reading, we’re starting brown bag lunch sessions in which we are going through a book and inviting other people to listen with us, that at five pm each day we are organizing sessions where we invite people to come and we show them what we’ve learned, and then in that way I think we might be able to give the spark of learning to other people.
That’s one thing that comes to mind. Another thing that comes to mind is that if you’re a group of seven testers together in a room and your manager is a tester, well what are you going to learn about? Testing. And you are a group of five programmers in a component team on module four, and all you do is work on the seven thousand lines of code in module four for three years, what are you going to learn about? So your opportunity for learning is quite bounded as well which can also then lead to the mindset of getting out of learning. So it’s important to notice how single function teams and component teams systemically set up the cultural and environment where people stop learning as well.
In addition, when you re-organize into cross functional teams and cross-component teams, so you and I are on a team together, you’re an expert in UI design and Groovy and I’m an expert in specification by example and acceptance test driven development. So, now I’ve got tremendous opportunity to learn from you and vice versa by virtue of the fact that we are in the same room together. And because in Scrum the only job title is team member, you’re no longer business analyst or tester or whatever, and then there is all kinds of tasks on the wall and I can pick something from my secondary specialty, or tertiary specialty.
Then through pair work with you, if that’s your primary specialty and I’m taking on a task that I know less about, there will be new opportunities for learning. So, what I’m trying to say here is that through the Scrum model we set up the organizational environment that creates the opportunities for learning, which don’t exist in traditional organizations as well.
Then another element that I’ve seen that can help is to formally ask the question with the team during a retrospective, "Team, we have some kind of imbalance of the team specialization to the requirements that we’re getting in, why don’t we formally identify secondarily specialties?" So in some groups it’s useful to actually say "Well, my primary skill is testing there’s no way that I’m going to be able to learn how to program it’s just outside of my scope. But I’m really interested in UI design and I’d like to take that as my secondary specialty, and I want to go on that course, and I want to get these books, and I’ll spend time with this person in this other team who’s a specialist in that area". And so in that way having these conversations formally about secondary specialties can also structurally set up the environment to make it happen.
So then a combination of organic leaders demonstrating learning with an environment that is designed to support learning, might provide the spark for people to actually learn.