In the blog post in defense of agile Nic Ferrier elaborates why he thinks that software production can fail due to experienced programmers who are so focused on programming that they can't see the world around them. According to him an anti-agile culture of only programming is not going to solve the problems that the software industry is having. Programmers and business people should invest time and energy to understand each other and work together to fix problems in software delivery.
InfoQ Interviewed Ferrier about deploying agile effectively, the purpose of stand-ups, on the need for managers or Scrum masters in agile and alternatives for sprints to deliver software, how focusing on architecture can improve collaboration and how technology can help us to avoid some of the organizational problems that teams experience.
InfoQ: You mentioned that there is no proof that agile works, but in stead there are many experiences of things that worked and didn't work. Are there ways that we can use that knowledge to deploy agile more effectively?
Ferrier: The big problem with changing work practices is that so few of us are prepared to publish our experiences. Few people in IT would claim to be un-scientific but the organization of IT departments is hard to study well because organizations don't like publishing the data.
There's a great book published recently called Leprechauns of Software Engineering and it recounts a lot of the stories that get held up in software like 10x productivity and explains where they came from. In each case there's no real evidence.
We don't know how to measure fundamental things like productivity of software developers because we don't even know what good programming is. So the thing to do is approach everything you do with a lot of criticism. Don't let it paralyse your work... but at least you can be critical and say "this doesn't seem the best way to do things".
The central tenet of agile to me is to retrospect fairly regularly. That's what let's you check whether you're getting better or not.
InfoQ: What if things are becoming worse, or when you're being hindered by something? Would deploying good practices be a solution to that?
Ferrier: Firstly you're in a good place if you know it's getting worse. At least you have measurement. I think to get out of getting worse or staying bad you keep trying experiments. What those experiments ought to be is open to question. But if you've got a software problem it probably makes no sense to try handcream on your developers. You know you've got to try a software type solution. A good place to look for those is the Internet, you can ask how other people solve problems that you have. That doesn't mean those solutions will work for you though, you have to be mindful of that and always prepared to try something new.
And remember there is no guarantee of success. But the best thing we know is to keep trying to fix things that you know are sub-optimal.
In his blog post Ferrier wrote the following about stand-ups:
People don't tend to start things because they are stupid though. There must be a reason for standups. There is. In a complex software project there are usually quite a lot of people doing a lot of different things. So it's good to catch up fairly frequently to exchange information. Why are these things held standing up? The original intention was to stop them going on for ages, because people don't like standing up.
InfoQ: What if teams don’t see the need to do stand-ups and start skipping them? Should Scrum masters or agile coaches insist that they have to be done? Is there an alternative?
Ferrier: A scrum master who insisted on something would be doing a really bad job. The definition of scrum master is "servant leader", so being dictatorial is right out.
You absolutely don't *need* to do standups. They exist as a way of doing a daily catchup with everybody involved. You could replace the need for that with a ton of other communication. I've done "standups" over chat, with a quick video call or just by having everyone religiously note down what they're doing in an shared editor.
But most organizations aren't capable of doing that because the comms systems aren't that good. In a startup a team might be able to use whatever they want, even the latest web rtc video chat tools. But that's rarely possible in a big company. They have constraints. So it's really unlikely that you'll be able to pull everyone you need in some comms system other than a meeting.
The key thing is to do it, to have the information exchange. The reason this is often a meeting of some form is because people don't do it unless they're made to. People go to meetings when they're told to. Well, mostly. But if you cajole people to do the information share it can work in other ways. Note that it's not just writing the information share that's important. You also have to READ it. Just as when you have a meeting it's not just giving your status update that's important but listening to everyone else's. That's why the meeting (when you have one) needs to be short.
Remember that a standup is not just the tech team. It's the product owner from the business. The test people. Any support people. Anyone involved in what you're doing. You've got to find a way to keep all those people involved in what you're doing, preferably everyday.
InfoQ: You stated “I do not believe managers are the answer to economic problems”. Can you elaborate what you mean with this?
Ferrier: One of Erik's suggestions was that we could make IT into a purely command and control situation. It's simple, bring in some managers to tell the programmers what to do and then they do it and then there's profit.
Except that is such an inanely naive view of how management works that it's laughable. Managers do tend to be command and control. Which stops them learning about the highly subtle production of software. You can't just say "I want a big gold cow" and have developers make it for you. That's the builder myth. Software isn't bricks and mortar or electronic. It's much more malleable. So you have to build it in a malleable way, with high adaptability to changing circumstance. Also called agility.
InfoQ: Is there still a need for management and managers in agile?
Ferrier: Not always. Fred George talks about programmer anarchy where programmers do what they think the right thing is. You can run a team like that when they are really good engineers (which means they know what they're doing AND they don't over engineer) AND they know the domain really well.
Most enterprises are just not brave enough to try this but they could do. They have people on the IT side who have tons of domain experience. Mostly I find the product ownership in enterprises is really immature and not understanding technology. Product owners too often come from the business side and see themselves delivering what they want, or what their boss wants or what the board wants; too often they think they're working on something called a "project"; there's little understanding of lean development or desire to learn about how to build better products. Just lots of preconceived ideas.
There is a possibility that Scrum masters drive wrong behavior in teams as Ferrier explained in his blog post:
You don't need a Scrum master for a team, in fact it actively discourages the team from looking after themselves. But in a situation where you have developers who have been tightly project managed day to day maybe a scrum master type who is more servant leader than project manager would help teams learn to work things out for themselves a bit.
InfoQ: I understand the potential problems when Scrum masters start managing too much and team member leave everything up to the Scrum master to organize. Would it be possible to find a balance, where Scrum masters and team members collaboratively take care of the organizational tasks?
Ferrier: Well, maybe. But it seems like in a really well performing team there wouldn't be anyone who wasn't bringing something more than just being the scrum master. Scrum masters kinda need to exist because people have been managed so much.
InfoQ: So there might be a need for Scrum masters initially when teams adopt agile ways of working, but in time they may become redundant?
Ferrier: That's it, absolutely. And that's one reason why agilists sometimes prefer Scrum masters to come from the team, to be technologists. Because then those people will naturally find a way to melt away their scrum mastering role. I think that doesn't always work though. But that's often because business managers won't let technical teams develop in an agile manner.
In his blog post Ferrier explained why he considers sprints to be bad solution for teams except the lowest agile maturity:
Sprints are utterly ridiculous ways to produce software. They always create a situation where you can't deliver fast enough, they encourage programmers to hoard code and to make source code branches and more and more busy work in the process.
InfoQ: What do you propose as an alternative for sprints to deliver software?
Ferrier: There's a very well known state, flow, where teams are able to pickup ideas from the product owner and just get them done. For that to happen there's a lot of stuff that needs to happen, the product owner has to be understanding the next smallest thing that could be done and have a good idea for testing how it can be proved to be working or not, the developers have to be doing very tight engineering, the test team have to be well down the path of discovery testing, lots of things have to be right. But teams do get there.
But let's be clear. I'm not proposing you try and walk before you can run. But I am saying if you're stuck on Scrum a year after you started that's no good. It's not an end state, it's agile on training wheels.
InfoQ: Conway's law suggest that are organizations are constrained to produce designs which are copies of the communication structures. As a result the architecture of a system often reflects the organizational structure. Do you recognize this? Is it a problem?
Ferrier: It absolutely is. I have started to try to explain the problem like this: big companies tend not to build products that are user centric but systems that are company centric. They build what they can build, accepting internal silos by avoiding internal dependencies. This means the things they build don't cut across business units or management lines. So the product isn't really a product, it's just some software. A product would be different in the sense that the company would be saying "here's what we do for you, consumer".
InfoQ: What do you suggest to address this? Can focusing on the architecture help us to find better ways to collaborate in software development?
Ferrier: I think there would be a danger of over focusing, I've certainly seen IT organizations do that. Many large organizations have teams of architects who don't actually deliver anything but advice to other teams. I question the value of that.
But one reason why architectural governance became a thing in large companies is because delivery teams seemed not to care about architecture.
I think one very serious problem is that there is little understanding of how architectural change can enable better delivery practices. This is because a lot of agilists aren't actually that technical. They're people who understand humans well.
An example of what I'm talking about is dependencies. Dependencies between "things", business processes, technology, or whatever are hard. If we have dependencies between teams of people that gets really hard. And that's where we get into project management and GANTT charts and coordination and all that risk of failure.
But dependencies happen because we can't break things down enough and build sensible contracts that allow people to move forward on either side of the contract. With product questions that's still really hard (though doable) but with technology it's kind of a solved problem. Code reuse should be delivered through service orientation, the services should be as granular as they need to be to avoid dependencies and we can support change to the contracts the services offer through additive (never breaking) change.
I'm talking about microservices of course, which is a trendy thing. But the principle isn't really new. Microservices is a code word for understanding the right level of granularity.
So a lot of the time, when we see a project manager turn up on our scrum team it's because we are not in control of our architecture.
InfoQ: How can technology help us to avoid some of the organizational problems that teams experience?
Ferrier: I think getting that granularity of technology right is really important. It's that which can really solve organizational problems.
There is a lot of resistance to this from some technical people by the way. I've not really got to the bottom of why that is. It may just be a natural conservative resistance to "hipster bullshit" but I think it may also be a deeper problem about acceptance of new technology and techniques of technologists themselves.
Why should technologists be any different than non-technologists at accepting new technology? I think that many people find things they're comfortable with when they're young and then they stop learning. And then they actively resist new things because they'd have to learn the new thing...
And this is kind of absolutely poison to solving problems. If the solution to a problem is something new then you're not going to get it from conservative technologists.
About the Interviewee
Nic Ferrier is a programmer who believes that we can solve problems by writing programs. He went to the school of hacker hard knocks, has been working in software for 30 years and now spends much of his time trying to teach enterprises about DevOps. In his spare time he writes software to solve his own problems.