A software engineering discipline must be iterative, based on feedback, incremental, experimental, and empirical. Craftsmanship is not sufficient; engineering is an amplifier, it enhances creativity, collaboration, and inventiveness. Continuous delivery is grounded in engineering principles. If you are more rigorous at the start of a project then you can create better, more innovative solutions, spend less time fixing bugs in production or hacking work-arounds to deployments and configurations, argued Dave Farley.
Dave Farley, independent software developer and consultant, presented Taking Back "Software Engineering" at QCon London 2018. InfoQ is covering this conference with Q&As, presentations, summaries, and articles.
The term Software Engineering was first used in 1968 at a NATO conference. Although progress has been made towards an engineering discipline, software isn’t engineering yet, according to Mary Shaw:
There has been a lot of effort over the years in software development methods. This established the production methods that support commercial practice. However, it did not establish the codified basis for technology that is required for engineering practice.
InfoQ spoke with Farley about what defines software engineering and why it is important, how craftsmanship relates to engineering, and what can be done to develop an engineering mindset.
InfoQ: How do you define "software engineering"?
Dave Farley: I think that there are a lot of misconceptions amongst software developers about what "engineering" is, let alone what "Software Engineering" is. I seem to get myself into a surprising number of conversations about "Bridge Building" for some reason. So the first thing that I would say is, "Engineering" is NOT the same as "Production Engineering". Doing something for the first time is very different to doing something for the 1000th. Engineering is an intensively creative discipline! Think of the engineering that goes into the creation of something like the "Curiosity Rover" or the "Falcon Heavy" or the first iPhone!
My definition of engineering in this context is:
Engineering is the application of an empirical, scientific approach to finding efficient solutions to practical problems.
For software engineering I think that there are some specifics that MUST be in place for us to achieve this "empirical, scientific approach". For me, any "Software Engineering Discipline" worth the name must be:
- Iterative
- Based on feedback
- Incremental
- Experimental
- Empirical
My talk at QCon London about Taking Back Software Engineering explored each of these ideas in more detail.
InfoQ: What makes software engineering important?
Farley: Software is important in the world! Strangely, we software developers are the people that are probably changing the world the most at the moment. Ours is still a young discipline and we have been, and continue to be, growing explosively.
My perception is that maybe the majority of software developers have never seen something that we could honestly call "engineering" applied to software development. When we do apply these principles, what we see is fairly dramatically more efficient, in terms of speed and quality.
This is not surprising as "Science", in which Engineering is rooted, is Humankind’s most effective problem-solving technique. I think that we should apply it to the very hard problem of Software Development.
Another angle on this is the evolution of Software Development as a profession. The VW emissions scandal has led to several people going to jail. Including software developers. We are responsible for the decisions that we make and the code that we create, morally as well as economically. My view of our industry has been that often we ignore these responsibilities. Unless we grasp them, and take control, these things will be imposed upon us.
An engineer that builds a bridge, or a car or an aeroplane, that kills people will be held professionally responsible. We too will face that challenge. What kind of defence for our actions will we have if we have not followed effective disciplines for testing, measuring and evaluating our ideas?
My last point though is more personal. Working with some engineering discipline is a LOT more fun. I can create better, more innovative solutions to problems. Tackle bigger problems, spend less time fixing bugs in production, or hacking work-arounds to deployments and configurations if I am a bit more disciplined in the way in which I approach my work. In my opinion, "Engineering" leverages and amplifies, my capabilities without detracting in any way from the creative aspects of software development.
InfoQ: How does craftsmanship relate to engineering?
Farley: I believe that the "Software Craftsmanship" movement was created in response to the horrors of "big ceremony", waterfall style processes in the 1990’s and early 2000’s. Craftsmanship is a step forward to these planned approaches, which are fundamentally the wrong model for processes that are variable and include a significant amount of "discovery".
Craftsmanship was the correct step, but if we look at the history of production, Craftsmanship is a low-quality process. Think for a moment of a craftsperson-built iPhone or Jet Fighter. What about "Craftsman"-led space programme?
Craftsmanship is not sufficient. Engineering is an amplifier; it doesn’t prevent creativity, collaboration, inventiveness, it enhances it.
I agree with many of the ideas from the Software Craftsmanship movement, in particular the ideas of apprenticeship style training and continual learning and improvement, but these ideas are equally applicable to a "Software Engineering" discipline and such engineering discipline takes us further, giving us routes forward when we are stuck and amplifying the quality and productivity of our work.
InfoQ: What can be done to develop an engineering mindset?
Farley: Ultimately I think that this may be a "generational" change. If we could agree some principles for "Software Engineering" then it should be taught, it should be common practice, it should be expected!
I think that my principles are a good starting point: iterative, based on fast feedback, incremental, experimental, and empirical.
I am a bit biased in that I also think that a Continuous Delivery approach to development is grounded in these engineering principles, but it must go deeper than that.
I think that there is a greater consensus about what really works than I have seen before, at least amongst the people that we may consider thought-leaders in our profession. So now may be a good opportunity to, once-again, try to define what "Software Engineering" really means.
If we could gain broad agreement on such a definition, then maybe we could start advising educational establishments and professional bodies to teach the right things. A trivial example: how many people who studied "Computer Science" learned about the "Scientific Method" and the importance of "Experimentation" on that course? How many people learned "Ethics" as part of such a "professional" course? These things are pretty normal in most engineering disciplines.
InfoQ: How about the current generation of developers, juniors and senior tech people? What can they do to catch up with software engineering?
Farley: My number one piece of advice for any software developer, junior or senior, is to adopt a more scientific approach to problem solving and take the scientific method seriously.
Think in terms of experiments, gather data to make decisions, try ideas out. Don’t assume that your first guess is correct. In fact assume that all of your guesses are incorrect and work in ways that will 1) help you discover how you are incorrect quickly and 2) won’t kill you when you get things wrong.
InfoQ: What is the difference between a "Craftsperson" and an "Engineer"?
Farley: A "Craftsperson" guesses about what may work. An "Engineer" guesses and then measures their guess to see where they got it wrong.