Abstract
The past sixty years have seen numerous "revolutions" in the realm of software development: ‘structured,’ ‘CAD,’ ‘objects,’ ‘SOA,’ and most recently, ‘Agile." Despite these advances, our ability to successfully complete software projects has improved only marginally. The time for a real revolution is overdue, and this essay introduces and outlines a direction that such a revolution might take - the establishment of a Great Art (Ars Magna).
[Brief tilte note: There have been numerous works titled Ars Magna. Two of the most relevant are: Gerolamo Cardano, in 1570 - "One of the three greatest scientific treatises of the early Renaissance." Cardano’s book dealt with Algebra and contained the first published solutions for both cubic and quadratic equations. Ramon Lull 1305 - considered the first work on computational (combinatorial) science, and one that influenced Leibniz’s work almost 400 years later.]
Why Revolution?
The past sixty years have been dominated by a specific idea about software development - that it is, or should be, a scientific and engineering discipline. Every advancement in the profession - with the exception of object orientation and agility - have been based on that idea. The scientific-engineering approach has been highly successful - sometimes!
The World runs on software, software that works.
Airplanes - that could not fly without software - cross the sky. Banks process billions of transactions daily, using software that is reliable to greater than Six-Sigma standards. Millions of people have hearts that beat 24/7/365, thanks to software. Modern life - absent the Internet and the software that keeps it running and growing - is almost inconceivable.
Everyone knows of Moore’s Law and the dramatic increase in the computational power of hardware chips. But part of that increase is due to software - specifically, replacing hardwired instructions with software equivalents. It was software (parallel programming) that allowed the most significant advances in supercomputing - replacing special purpose hardware with off-the-shelf generic components and allowing massive scaling. Computational theory (the math) has advanced significantly and we are on the verge of creating an entirely new kind of computing - quantum computing.
Today there is little doubt that we can build artifacts - devices that combine leading edge hardware with leading edge software to do amazing things. We obviously can use software to create experiences - movies and games - that rival, in terms of our sensory and perceptual abilities, anything the natural world can provide. We can build special purpose applications - an email system, perhaps - that work, work well, and work, effectively, all the time.
Unfortunately, our failures outweigh our successes; as the following depressing numbers indicate:
- $250 billion expended to develop 175,000 projects.
- 31% of projects are cancelled before completion
- 57% of projects show 189% cost overruns
- 16% of small/medium projects are delivered on time, within budget
- 9% of large projects are delivered on time, within budget
- Large projects are delivered with 42% of features specified/planned
- Small/medium projects are delivered with 78% of features specified/planned
- 94% of projects are restarted at least once
- Project development time averages 18-24 months for small/medium scale projects, and 3-5 years for large scale
Objects and Agility were deliberate attempts to define and follow a different path. Sadly, as they became ‘mainstream,’ both were rapidly co-opted, diluted, and modified to the point that, as Alan Kay said about objects, "The revolution has yet to happen."
We expect that the situation will only get worse. The pace of change is accelerating and enterprises no longer have the luxury of spending 18 months to 5 years developing software that will be effectively obsolete before it is delivered. The problems we are trying to solve with computing technology are increasingly "wicked" in their nature. We are beginning to recognize that the systems into which we deploy software and the systems we use to develop software are "complex" - non-deterministic and with self-organizing and emergent properties. Ultra-Large Scale systems are qualitatively different from anything we have attempted before - and like the Internet, are likely not amenable to being designed and implemented using formal engineering techniques.
The absence of a revolution does not mean a lack of revolutionary ideas. Clearly, agile and objects were self-conscious attempts to radically re-invent software development. Numerous other ideas have been introduced over the years, "rapid-prototyping," "software development as Theory building," and the "West Coast" - "fuzzy" AI of the Lisp community are but three examples. There is also a long, minority, tradition arguing for software as an art form rather than an engineering method. The divide between academia and practice - with practice regularly diverging from the formalism of science and engineering is another source of revolutionary ideas.
The fact that none of these revolutionary ideas has resulted in an actual revolution reflects the power of the science-engineering paradigm coupled with the relatively sporadic and isolated attempts to introduce radical change. If a real revolution is to occur it must be grounded in an alternative to the prevailing paradigm.
Points of Divergence
Traditionalists, software engineers and computer scientists, claim that the "failures," noted above, result from a lack of rigor. If everyone would simply adopt more formal (mathematical) techniques for specification, programming algorithms, testing and proof, all software projects would succeed. Quality and process advocates suggest that adoption of a more formal and mechanical process (e.g. Six Sigma, TQM, CMM Level 5, and even Lean development) will solve most if not all, ills.
The traditionalists may have a point - we actually lack the kind of empirical information that would allow us to decide. But, we do know that teams and organizations are "trying to do it right," (i.e., according to the precepts of software engineering), but are still failing. Despite our best efforts, our ability to consistently deliver high quality software on time and within budget has only marginally improved over the past sixty years. This leads to the conclusion that traditional approaches are doing something wrong, or are approaching the problem from the wrong perspective.
An alternative perspective can best be shown by contrasting fundamental "points of divergence" - similar in spirit to the Agile Manifesto. Nine of the points are conceptual and address the assumptions or presuppositions that differentiate Ars Magna (AM) and traditional Software Engineering (SE).Four of the points introduce values that may be shared by AM and SE, but they are emphasized and central to the practice of Ars Magna.
The concepts:
- Artifact-versus-system. SE focuses on the artifact to be delivered - the program. When SE talks of systems they mean the inter-related construct of programs and hardware. An artifact can be specified and engineered to meet specifications. Specification conformity is the measure of success in building an artifact. AM focuses on the complex adaptive system in which the software is ultimately embedded - i.e. the business enterprise or the socio-cultural system. Success is measured in terms of the stability, flexibility, adaptability, and "enhancement" of the contextualizing system.
- Deterministic-versus-non-deterministic. SE is grounded in world-view that came from 19th century physics - i.e., the universe is a deterministic system (a clockworks / a machine) that can be understood and manipulated via formally defined laws and relationships. AM views the world, instead, as a complex adaptive system, non-deterministic and highly dynamic.
- Production-versus-Theory. SE is committed to the concept of a formally defined, manageable controllable, and, ultimately automated, production process. AM adopts the position advanced by Peter Naur, that development is ultimately a human act of "Theory Building." (We will have more to say about this later.)
- Mechanical-versus-organic. This difference is most obvious in the choice of metaphor, borrowed concepts, and the definition of fundamental terms. SE speaks of bridges and buildings - AM talks of cells and ecologies. SE believes in types - AM in prototypes (ala Lakoff and Johnson). SE defines objects as abstract data types with attributes and functions - AM objects are anthropomorphic homunculi with knowledge and behavior.
- Prediction-versus-Exploration. This is most obvious with regard process and management; with SE convinced that enough can be known to accurately estimate and predict the course of the future - e.g. "Plan the Work and Work the Plan." AM favors trial-and-error exploration and basing projections of what might be on empirical observation. SE conceptualizes (somehow) ‘The Answer’ which is then simply implemented. AM ‘pokes around’ until ‘The Answer’ reveals itself.
- Episodic-versus-Continuous. SE is focused on discrete, isolated, projects with clearly defined beginnings and ends; that proceed according to their own internal logic in timeframes that are inconsistent with the natural cycles of the enterprise or organization that chartered the project. AM is focused on continual, but small, changes to a living and highly dynamic system. Changes are limited in scope and scale to that which can be accomplished in short, natural - i.e. consistent with the embedding system, cycles measured in hours to weeks, seldom months, and never years. The AM development process resembles the SE "maintenance" process.
- "Self conscious"-versus-"non-self conscious." The terms come from Christopher Alexander’s first book, Notes on the Synthesis of Form. A thorough discussion of these ideas is not possible here (we will note, however, that Alexander disparaged the self-conscious process) but SE is self conscious and AM is non-self conscious.
- Control-versus-Coordination. The concept of centralized hierarchical control has been central to SE since the days of structured programming. The old Program Structure Chart with its Afferent, Efferent, and Transform modules reporting to a single Master Control Module is still visible in most programming languages (e.g. the mandatory ‘main’ function in a Java program). AM has failed to totally divorce itself from ‘control’ - witness the Model-View-Controller architecture at the root of early Smalltalk implementations - but, philosophically, it eschews control in favor of object autonomy, cooperative distribution of workload, and coordination.
- Art-versus-science. An old debate, with SE clearly against any vestige of ‘art’ while Ars Magna IS Art.
The Values:
History. Few disciplines are as ignorant of their own history as computing. Ars Magna stresses the need to be aware of our own technical past. Awareness and acknowledgement of "Old Masters" and the study of their work and ideas is essential.
Philosophy. We focus too much on practices, techniques, and tools and far too little on the philosophical presuppositions, the cultural biases, and the unfounded assumptions behind the particular. Philosophy is critical to understanding why things are the way they are. Philosophy provides us the knowledge foundation that supports decisions to adopt technology and practices and to appropriately adapt them to new circumstances.
Liberal Arts. The Ars Magna professional must have an understanding of an incredibly broad range of knowledge. The computer scientist can be focused within the discipline but software developers must be able to comprehend myriad domains, be able to recognize and exploit, as metaphors, ideas and concepts from other fields of enquiry, and, perhaps, most importantly, be able to integrate diverse threads of knowledge.
Humanity. Systems are designed and crafted by humans, for humans.
The concepts and values, just noted, define the point where the practice of Ars Magna diverges from mainstream software development methods. But what is the Great Art itself? The art of what?
Simply put, Ars Magna is the great art of Theory Building.
Production-versus-Theory
In 1985, Peter Naur (co-author of the programming language syntax notation, "Backus-Naur Form" or BNF) challenged the prevailing notion that "programming" was "the production of a program and certain other texts." The word programming is in quotes because Naur used the single word, "to denote the whole activity of design and implementation of programmed solutions."
Then, and now, mainstream software development is premised upon the idea that the goal is to produce an artifact (or artifacts) using a rational and replicable process - a production model. When Agile, specifically XP and Crystal, was first introduced it was a conscious effort to escape the production mindset. Alistair Cockburn goes so far as to, in his book, Agile Software Development, reprint Naur’s 1985 paper, "Programming as Theory Building." Unfortunately, Agile has moved closer and closer to the mainstream notion of production. Scrum and Lean are the two most popular and widely used variations on Agile ideas: Scrum is all about process management and Lean (both the production and the product variants) is all about process optimization.
Although the implementation of Agile has come to be about production and the management of production, Agile values and practices, especially as articulated in extreme programming (XP), are directly supportive of Theory Building and will also be Ars Magna practices and values.
A Theory, according to Naur, is the shared mental understanding, of a group of people, of "an affair of the world and how software will handle or support it." The Theory inside the head of an individual programmer (remember the more expansive definition of this term by Naur) involves: " ... the activity of matching some significant part and aspect of an activity in the real world to the formal symbol manipulation that can be done by a program running on a computer."
It is critical to note that in both the group and the individual - Theory involves an understanding of the World and what goes on inside the computer on behalf of the World. Theory is focused on a single system - the World or an enterprise - composed of discrete elements. Elements are defined by the role they play in the system and by their interactions with other elements. Some of those elements might be ‘improved’ or better able to fulfill their role in the system if they were automated - if they were implemented as programs running on a computer.
This view of theory radically redefines the software development work unit. Strictly speaking any given act of programming will involve a single element of the overall system and perhaps a single behavior associated with that element, or, even more precisely, "some significant part and aspect of an activity." This redefinition is not totally alien, however; XP does essentially the same thing, defining a unit of work in terms of a User Story - one thing the user wants or expects from the ‘system.’
Partitioning the software development workload in terms of individual system elements, individual element behaviors, or single User Stories - has the necessary effect of simplifying the programming task. Individual users stories should take a team (at minimum a pair of tester/programmers and an On-site Customer) no more than a couple of days to implement, and deploy. The solution, once in production, provides feedback - is it usable, useful, and helpful - which confirms (or contradicts) the Theory.
A Theory is a gestalt - a comprehensive and complete - understanding of the single system. The Theory, therefore, encompasses the understanding of any need to coordinate and communicate across multiple elements. The System that is understood by the Theory provides both architecture and what XP called, the System Metaphor. Theory provides the necessary context for detailed understanding of any individual element of the system, what it is supposed to do (including obligations to all other system elements), and how it is to do it.
A team possessing a Theory can, according to Naur:
- " ... explain how the solution relates to the affairs of the world that it helps to handle."
- " ... explain why each part of the program is what it is, in other words is able to support the actual program text with a justification of some sort."
- " ... respond constructively to any demand for a modification of the program so as to support the affairs of the world in a new manner."
It cannot be demonstrated here[1] but a Theory of a single system with some automated elements has the effect of dramatically simplifying the software. No more million line programs; no more thousand-line ‘main routines’ with complex Boolean or nested ifs, case statements, or cyclomatic complexity. This simplicity, coupled with the redefinition of a unit of work means that any piece of software can be changed or replaced in a day or two and that the software and the system in which the software is embedded (the enterprise) change and evolve in the same timeframe. Because a Theory is of both "an affair of the World and how the software handles or supports it," software and business elements are always reflective of each other and in synch - a long sought goal, "business ecology," is realized.
An important caveat: the vast majority of any Theory exists only in the minds of the people that developed the theory! For Naur:
"A main claim of the Theory Building View of programming is that an essential part of any program, the theory of it, is something that could not conceivably be expressed, but is inextricably bound to human beings."
A consequence of this view is that ‘documentation" is of very little value to anyone except those possessing the Theory in their collective heads. (It also different teams can ‘document’ requirements, design, or a code base and pass those to different teams who will somehow obtain the Theory simply by reading and understanding the documentation.) "Documentation" does have a role and a purpose. Reinhard Keil-Slawik shows how artifacts, like whiteboard drawings and Story cards, facilitate communication and collective understanding - i.e. the actual development of the Theory - and serve as evocative triggers that bring certain aspects of a Theory to conscious attention.
Naur suggests that it is critical that "the programmers having the Theory of the program remain in charge of it." In today’s work environment this is not practical - but XP concepts, like Whole Team, On-Site Customer, Pair Programming, and Collective Code Ownership expand the size of the group holding a Theory to a "critical mass" that can accommodate normal organizational turnover without losing the Theory. The ultimate goal would be for everyone in the organization to possess a common ‘Theory of the Enterprise." This common Theory would not impose the need for everyone to understand all aspects of the Theory - coders would still know more about ‘how’ an automated system element does what it does and a CPA to know why a tax calculation formula has to be what it is. Everyone would share a common understanding of the system, the behavior of its elements, and relationships among those elements. Everyone could, therefore, be in a position to recognize potential innovations and either implement them directly or communicate them to others who can do so.
A Theory is what is built - the result of Theory Building.
Theory Building
Ars Magna is not a process, nor is it a method. It is a set of activities and practices that reflect values and ideas. Ars Magna is antithetical to the notion of method in the same way that Theory Building is contrarian.
"To begin with, what is a programming method? ... Here a programming method will b taken to be a set of work rules for programmers, telling what kind of things the programmers should do, in what order, which notations or languages to use, and what kinds of documents to produce at various stages.
A method implies a claim that program develop can and should proceed as a sequence of actions of certain kinds, each action leading to a particular kind of documented result. In building the theory there can be no particular sequence of actions, for the reason that a theory held by a person has no inherent division into parts and no inherent ordering. ... As to the use of particular kinds of notation or formalization, again this can only be a secondary issue since the primary item, the theory, is not, and cannot be, expressed.
It follows that on the Theory Building View, for the primary activity of programming there can be no right method. ... the quality of the theory built by the programmer will depend to a large extent on the programmer’s familiarity with the model solutions of typical problems, with the techniques of description and verification, and with principles of structuring systems consisting of many parts in complicated interactions. ... Where the Theory Building View departs from that of the methodologists is on the question of which techniques to use and in what order. On the Theory Building View this must remain entirely a matter for the programmer to decide, taking into account the particular problem to be solved."
Naur’s depiction of Theory Building can be simplified to, "Do the Right Thing at the Right Time as a function of the Right Context." This ability is based on a comprehensive knowledge of the problem domain and how it works, the problem, exemplar solutions, and the ability to think and not simply follow rote procedures. Kent Beck talks about XP in similar terms when he describes three phases of XP as: 1) out of the box - simply do what the book tells you to do; 2) adaptation - modify your book learning to suit particular contexts and problems; and 3) "transcendence" - do the right thing at the right time as a function of the right context.
This description should not lead one to believe that individuals or teams must establish a Theory before any productive work is done. Theories are developed the exact same way that Agile teams develop software - in an iterative-exploratory-incremental manner. In fact, you could simply repurpose the Agile (XP) practices as Theory Building practices and recognize that; at any given time, any set of practices, might be the most appropriate actions to take in order to advance and deepen the Whole Team’s Theory.
We can also relate common practices and tools directly to items that enhance the quality of a Theory - e.g. Patterns ("model solutions of typical problems"); TDD ("techniques of description and verification"); and User Story, Planning Game, and Retrospective ("principles of structuring systems").
Ars Magna requires thinking - as does Theory Building. And, ultimately, It’s The Thought That Counts.
Ars Magna recognizes five different "modes" of thought that must be mastered:
- Systems Thinking - In addition to the General Systems Theory that Gerald Weinberg, and others, attempted to bring into software development and computer science, it is also necessary to understand Complex Adaptive Systems and concepts like emergence, self-organization, and scale-freeness. Systems must be understood as more than a complicated interacting set of hardware and software components.
- Object Thinking - The ability to use the object metaphor (a homunculus with behavior and knowledge that is used to provide service to others) to decompose complex systems into their constituent elements and distribute responsibilities across those elements in an appropriate manner.
- Agile Thinking - A comprehensive understanding of the values, principles, and philosophy that shaped particular Agile practices and how to use that understanding to adopt, adapt, and eventually transcend the practices.
- Design Thinking - Importing and repurposing design concepts and tools - e.g. Design Brief, prototyping, visual thinking, metaphoric reasoning, liminality, connection and juxtaposition, divergent and convergent thinking, and bounding constraints - into Theory Building and Ars Magna.
- Computational Thinking - A deep understanding of what goes on inside the machine and the "how" of automating World System elements without adversely affecting their role or their essential nature. [Computational thinking should NOT, be used as the foundation for understanding the World around us as some have recently advocated.]
Conclusion
This essay is an intentionally provocative and controversial call for a real revolution in how we conceive of and practice software development. It asserts that most of software development (excluding low level software like device drivers, up to, perhaps operating systems) should be recognized as an Art, not a Science or Engineering discipline. It suggests some conceptual foundations on which the Art - Ars Magna - might be based along with connections to important but neglected or abused ideas in the profession; theory building, objects, and agile, primarily.
The intent is to raise awareness and stimulate discussion not to offer a prescriptive and comprehensive definition of Ars Magna (something will require one or more books to treat properly).
[1] The data behind this assertion are metrics that were observed in early (before they were diluted and corrupted) examples of Agile projects and Object projects.