Evaluating and Improving Architectural Competence - A New SEI Paper

The Carnegie Mellon® Software Engineering Institute (SEI) recently published "Models for Evaluating and Improving Architecture Competence", an 87 page report that examines how software architects and organizations can foster environments that consistently produce high quality architectures. From the abstract:

Software architecture competence is the ability of an individual or organization to acquire, use, and sustain the skills and knowledge necessary to carry out software architecture-centric practices. Previous work in architecture has concentrated on its technical aspects: methods and tools for creating, analyzing, and using architecture. However, a different perspective recognizes that these activities are carried out by people working in organizations, and those people and organizations can use assistance towards consistently producing high-quality architectures.

The report considered four models of human behaviour and competence, as it pertains to software architects: duties, skills and knowledge; human performance engineering; organizational coordination; and organizational learning.

The duties, skills, and knowledge (DSK) required of a software architect

The authors (Len Bass, Paul Clements, Rick Kazman and Mark Klein) state that "duties, skills, and knowledge form a triad upon which architecture competence for individuals rests", and listed attributes of each area based on considerable research effort consolidating a vast amount of raw data.

Identified duties included:

• Architecting - Creating an architecture, Architecture evaluation and analysis, Documentation, Existing system and transformation and other architecting duties not specific to the above categories
• Life-cycle phases other than architecture - Requirements, Coding, Testing, Future technologies, Tools and technology selection
• Interacting with stakeholders - Interacting with stakeholders in general, or stakeholders other than clients or developers, Clients, Developers
• Management - Project management, People management, Support for management
• Organization and business related - Organization, Business
• Leadership and team building - Technical leadership, Team building

Identified skills included:

• Communication skills - External communication skills, Communication skills in general,
• Internal communication skills - Interpersonal skills Within team, Outside of team
• Work skills - Leadership, For effectively managing workload, For excelling in corporate environment, For handling information
• Personal skills - Personal qualities, For handling unknown factors, For handling unexpected developments, Learning skills

Identified knowledge facets included:

• Computer science knowledge - Knowledge of architecture concepts, Knowledge of software engineering, Design knowledge, Programming knowledge
• Knowledge of technologies and platforms - Specific technologies and platforms, General knowledge of technologies and platforms
• Knowledge about the organization’s context and management - Domain knowledge, Industry knowledge, Enterprise knowledge, Leadership and management techniques and experience

The report suggests three ways individuals can improve their DSK competence:

1. Gain experience carrying out the duties: Apprenticeship is a productive path to achieving experience. Education alone is not enough, because education without on-the-job application merely enhances knowledge.
2. Improve nontechnical skills: This dimension of improvement involves taking professional development courses, for example, in leadership or time management.
3. Master the body of knowledge: One of the most important things a competent architect must do is master the body of knowledge and remain up-to-date on it. Taking courses, becoming certified, reading books and journals, visiting websites and portals, reading blogs, attending architecture-oriented conferences, joining a professional societies, and meeting with other architects are all useful ways to improve knowledge.

Human performance technology, an engineering approach applied to improving the competence of individuals

In this section the authors discuss how Human Performance Technology (HPT), as pioneered by Thomas F. Gilbert, can be used:

• We wish to engineer worthy performance; that is, performance that produces value at reasonable cost. Worth = Value / Cost.
• The Potential for Improving Performance (PIP) is the ratio of exemplary performance to typical performance, where exemplary performance means the best performance for which we could reasonably hope. PIP = W_exemplary / W_typical. If the best worker in the best environment turns in a performance worth 75 units at a cost of 10 units, then that worker’s performance is 7.5. If the average worker turns in a performance of 2.5, then we have a PIP of 3, which signals the opportunity to triple what the average performer produces through improvement steps.
  
  High PIPs are opportunities for great improvement and should be the focus of effort. Low PIPs (near 1) will require a lot of investment to further improve. PIPs in extremely competitive fields tend to be low,9 as do PIPs for highly repetitive tasks. This reasoning assumes (which Gilbert explicitly does) that "poor performers usually have great potential." Or, "the more incompetent a person or a group of people are, the easier it is to improve their performance." The ratio, to be meaningful, must be stated for an identifiable accomplishment, because there is no "general quality of competence."
• The key to performance improvement is finding the right measures that accurately reflect the value of the performance.

(an modern evolution of HPT can be seen at The Six Boxes Model).

The authors identified two problems in applying HPT - How to measure the worth of an architect's performance? What infrastructure is required to calculate the worth?

As yet, we have only preliminary thoughts on the first question and none to report on the second. We propose to use the duties from the DSK model to isolate the various aspects of the architect’s job.

Organizational coordination, the study of how people and units in an organization share information

This section of the report focuses on multi-site development. The report suggests that a Dependency Structure Matrix can be used to expose interaction between modules, with the premise being the each team is working on a module. The DSM provides a measure for how much coordination is required within the organization:

An organization’s ability to match the coordination capability with the coordination requirements imposed by the architecture is a facet of architecture competence.

It then discusses means of coordination, including a discussion on how an architect can act as intermediary between teams:

For example, some benefits of having an architect as an intermediary between the two sets of developers are the availability of additional contextual information and the ability to utilize the decision when making future decisions. Some costs of an intermediary include the possibility of bottlenecks, the potential of introducing delays, and the potential loss of information when it is passed between developers.

Organizational learning, an approach to how organizations acquire, internalize, and utilize knowledge to improve their performance

The report asserts that architecturally competent organizations have the ability to turn experience into knowledge through what it terms "mindfulness":

For example, an architecture team that conducts a review or architecture analysis after completing a module or subsystem to learn what went well and what went wrong exemplifies a group engaging in mindful learning.

Conclusion

The final chapters of the report focus on the principals that the models embody, such as:

Mentoring other architects is likely to lead to high-quality architectures because being mentored is an effective way to gain real-world experience and thus become a more capable architect.

Additionally, the report discusses how questions for an assessment of competence could be carried out, for example when looking at duties:

Duty: Creating an architecture
Question: How do you create an architecture?

• How do you ensure that the architecture is aligned with the business goals?
• What is the input into the architecture-creation process? What inputs are provided to the architect(s)?
• How does the architect validate the information provided? What does the architect do in case the input is insufficient/inadequate?.

The authors avoid proposing a grand "unified model of competence", but instead provide strategies for improvement using the models, concluding with:

Together the four models provide strong coverage in a number of ways. They provide a basis for measuring past performance as well as current activity. They cover the range of observational possibilities: artifacts, processes, people, and organizations. Finally, they apply well as a group to individuals, teams, and organizations. This coverage gives us confidence that the four models together will produce valuable and reliable results.

The report can be downloaded here (PDF, 990 KB).