IT And Architecture: Inside-Out Perspectives

The software industry is in disarray, costs are escalating, and quality is diminishing. Promises made by Cloud Computing are still far from materializing on any significant scale: the panel discussion at the recent Batler Group’s symposium [“Business Process Management and Service Oriented Architecture”] admitted that the real cost saving’s available are in Public Cloud only, but it is not transparent enough to be considered by medium-to-large businesses as an IT cost driving solution. There are now many such catch-phrases within the industry, but do they address the real problem? What is the real problem?

The industry is throwing methods and tools at the perceived lack of productivity in software development of less skilled developers hired massively nowadays, creating a large learning curve and a fragmented environment. What is the real catch? Aren’t systems simply tools to assist the business in accomplishing what it must do? Or are the systems more than tools? Do we only need to choose the right tool and the right method to achieve software nirvana? Why is it that software development seems to be so hard?

Two IT professionals - Bruce Laidlaw and Michael Poulin - each with more than 30 years of experience compared their conclusions about the past and present of IT, focusing on what we are doing to make progress. Bruce has been exposed to just about all methods and techniques in the industry through involvement at every level, on projects ranging from a few thousand dollars to projects consuming over $300M, and initiatives which have consumed over $2B and counting. His experience covers the small mom and pop operation, private industry, governments at UN, national, provincial and local levels, and a whole rainbow of industry sectors. Michael has worked mostly in large software development houses and the IT departments of several major financial institutions on both sides of the Atlantic. He concentrates on bridging between the business and technology at the cross-functional and enterprise levels and contributes to the work of a few global Standards Bodies. His background includes many years of mathematics algorithm development, advanced software design and testing, distributed computing, service orientation and development governance.

Bruce and Michael have boiled down their experiences within a few topics:

• state of the SW industry
• business needs
• architects as a business-technology glue
• how we do things in IT
• architecture as an agility instrument of delivery.

They discuss these topics from their own perspectives with the objective of uncovering what we as an industry need to know and to do to move IT forward.

The State of the SW Industry

To know how to move Software (SW) Industry, we need to know where we are and where we have been.

Bruce

Well designed and well-built software has been implemented throughout time since the beginning of the SW Industry regardless of methods and tools. Poorly written software has also existed throughout that time and continues to be developed today. Unfortunately, most software systems developed from the early days until today fall into the poorly designed and poorly written category. And to be honest, some software developed today falls into the poor category as well.

To be objective, one has to stand back and ask the question: Is the enterprise better off today with the systems it is buying or building than it was 30 years ago? I believe overall that answer is a resounding no. Here are characteristics of the software industry as I see them today:

1. High Cost: Projects that would have cost tens of thousands of dollars years ago now cost millions. Those projects, even in today’s dollars, would still be a fraction of the cost paid today.
2. Low Quality: Software quality is certainly no better today than it was years ago despite ‘advances’ in testing and ‘methodologies’ aimed at correcting this situation.
3. Endurance: There is a myth that software systems last only 5-6 years before they have to be replaced, while most enterprises are still operating on systems that are 20-30 years old.
4. Technology: There is a theme and attitude within the industry that technologies have advanced well beyond where they were 30 years ago. With respect to hardware, no real argument, machines are smaller and faster; however in software, absolutely not. In fact, it could be shown that productivity has declined in general. Some innovative approaches to business have mitigated some of that by spreading the work around, but software is still pretty much an art, and the tools, though perhaps different, are just tools.

Michael

For the last decades SW advanced in all dimensions: from nuclear power plants to socializing customer forums that affect merchandising. For instance, did we have programs for planning a landscape for our backyard gardens 30 years ago? I did not see many of them and the reasons for this was in our opinion about the role of SW technology – such things as gardening were not serious enough to consider spending expensive computer resources on them. Nowadays, SW is everywhere and does so many different things that any one example would not be informative enough.

Nevertheless, for example, comparing apples to apples, modeling in the financial industry, the level of administrative overhead which has grown on the top of the SW program has increased dramatically and the same has happened with respect to cost. Some people attribute these to the architectural/design and management infrastructures that today are built to handle all kinds of SW programs. However, a majority of these infrastructures are assumed to increase quality of SW products. Why, in this case, do we see an opposite picture in this area? I believe that there may be a lot of different explanations and I do not pretend to cover any significant part of them. I will only point to some aspects that jump out at me:

1. The scale and complexity of regular SW projects have outgrown the skills of the average SW developer. For instance, people still approach SW projects with the same two ideas as many years ago: a) we have to deliver something that works; b) we deliver the best that we can. However, ‘something that works’ and ‘the best we can’ are not the things that the business needs in too many cases. The IT projects require a much more professional contribution to succeed then before in the form of either architectural work or analytical work within the Business that can fill the gap between needs and skills.
2. Proponents of simplicity oversimplified solutions, referring to resource constraints as the driving factor. Any further change resulted in the primitive, haphazard, spaghetti-code that coupled everything with everything and contributed into the rocketing cost for following projects.
3. Massive production of SW Engineers capable of writing programs, i.e. knowing how to code, but lacking the overall vision of IT’s role in the enterprise, i.e. lacking the understanding as to why they write what they are asked to write.
4. An attempt, related to massive production of programmers, to organize their work in a manufacturing/assembly-line style. However, even simple programming is a much more complex task than an assembly-line operation and people resisted this approach. This statement seems going contrary to popular Lean development and management method in IT only on the grounds that it has come from the manufacturing in Japan. In reality, adoption of Lean to SW has significantly modified the initial lean manufacturing concepts and benefits from applying these principles still need to be proven.
5. SW development methods have shifted towards satisfying the interests of individual stakeholders instead of meeting the corporate needs. In many cases, Agile Methodology was used to calm down disappointed stakeholders while it also allowed the business to substitute business requirements with aspirations.
6. Modern Tools for writing code have contributed to the reduction of SW quality to some degree. Good tools cost good money and, if you are not a perfectionist, you can justify spending this money to your CFO only by demonstrating the failures that happened before. At the same time, cheaper tools are more primitive and encourage programmers to write simple code only. Simple code is not always the right code (though the right code is usually simple) but it is suitable for delivering ‘something that works’. That is, such tools promote a mindset of low quality development into the “standard approach”.
7. Overseas outsourcing has dropped SW quality returning us a decade back. Being blinded by questionable “economics” of overseas outsourcing, business and IT decision-makers moved IT into an immature technology culture with different understanding of the business values and IT’s role in the company (cultural differences in the business environment.) Though the situation improves slowly, local developers still have to ‘refine’ deliverables from the overseas contributing a lot to the already increased cost of outsourcing.
8. “We will fix tomorrow what we have screwed-up today.” This regressive project management philosophy has found a “theoretical” excuse in some development methodologies that insisted on implementing only given requirements without concerns about tomorrow’s changes. The business allowed itself to be fooled by the ‘lower costs’ of such projects and had to pay a lot (often much more) for the ownership of the project results.

Understanding the Business of Your Own Enterprise

If you work in an Information Technology department of any organisation of any size, there are two questions you have to ask yourself periodically: “what am I doing?” and “why am I doing what I’m doing?” If your answer is something like “I am doing this for my paycheck and I will do whatever it takes to get paid”, nothing is wrong with that and no questions asked. Though it is desirable, not everybody has to be an enthusiast. However, if you are working for an employer and your answer is “I want to produce the most elegant solution or the neatest code”, we have to talk about it because this motivation usually creates waste. The ideal approach would be like this: “I want to produce code that is meaningful to the business and enables the business to grow without having to rewrite this code”. If the people developing your system do not understand your business, and make no effort to do so, then you are going to either fail to implement, or your business will be crippled by the system that is implemented.

Bruce

Who understands your business the best?

In most cases, the persons that understand the overall role and vision of the business the best are senior managers. That vision is not always represented in the lower levels of the organization. The lower levels will have a much better handle on the processes of the business, how the business is organized to accomplish its vision right now, but the lower echelons don’t always understand why a process is the way it is, or when it should change. This is more the case for large corporations than for smaller firms but is a persistent challenge when attempting to understand a business.

Some methodologies begin by having business analysts look at the ‘current system’, whether that is a manual process or an automated one, or some hybrid of the two. It is true that a business analyst can gather some information about your business from looking at the current systems, but more likely than not, what they will learn is how your business works today, which may have little to do with how it should be working. This activity of mapping the current system has its place once we know what the new system will look like and need to plan a transition, but for analysis and requirements purposes it is hugely flawed and can waste a lot of time and money, and set the project on the wrong path right from the start.

Who is this business analyst that will capture this understanding of your business? What is their training? What skill set do they require to do the job effectively? Do they need to be an expert in your industry sector in order to do the job well?

In the ‘olden’ days, a new recruit into IT went through a series of progressive steps, one such example follows:

1. Junior programmer
2. Programmer
3. Senior programmer
4. Programmer analyst
5. Analyst programmer
6. Analyst
7. Business analyst

It was understood that a new recruit might have the technical skills in place: some favorite language de jour or database technology, etc., but that they would be ‘junior’ in their ability to apply those skills effectively. Part of the training that allowed one to go up through the ranks was to begin to see patterns and to understand how businesses work. What is a general ledger? What is an inventory system? How do those two systems interact? Through this mentored growth, the computer skills were honed, but so too was business acumen and knowledge. By the time someone reached the stage of the business analyst, they understood fully both how generic business works, and how technology and systems come along side to enable that business.

Yet, there were no “good ‘ol days” in software. There were many of the same problems then as there are today. Not every company had the foresight or patience to manage this growth process within their IT staff and all of the normal human foibles got in the way of the right people getting promoted into the right jobs.

However, when it worked, it worked very well.

Today, those seven levels identified above are two, and they are not related to one another as levels, they are different disciplines and there is no difference between ‘developers’ today, whether they have 1 years’ experience or 20. Similarly with ‘business analysts’, you either are one, or you are not, regardless of experience. The downside of this is that very often junior people are controlling the outcome of very expensive projects.

It is also true that experience, though a necessary component of competent architects and analysts is not a sufficient condition in itself. Some people have 20 years of 1 years’ experience it seems. (It is my opinion that a competent architect takes on the role of the business analyst during initial project phases, however the industry has taken to creating ‘business analysts’ outside of the architecture group, and without the experience and depth acquired by the architect. For this reason, I will refer to the ‘competent business analyst’ as a distinct position, but assume competent means they are essentially an architect as well.)

So, what are the hallmarks of a good business analyst or architect? The following expresses my two cents:

• they have worked on projects across a broad spectrum of industries
• they have ‘been there and done that’ in regard to building and successfully implementing systems
• they are conceptual thinkers, able to quickly see the similarities and patterns from one business to another
• they are quick learners and can communicate clearly what your business is about in a very short time; perhaps better than any of your staff might
• they ‘get it’ when you express what your business is about, and what it needs.

The last point begs a question: who knows what your business needs? Is the customer, the business user or owner, the one who should be articulating system requirements? Are they trained as systems architects or analysts? No. The business user can best describe what the business is about and what it needs with respect to information to do what it has to do. They can even have a voice as to how they see things working, but that is where the input needs to change from the direction to a suggestion. It is (should be) the business analyst/ architect that can best ‘systematize’ the business requirements for the business user.

Many methodologies today break down on this point, and many systems fail as a result. I have heard time and again how the system’s staff complained that the requirements weren’t properly articulated by the customer. Sorry folks, it’s not the customer’s job to develop the requirements, that’s your job, my analyst friend. The analyst must have the skills to listen to the business, and from those conversations, uncover the true requirements. In the vacuum of solid analysis, and the trust relationship that is built with management when an analyst ‘gets it’, the business will force its view of things. Unfortunately, it does so to its peril more often than not. They get what they asked for, but not what the need.

Michael

The role of the business was undermined or ignored by IT for several years. IT people constructed a belief that they know better what and how to do things for the business. Even in centralized IT, developers did not see further than their projects, i.e. what their colleagues did, and some SW development methods promoted this focus on delivering something. When the bridge between the Business and Technology had almost crashed, IT started talking about agility with the Business having a very little understanding of the subject.

I learned about this problem by working hard over the last 10 years with service-oriented solutions. I watched how many companies failed with their SOA initiatives despite being told that SOA magically delivered agility. I know that one of the major reasons for these failures is not only that IT misunderstood SOA, but that IT tried working with the process-centric Business when offering service-centric solutions. In other words, IT reacted to how the business worked instead of reacting to how the business should work.

In many industries, IT has converted from a tool-developer into the solution-developer, which includes business solution development. However, to produce a right solution, IT has to know correct business tasks or business needs. Dealing primarily with the low-level operational business teams, IT is locked into the problems (in the majority of the cases) of this level of the business while more often than not this level reflects business requirements of the past. The truth is that the low-level operational business processes are nothing more than the business’ realization of the functionality and, over time, these processes become detached from the current needs and strategic corporate goals / objectives.

To be agile with Business, IT has to know where the Business is going and where it plans to go in the future, and adjust SW development appropriately. Adoption of business changes in IT takes much longer than in Business because of hidden and undocumented dependencies in the code and significant sharing of the program libraries between unrelated applications, i.e. one change affects many things that must be re-balanced and stabilized before becoming available to the business users again. This means that IT has to be even more knowledgeable about anticipated changes in the Business world than the middle- and low-level business operational units. The economic turbulence of the last decade has created conditions where IT could not ‘grow’ business-savvy SW specialists. This means, IT cannot rely on naturally absorbed business knowledge in the technical teams anymore; IT has to have a dedicated stuff that is obliged to work with Business. This stuff comprises Architects and Business/System Analysts.

So, IT Architects and Business/System Analysts have to work closely with middle-to-senior business personnel to learn intentions and directions, on one hand, and provide guidance on existing and new technical capabilities on the other. This work cannot be done on a project-by-project basis; it has to be done either as parts of the full-time duties, or in the scope of global programs. In other words, IT has to have an organisation of Architects and Business/System Analysts across current and future projects. In situations of outsourcing or frequent lay-offs, it is the very Architects and Business/System Analysts, not managers, who constitute the IT treasure for the enterprise. If you cut the business knowledge carriers in IT, you cut the IT brain. Then you have to work with brainless IT...

Following this line of logic, let me walk one step more toward the business. It is obvious that the middle-to-senior business people do not have time (or desire) to help IT with its problems. An understanding of this fact resulted in the emerging of the institution of Business Architects within the business. Together with Business Analysts, who actually work with management on implementing tactical and strategic business tasks, Business Architects deal with top-level business planning identifying business needs, constructing business architectures, and formulating business tasks for implementation. Business Architects also interact with IT management and Technical Architects. This means that Business Architects and Business Analysts within the business are the people who construct core Business Requirements (BR). These requirements are finally split between business units and IT.

From my experience, and the practice of my colleagues, I know several cases where the BR were extracted from, and dedicated to, the subjective needs of different low-level business processes, and these BR “drowned” IT by exhausting all resources. I see only one mitigating solution for this problem – all BR that are planned to be handed over to IT should go through some form of business validation against the Business Strategic and Divisional/Departmental Plans and the validators should be the Business Architects and Business Analysts within the business. I have witnessed this approach – and this works very well, and when IT pushes for business validation of the BRs, respect for IT grows.

The Role of Architects in Modern Enterprises

Over the last few years, it has become a bone tone agreeing on semantics before getting into debates. So, for this discussion we both agree that Architecture (software, business, technology, hardware, etc.) is defined (not described) in accordance with the IEEE 1471 standard. We can only add to the IEEE 1471 that the elements comprising the architecture

• must be fundamental, self-sufficient and coherent
• can exist independently from others (i.e. cannot be derivatives from others)
• should provide the “views of” (inside-out) rather than by the “views on” (outside-in) the architectural elements.

The last statement has to be explained. Particularly, any view on the thing, regardless who is looking at it or where from (what the viewpoint is), always carries a very high risk of misunderstanding what the thing is in reality because a) it is subjective, 2) it does not know the boundaries of the thing and can consider a part of environment as a part of the thing. In a contrast, a view of the thing is based in the actual knowledge of what the thing is and can produce consistent views for any interests of external stakeholders.

While a notion of Architecture is relatively clear, we still cannot say the same about the role of Architect. There is no standard industry definition of what the characteristics are for an Architect, and there is a definite partition between the two primary camps of opinion. One says that an architect is a super-technician that understands intimately some or all of the current technologies: such an architect is called a .Net Architect, or a JEE Architect, and so on.

The other camp suggests that an architect is less technical, and is more concerned with ‘systems science’ and its methods, which have nothing to do with automation per se. This discussion presumes an architect to be of the latter kind, but one that has also a depth of technical expertise as a result of years of experience. By the 'less technical', mentioned above, we mean that the Architect is the person who knows why, where and when, e.g., messaging has to be used in the solution; whether it is a vendor-developed messaging or home-made messaging is immaterial from the architectural perspectives (though it is very important to the implementation and management viewpoints). This architect will not be the leading-edge technical expert, but will be able to communicate between the business and technical worlds competently. In this context, the super-technician is referred to as a Software Engineer or a Tech Lead rather than an Architect.

Bruce

Titles are thrown around with too much ease today so I hesitate to suggest that the best business analyst is a systems architect, but in a world where those people have learned their craft through experience and skill, the architect is the one with that experience and insight usually. In that case, the role of a business analyst is played by the architect prior to the architecture being crafted. In either case, the resource doing the business analysis should meet the specifications listed already.

When the business requirements are properly articulated, it should be clear to anyone within the business who reads the requirements, just what is being requested. The requirements do not state the design, though the requirements can highlight some directions for consideration within the design. For example: if the vision of management is for a customer self-service portal, then it is appropriate that the requirements say so. However, to suggest that a customer number be 20 alpha-numeric characters with a breakdown of meaning is not a business requirement, it is someone’s idea of how customers can be tracked and is best left to those experienced in these things.

The enterprise that understands the need for a coherent strategy within IT to meet the needs of the business, will invest in a business architecture consisting of a business activity model and a business information model. These models are NOT related to automation, but rather capture the essence of the business with respect to what it has to do, and what information it requires to do it. If these models exist, then all systems projects will have a firm foundation and place within the overall strategy.

In lieu of this kind of strategy, and lacking these models, a systems project is forced to do the best it can within the scope of its time and budget. With staff that has an enterprise vision, this approach can work, resulting in the enterprise business architecture incrementally. Unfortunately, this rarely happens. The best one can hope for in this environment is that the redundant parts of systems and standards don’t conflict too much with one another, or cause expensive data integration processes to be put into place to ‘fix’ the disjoint data and processes.

In conclusion, your business requirements are best understood when competent business analysts and architects work closely with those within your enterprise that understand the vision and direction of the business. Through this cooperation, a coherent and useful requirement document can be expressed that will serve your business, not only for immediate system projects, but for business requirements that are planned into the future.

Michael

The discussions about the position and role of Architects generate a lot of heat nowadays. Some enterprise managers still do not accept the project role of Architect over the Business Analyst. At the same time, the institution of an Enterprise Architecture and related Architects conquers traditional business areas more and more.

Technical Architects have brought a structural reasoning and formal logic into the business operations built on very personalized human interactions and relationships. This may be good for some business activities but not for all. Mechanical application of automation and formalization certainly works for tangible Value Chain methodology but people are not mechanisms. Neglecting intangible, interaction-oriented values of Value Networks makes business and technical innovations more difficult to implement and, thus, slower and more costly.

However, close cooperation with Business Architects can benefit not only IT Architects but also the enterprise itself. Such cooperation is not very productive over the border between Business and IT and requires a new organisational form. Both Business and Technical Enterprise architects have responsibilities covering an entire enterprise, which means that the only organisational form suitable to the task is a cross-functional and cross-departmental team where Business and Technical Enterprise architects work together. The authority of this team is equal or above the authority of separate business and IT departments and oversees enterprise level programs and projects.

The Enterprise Architect is at the head of the hierarchy of architectural function in the organization that cascades throughout all divisions and departments (in Business and Technology) down to each development project. Architects define what should be done at each level of organisation in accordance with the corporate Business Strategic Plan. The corporate management, in turn, concerns itself with how the architectural decisions and solution should be implemented. This means that when a project is set-up by a Project Manager, the related Architect verifies that the project goals and means are set and interpreted properly.

Technical Business/System Analysts contribute into the BR by finding and documenting requirement details and dependencies. Technical Business/System Analysts are not the “staff that have an enterprise vision” but rather focus on particular project needs. What other projects do and whether they overlap (in implementation) is usually beyond the scope. That is, relying on the BR, gathered only by the technical Business/System Analysts; IT has a high risk of successfully moving in the wrong direction because significant dependencies may be easily bypassed. Unfortunately, we have seen this occur too frequently.

I strongly believe that all business ideas, proposals, formulated needs and tasks that are intended for IT implementation, support and integration have to be verified and validated inside the business before being introduced to IT. Business Architects together with Business Analysts within the business have to construct a requirement control pipeline at the ‘entrance’ into technology. Technology Architects transform BR into architectural solutions and pass them to the Technical Business/System Analysts for enriching with details.

So, the pipeline may work like this:

1. A business need is formulated by the business requesters.
2. Business Architects and Analysts review the need, challenge it against Strategic Business Plans and corporate goals.
3. Business Architects and Analysts analyse the impact of realization of this need on the surrounding business execution context and vice-versa.
4. If collectively approved, the need is introduced to the IT Architects who consider IT capabilities with regard to this need and introduce several high level solutions back to business.
5. If IT solutions impact the business need somehow (e.g., optimizes it to some extent), a final agreement has to be reached via an iterative process.
6. Agreed needs are formulated as the core BR and passed to IT.
7. IT Architects and System Analysts work on the details and produce business/technical requirements for the low-level design and implementation in IT.

This is a universal business requirements construction process that can be applied in any industry and enterprise. This process is quite suitable for Business and IT governance and related quality controls. I think that potentially lost revenue from improperly implemented IT products highlight the worth of the efforts and time spent on defining real business needs in any marketing rush.

A Process for Building “good”/”bad” System

Bruce

Assuming that the ‘right people’ have been found and the requirements have been developed as described above; are we guaranteed to have a better system? Well, yes and no.

If the requirements are well developed, then we have a much better chance of getting a ‘good’ system than if they were not well developed, so from that perspective things are looking better. Unfortunately, there is still a significant opportunity to fail.

A report by Dr. Paul Dorsey highlights the top three reasons why projects fail as follows:

1. Top management support.
2. A sound methodology.
3. Solid technical leadership by someone who has successfully completed a similar project.

Interestingly enough, it does not matter what the ‘process’ was – in other words, the style of development was not an issue. Where then did this failure of management commitment become evident?

It seems that when a project begins, management classically has little patience for analyses. An expression ‘analysis paralysis’ gets bandied about, and time lines for analysis get sacrificed on the altar of ‘producing results’, as though a design was an absolute waste of time. I have worked with very strong technical teams that have ultimately failed to deliver a successful project, simply because management did not let them do their work. Unfortunately, the IT industry has created this situation as a result of using junior staff to perform tasks well above their experience level. In response, many methods have been conceived that get around the problem by simply not doing the analysis. Problem fixed! Well, sort of.

In the first few paragraphs of this discussion, it was stated that the method employed is not the deciding factor in whether or not a system project is successful. Holding to that concept, the methods: Extreme Programming, Agile, Rational Unified Process, and so on will not in themselves make things right. What these methods do is try to accommodate the already broken process, making the best out of a bad situation: and to their credit, they do that to some extent. Unfortunately, if the business requirement is not communicated well through a competent design, what will the developers develop? These methods put the business person with the developer in hopes that the requirements will unfold as they work together to build the system. That is fine if the business persons are exceptional in their understanding of the enterprise and its vision, but not so fine if that is not the case. There must be a vision of what the overall system is to become. That is not to say that the entire vision must be fully articulated prior to any work being done – this vision can be incrementally grown as long as the keeper of that vision (the architect) is competent and knows where s/he is going with the design.

The question is, what makes a system ‘good’ or ‘bad’ for the business? Who judges this, and is it really a black and white judgment?

In reality, any system no matter how poorly conceived, generally accomplishes at least a part of what it was intended to do. It may require more money and time than originally planned to get there, but most systems do get there in some fashion. Unfortunately, a poorly conceived system has a very little capability of making sure the business can grow and will not be able to meet its needs over time.

A ‘good’ system is going to be defined here as a system that meets the business requirements, allows the business to change over time with agility, and is seen as a true enabler of the business by its users.

Conversely, a ‘bad’ system is defined as one that may accomplish some or all of what the business needs to do today to some degree, but does so in such a way as to hamper the ability of the business to change how it does things tomorrow.

Michael

My interpretation of a ‘good’ technical system is tied to the Execution Context (business and technical). I have noticed that in a prosperous time, business changes in the external business environment happen less frequently than in downtime. Depending on this, business sets different expectations and requirements for the same technical system. For example, at the end on the 90s, in the USA, system scalability and performance were the major objectives; nowadays system flexibility/extensibility and security take the top priority.

Business flexibility now demands that technical solutions would be flexible enough to adopt business changes with minimal investments in system alteration, and with minimal investments into the alteration of surrounding systems, and all together should be done for minimal time (this is how I define system ‘flexibility’). This is in a sharp contrast with the old days ‘pragmatic’ practice, which preaches that given requirements must be delivered ’in stone’, with no ‘hooks’ for modifications and replacements like there will be no changes literally coming tomorrow.

The natural process of the implementation of business needs in IT is usually broken and it is not necessary the fault of IT. IT has constructed and inherited a lot of old legacy systems with multiple patches and ‘spaghetti’ code. If the business wants to move forward quickly, it has to fix the IT assets making them flexible. This requires time and investments that are usually absent. Cloud Computing and outsourcing will not help much in this because the old legacy systems in IT carry the corporate treasure – knowledge/logic and data, which is not easy to outsource.

Quoting Forrester Research, let me say that “your business can change only as fast as your technology can”. So, a ‘good’ system is a flexible system that is useful today and allows for extensibility into tomorrow. In ‘agile language’, a ‘good’ system meets business needs for daily activities and smoothly transforms in line with the Business Strategic Plan.

IT generates one agile method after another just to build good systems. Majority of agile methods are based on assumption that business cannot and will not work differently than it does today; these methods do not dare to challenge the way how business works even if this way is contra-productive for the enterprise. This leads to the trend where many “agile practitioners advocate simple design where there is very little or no emphasis placed on upfront architecture and design”. Indeed, the development team is accountable for the delivery of a piece of software in accordance with the Business Requirements; the business value of this software does not concern the team: “If the Business requested this they know what and why they do”.

It is not a rare situation where agile teams promise to add an error handling mechanism, escalations, notifications/alerts, compensation logic, transactional integrity, security and operational stability later on, in the following “time-boxes”. However, all these technical things are IT concerns; they are attributes of Quality of Service (QoS) that the business expects from technology right away, not next time. The business logic is straightforward: “if the software works already [well, does it really?], why wait longer and invest more into the ‘final’ release? Instead, since IT works that well, let them do another task.” This is how agile teams set the traps for their own business and then suffer from overwhelming work of fixing ‘old’ problems when ‘simplifying’ the new ones.

Getting back to emerging Lean methodology, it is important to review real lean manufacturing principles formulated by Taiichi Ohno:

• build only what is needed
• stop if something goes wrong
• eliminate anything which does not add value.

Undeniably, these principles are very rational and may be easily interpreted in the manufacturing environment where final result is fully defined and specified. However, Mary Poppendieck and Tom Poppendieck, when formulated 7 principles of Lean project management method, had deviated from the lean manufacturing in several points.

Let us look, for example, at how Lean method would work in the SO ecosystem where a Lean Team builds a SOA Service. Based on amount of attention put on the Lean principle “Eliminate Waste”, it is the first task for the Team. By the way, eliminating waste in the software development is much more challenging task than in manufacturing because there are many uncertainties in any Business Requirements and architectural vision does not necessary reflects in them. So, if we hate and neglect up-front architectural design, how can we justify what is the waste and what is not?

In SOA, any service may be used by any consumer while the one, who initially requested the service and whose requirements were used to construct the service, may be forgotten a while ago. SOA Service must serve unknown users or, in Lean terminology, customers. Every customer can find its own value in the service and this value may be very different from the one targeted by the development team. Does it mean that the manufacturing definition of waste does not work in service-oriented development? Or, maybe, it does not work in any SW development without prior strong architectural design...

Another Lean principle – “Delay Commitment” – conceptually contradicts the principle of “Eliminate Waste”. We might delay commitment, i.e. delay completion of product implementation, because we expect some business changes in our tasks. If we anticipate changes but do not know what they are, how can we identify waste - one thing may be the waste while another one may be very useful feature for the change to-be. I think that the service-oriented formula – “design for changes” - works much better than delayed commitments.

Two other principles – “Build Integrity In” and “See the Whole” – may question the principle “Empower the Team”. The Lean Team is supposed to focus on particular task, which makes the ‘whole’ difficult to observe. And it is even more difficult to build integrity with other elements of IT that are not in the scope of the project. Both these principles belong to the cross-team architectural function rather than to the development team; it is the Architecture that is in position to define the waste.

Finally, it is interesting to note that the principle “Empower the Team” (versus empower the management) is added into the IT Lean method while the original principle ‘Stop if something goes wrong’ is excluded. Does this mean that only ‘sunny days’ in development and execution are anticipated? Without the stopping principle, empowered team and the Lean project manager do not have a mechanism to retreat, i.e. they continue delivering whatever available, including the waste. Thus, there may be only one conclusion out of two choices: either the IT Lean method is still immature in its foundation or it is a wrong method, again.

Can we do better?

Bruce

So, in this discussion we are saying:

• the method isn’t a deciding factor in success
• the business needs a business architecture
• the business should specify need, the architect/analyst should specify systemization of requirements
• management needs to endorse a method and support it consistently
• the measure of a good system is found in its ability to enable the business.

How do we get there? The harsh reality throughout the history of the IT industry is that you cannot always find the best people, and have to make do with less experienced or less capable staff. Is there any hope in that reality of producing good systems? I believe there is.

If there is a good business architecture that is well articulated, then there is a solid basis for moving forward with requirements. If there is a solid application architecture built upon that business architecture, then the specification of projects can be done in a controlled and progressive manner.

If there are instituted sound architectural practices and guidelines, and an expressed enterprise vision that is supported by senior management, the rest falls into place without as much reliance on the experience and skill of the project teams simply because there is a solid framework already defined within which they work. This is not a perfect approach, but that is the reality we are faced with.

Is it possible to provide those ‘necessary’ pieces without taking years of time?

If the right people can be found to model the business, it can be modeled fairly quickly. Regardless of the business (government, manufacturing, military, etc.), a team of two modelers, working together, to capture the activity and information models, can gather those models quite quickly. In a medium to a large environment, a six month project should suffice to accommodate the need.

The derivation of the application architecture beyond that could require another 2-3 months depending on the coherence of the senior management vision and direction. The time frames involved are subjective and very much dependent on the culture of the organization and the degree to which the organization is buried in legacy applications.

The recognition of the things discussed above can lead to better and more consistent development projects. But there is another aspect of this that can free the business from decades of poor implementations and redundant systems.

The application architecture spoken of above includes an inventory of all of the applications currently in use within the enterprise, and an assessment of how those applications assist or hinder the real needs of the business. Taking a more business requirement focus when looking at these systems provides opportunities to consolidate based on function and on data. Reducing the number of applications in the portfolio can not only reduce operational costs, but provide significant benefits through more reliable and timely data, and a much more responsive development environment to meet the challenges of business change.

Michael

My answer to the question of this topic is ‘Yes, IT can do better but not alone’. IT has to involve the business in:

1. All decisions made with regard to technical solutions.
2. Capability reviews toward business strategies.
3. Joining development and maintenance funding.
4. Evaluation of produced technical systems.

Such involvement has to make the business responsible for its needs and preferences.

When technical architects propose a few alternative solutions to be implemented, the clients and stakeholders have to know that if they choose the cheapest but not forward-driving solution, they will be held responsible for this decision, no IT. I have seen so many situations where the cheapest solution in development became extremely expensive in maintenance. How can one balance these two? It is easy – the business and IT have to target the cost of ownership of the technical solution. Targeting the project/development cost alone has to become the absolutely exceptional cases.

When IT reviews its technical capabilities, it is not a rare finding that technical platforms are old if not obsolete and may be not even supported by the vendors any more. Can such platforms properly contribute into the business strategic goals and objectives? The answer is, probably not. That is, IT has to say in open that available systems do not support strategic business needs demonstrating this with the “cost of ownership” of existing solutions whose maintenance consumes the major part of the IT budget leaving very little for strategic development.

When a new solution, product or system is developed, it has to be reviewed by future business consumers. Traditional User Acceptance Testing usually does not check deeper than the business functionality. In contrast, the business review has to challenge IT products in all of the exceptional or non-standard business situations, for example the case where scalability has to be demonstrated in the real environment rather than in a dedicated test system. If the system passes the review, it will live well; if it does not, it is possible that some business requirements might be refined. In any case, IT together with the business, has to re-set the business expectations and identify improvement directions in the collaborative and proactive manner rather than fighting with reports about bugs in production.

How can IT involve the business more? Probably, I’ll be not that original if I say: a) support and promote an idea of Business Architects in Business, b) support and promote the idea of cross business-technology Enterprise Architecture, instituted by mixing Business and Technical Architects, c) setup a periodic exchange of Business Analysts between the business and technical teams working in the same business domains, d) create a Director-level structure in IT that would be working directly with business management as a pipeline for all business requirements intended for IT, and e) create a cross-project analytical group with responsibilities to analyse low-to-middle level business operational processes and come up with optimisation proposals based on technical capabilities, i.e. make IT proactive.

I see one more aspect of potential IT improvement in Technology Governance. I really mean Technology Governance that covers all Architecture, Development, Data/Information and Management governance areas.  This governance is about not only what-is-what and which technologies may be used in the development and testing. This governance is about architectural and design controls over development, development and testing tools and methodologies, project management methods and collaboration over existing and ‘work-in-progress’ systems, products, and solutions. The governing policies have to cover IT involvement in the end-to-end business development, address solution ownership objectives and responsibilities defined in the SLA between IT and Business. Governance may not be substituted by management: governance says ‘what’ and ‘how’ things should be done while management enforces governing policies. If governing policies were be preserved by IT management, there would be a chance to streamline agility to the business via steering the entire IT function from the single authoritative governing position.