Architecture Is Objective, Design Is Subjective November 2009

In my previous column, I said that I was going to discuss coherence boundaries, but instead I’m going to digress and talk about “mere implementation detail” in more detail (if that’s not an oxymoron).

Over the years I’ve had a number of conversations (you know who you are) about my tendency to dismiss all sorts of side issues as “mere implementation detail” when producing an architectural framework. In fact, as a result of this series I’ve been contacted by a couple of people telling me that I couldn’t ignore detail – essentially saying it was impossible to produce a robust architecture unless you’re in command of all the facts because of the risk to the business of overlooking some important detail.

That statement is both true and false depending on what you regard as important detail!

It’s true in that the more knowledge you have about a subject then the more likely you are to make a correct observation about that subject because incorrect observations will conflict with what you already know. However, I’d also argue that making correct observations can become more difficult if you have too much knowledge – basically there is a threshold beyond which any more detail just gets in the way.

So this time around I thought I’d explain my thoughts on “mere implementation detail,” the problems it causes and why it’s important to avoid it in.1

What is “Mere Implementation Detail” Anyway?
“Mere implementation detail” means “information only important at a level of detail below this one” or “something I don’t need to tell you in order for you to understand what I mean.” In terms of carrying out or implementing any operation, there is always lots of additional detail that could be recorded or discussed about that operation that will be pertinent to some people but completely irrelevant to other people.

The trick is to get the level of detail right so that those that need to know something do know it and those that can get by without the detail aren’t forced to consume it.

This is true no matter what the subject area is or what level of detail is being discussed. Consider the scenario for buying a DVD Player:

In the above, there are three distinct interactions where information is exchanged (1) between my wife and me to establish the requirements; (2) between me and the Salesperson to establish a suitable solution to the requirements and (3) between the Salesperson and the Supplier to get the chosen option delivered.

Each interaction has a purpose in order to complete the overall process, but the detail needed at each stage is different. For example the Salesperson doesn’t need to know why my wife wants a DVD player but does need to know the basic functionality required. My wife in turn doesn’t need to know who the Supplier is.

From this example, we can see that unnecessary information may be propagated in both directions, but I think most people would conclude that the process of buying something like a DVD Player is much simpler if (a) only the key parties are involved in each stage of the process and (b) they only discuss the details necessary to get past that stage.

So removing mere implementation detail is not about avoiding questions or not thinking things through properly but is about only providing information when it is needed at the point that it is needed and leaving the next level of detail to be dealt with by the people best suited to the task.

Essentially if the additional detail doesn’t materially change the decisions reached at the upper level or change the approach taken at a lower level, then it is mere implementation detail and can be pushed to one side.

The Traditional View of Architecture
Before we can talk about the problems caused by mere implementation detail, I think it’s worth explaining my view on where Architecture fits into the scheme of things.

As I mentioned all the way back in Part 1, I think the core of [Enterprise] IT architecture is about patterns, principles and component frameworks and the high-level IT concepts such as master data management, business continuity, security and authentication, or information quality that need to be applied across the whole domain.

It is not about project management, requirements engineering, business analysis, application design or application development. All of them have an impact on the architecture and are certainly a consideration that the architect needs to discuss with the appropriate people, but they are all separate functions that influence the architecture but are not specific architectural activities themselves.

If we were following a traditional waterfall development approach with structured project management, then I believe the relationships between project management, architecture and development would look something like this:2

In this model, the IT architecture activity, like project management, should run alongside the actual IT life cycle rather than being embedded in it.

Although the diagram shows a single process box for “Define Framework,” most of the popular architecture frameworks subdivide this into multiple architectural views, e.g., the Conceptual View, the Logical (or Platform Independent) View, the Data View, Component View, Process View and so on, that elaborate and provide more pertinent detail as we move forward through the application development life cycle.

So, for example, in the Zachman Architectural Framework we have:3

So, in the traditional “waterfall” development environment I think it’s pretty clear-cut how everything fits together, and it’s this view that I keep in mind whenever I’m producing any form of IT architecture.

How to Avoid “Mere Implementation Detail”
Unfortunately, the only way I know to avoid inadvertent mere implementation detail is to be vigilant and maintain an objectives-based approach to creating the IT architecture focused on producing the least amount of documentation communicating the most amount of information.

This is the reason why I think architectural patterns should be expressed as diagrams and definition of terms and architectural principles should be captured in a shared and publicly accessible glossary. These are both excellent methods for reducing the size of the core architectural framework and allow secondary details to be elaborated over time without having to modify what has already been published.

In addition, the architectural framework should only describe the approach that is going to be taken. If there is a need to document the reasons behind the choices that were made or describe alternatives that were considered but discarded, then these should be placed in a separate document and put in a separate branch of the project archive. They are really just for intellectual interest and post-project reviews.

Recommended approaches can be documented separately as design guidelines that development teams can either choose to follow or ignore. Being just guidelines, they are open to discussion and change even after the architectural framework has been established because nothing about them is a commitment to actually follow the guidelines.

This is why I emphasised the difference between MUST and MAY in the article on architectural principles.

However, in order to be vigilant, it is worth understanding the many reasons why unwanted detail can creep into an architectural document and the following list is by no means an exhaustive one but does describe the ones that I think occur most often.

“Information Overload” and “Information Pollution”
The problems of information overload and information pollution from the perspective of the information consumer are very closely related.

Amongst the causes of this are:

• A rapidly increasing rate of new information being produced (e.g., publishing a new revision of a 50+ page document distributed to dozens of people “for your information” every time something changes no matter how trivial the change).
• Large amounts of historical background information to dig through (e.g., providing lots of contextual information or always explaining in detail how things were previously done as a preamble to explaining how things will be done in the future).
• Pieces of information are unrelated or do not have any overall structure to reveal their relationships (e.g., documents don’t guide the reader properly from “introduction” through to “conclusion” or levels of detail don’t cascade from highest level of abstraction down to lowest level of detail properly).
• Contradictions and inaccuracies in available information.
• A low signal-to-noise ratio (e.g., when the messages are unfocused or unclear and make it difficult for the reader to understand the point that is being made).

Most people nowadays are employed in busy, almost chaotic, work environments where there are constant demands on their attention and everyone is doing a dozen things at once. If we accept the findings of some researchers that more than 50% of the working day is spent in “unproductive activities” (reading email, answering phones, paper work, getting refreshments, discussing non-work related topics and unnecessary interruptions breaking thought flow), then the idea that people can spend hours reading background material is a bit of a myth.

In addition, people use information in order to make decisions and adapt to circumstances, yet cognitive studies have demonstrated that there is only so much information human beings can process before the quality of their decisions begins to deteriorate.

Most people have a limit on how long they can concentrate fully before their minds start to become fatigued – it’s generally around 45 minutes (mine is definitely less than an hour) after which the ability to absorb information starts to degrade.

Consequently, balancing level of detail provided against the time available to the consumer of the information is an important consideration. Including “mere implementation detail” in architectural documents results in both information overload, because we are supplying much more information than is required to understand the architecture, and information pollution, because we are telling the reader irrelevant detail.

The excess of information can even lead to “decision paralysis,” where the reader is unable to make a judgment about what is being presented to them because they cannot see what is relevant anymore or, even worse, make the wrong decision because it was (a) their responsibility to make a decision and (b) they didn’t wanted to appear incapable or indecisive by admitting they didn’t understand everything that was given to them.

Unfortunately, when “information pollution” occurs, we often exacerbate the problem by producing more documentation to explain the stuff that we failed to explain correctly in the first place in the misguided belief that providing more detail will make it easier for the reader to understand the point.

In fact, those that already understand mere implementation detail might say that a 4000+ word article, such as this one, on the subject is information pollution itself.

It is, however, definitely not the case that more is better when dealing with information overload or pollution, and the problem is generally best avoided by doing the exact opposite and removing the irrelevant detail rather than explaining it further.

Analysis Paralysis
Analysis paralysis occurs when the time spent analysing a problem outweighs the benefit that is gained by reaching a more precise solution and is generally a side-effect of information overload” on the part of the architect. It’s the complement of decision paralysis, and most of us will have met this problem at one time or another!

There are many causes of this such as:

• Having too many imponderables (i.e., questions that we don’t yet know the answer to) and, as a result, we just keep asking more and more questions at greater levels of detail in the hope that we finally arrive at an optimal solution.
• Being under pressure to “get it right first time” or, conversely, a fear of getting it wrong and having to explain why a mistake was made. This generally manifests itself in architectural documents that contain extensive justifications as to why particular approaches were taken.
• Scope creep that continually causes us to re-evaluate what has already been documented.
• Having too many options and not enough experience. Production of “option papers” that provide detailed discussion of choices that are available without any clear opinion about which option is being recommended. (I think an IT architect should always have a preferred option even if it’s the wrong one!)

The result of analysis paralysis is that the IT architecture is never finished – when asked about progress there are always a couple more issues that need to be addressed or a couple more questions that need to be answered. Nearly always they are very obscure or involve convoluted scenarios that will rarely, if ever, occur in the production environment.

In systems development, analysis paralysis typically reveals itself through excessive iterations of project planning, requirements gathering, program design and application modelling, with little or no extra value created by each iteration other than a lot more documentation.4

Okay, sometimes we have to answer unnecessary questions in order to get sign-off of the architecture from the business stakeholders and any other affected parties, but just because a question is asked does not mean that the architecture has to document the answer.

Architectural Confusion
“Architectural confusion” occurs when the architectural framework is used to address things that are beyond the traditional scope of architecture.

For example, let’s return to the Zachman Architectural Framework diagram:

At least two of the architectural layers, the “Functioning Enterprise” layer and the “Objectives” layer might not be regarded as architecture at all. “Functioning Enterprise” is actually the physical realisation of the architecture and the “Objectives,” being the encapsulation of the Business Strategy, is actually an input to the architectural framework that is defined by the business stakeholders rather than the architect.

We could even argue that the “Detailed Representation View” is the traditional software design activity (part of the software development life cycle) so also shouldn’t be considered as part of the architectural framework deliverables.

However, ignoring that, what this framework doesn’t give us is a prescriptive list of things that should and, more importantly, should not be included in the architectural documents produced for each of the cells. For example, there is no clear-cut definition of the separation between the Conceptual and Logical Views or whether data, such as configuration parameters, that is used solely within a specific Function to control its execution, should also be included in the Data View.

As a consequence of not having clear separation of architectural views and a set of rules for how each of those views should be defined, it is it is not unusual, for example, to see the business information model (which is supposed to be an enterprise-level conceptual model) contain data items such as system generated keys and “auditing timestamps” (i.e., create date and last update date) or use platform specific data types, such as “int” or “long” that have little meaning to non-IT stakeholders.

There can also be a tendency to replicate a piece of information into multiple cells just in case it belongs there which, of course, manifests as mere implementation detail when, for example, information belonging to a lower architectural layer is included in a higher layer.

Unfortunately, this is an issue that has been exacerbated by the widespread adoption of UML because a common notation is used for all architectural views so it isn’t always clear just by looking at a diagram exactly which view is being represented.

For example, a business information model can look exactly the same as a software class model or an interaction diagram between people (actors) can look exactly the same as an interaction diagram between functions even, in some cases, with the same labels, such as Customer, being used in both diagrams.

Of course, the more places we replicate a piece of information, then the more places there are where it needs to be updated when it changes, which will eventually lead to information pollution due to inconsistencies in the architecture.

Omitting an Architectural View
This is a much more extreme case of architectural confusion that occurs as a result of the predominance of rapid application development approaches (not to be confused with RAD, the methodology) underpinning most modern systems development methodologies.

In RAD methodologies, we have a bunch of “mini-cycles” – in “Agile Development” these can be as short as a two week project interaction – that exist within the overall systems development life cycle, where for individual components within the system we might go back to previous stages in the development process, e.g., add a new requirement and change what was previously thought to be finished and deployed.

Another feature of Agile Development is the “You Aren’t Gonna Need It” (YAGNI) principle which argues against worrying about things that you don’t need now because in the long term you probably aren’t going to need them anyway.

With YAGNI short-term tactical solutions are emphasised rather than long-term strategic requirements and many of its proponents have a strong feeling that “big architecture” is a waste of time because the deployment system will always be different and “the code is the documentation.”5

Consequently, because architecture is often seen as a project overhead, there is a tendency in the rapid application development approaches to reduce “time to market” by rushing through its production usually by combining two or more architectural views into a single view.

The most obvious cases of this are combining “conceptual system model” and “logical system model” into a single “platform independent model and combining “technology model” and “detailed representation” into a single “platform specific model.”

This certainly simplifies the production of the architecture an awful lot, but unfortunately results in more detail than is necessary appearing in the higher architectural layer than normal; and given that each architectural layer has a different target audience, this results in revealing unnecessary information to people that really shouldn’t need to see it.

Micro-Management of the Implementation
As many people have observed, information technology is full of “clever people”. As well as being pretty good at solving problems, they generally also have good knowledge retention and flexible thought processes – they build their own experience quickly by readily adapting what they know to fit new problems.

In fact, the IT architect may well be employed because they have prior experience of a specific solution to the problem area being addressed; and as a result of the pressure to get it right first time, there is pressure to just repeat what previously worked. This nearly always manifests itself as an over-elaboration of the architecture in order to micro-manage what the final solution will look like.

This is reasonable behaviour but unfortunately fails to take into consideration that:

• Technology and technology platforms evolve so last year’s state-of-the-art technology platform from Vendor A may well have been superseded by the latest capabilities of the equivalent platform from Vendor B.
• The development staff will have their own thoughts on how best to implement something and constraining them too much will create unnecessary resistance to what is being proposed. In the worst case, they will just ignore the architecture, do what they think is best and argue about it afterwards.
• The previous solution wasn’t really as perfect as you recall it being and you’re probably not aware of all the things that needed fixing after you left the project. As my dear old grandmother used to say, “Always make new mistakes” – history should be learnt from but never repeated.

For example, I recently reviewed the technical architecture that someone else produced. The architectural principles contained a statement about components being loosely coupled (a good principle!), but this was accompanied by a highly detailed specification describing exactly how all service users (the software that calls the service providers) would implement this loose-coupling using XSLT.

There was no doubt that that it was a really flexible solution and would add a lot of robustness to the implemented application. However, it ignored the fact that depending on the programming language, there might be better way of doing this (e.g., JAXB v2.0 has excellent dynamic marshalling features that a specialist Java developer would probably have better knowledge of than they would XSLT).

Anyway, the approach was technically accomplished but thought that it should be a design guideline rather than an architectural pattern because there were alternative options that could be employed and which option was utilised would not materially affect the overall structure of the architecture.

We had a bit of a disagreement over this but eventually agreed that the architecture didn’t lose anything by removing the unnecessary detail but did gain by removing 10 pages from the finished document (reducing information pollution) and allowing future flexibility in the implementation (design for change).

Conclusion
Hopefully this explains my thoughts on the need to avoid mere implementation detail – essentially we’re talking about avoiding unnecessary work until it is necessary, subverting our egos and staying focused on the architectural deliverables that are important in order to help other people do the things that they are responsible for.

So mere implementation detail is not about avoiding questions or not thinking things through properly but is about only providing information when it is needed at the point that it is needed.

From an architectural perspective, only providing information when it is needed has advantages in that:

Business stakeholders usually don’t care about technical details of the design – the attention of many an audience will be retained by not boring them with more detail than they are interested in.

It improves “time to market” by reducing the amount of work needed to elaborate a given architectural view before passing down to the next, more detailed level in the architectural framework.

The merits of each architectural component can be quantified and measured whilst design aspects are debatable. Removing design from architecture increases the chances of reaching general agreement over the approach without subjective arguments over technical merits.

The objective aspects are a matter of necessity to operate the business whilst subjective features can be modified, superseded or replaced over time without impacting on the nature of the business.

In my next column, we’ll look at the coherence boundaries in more detail because, believe it or not, there’s a lot of interesting architectural behaviour that takes place at the boundaries that needs to be taken into consideration.

End Notes:

1. The converse of mere implementation detail is significant architectural detail – I really enjoy talking about the SAD parts because I’m a SAD person.
2. This intentionally doesn’t reflect the iterative activities that take place in most systems development processes (the systems development life cycle) even within a “waterfall” approach. From the perspective of this diagram, they can also be treated as mere implementation detail . 
3. There are many published diagrams that are attributed to the Zachman Framework that describe the framework in different business contexts. The one shown here is the one that I think best describes the Zachman Framework as applied to information systems, but there are others that are just as applicable.
4. PRINCE2 has some very interesting things to say about this throughout the manual – it’s well worth a read even if you have no intention of ever using PRINCE2 for project management.
5. The “code is the documentation” approach misses the whole point of IT architecture, which is not only to describe to the business what we intend to build but to also have a blueprint to confirm that what was built is what we wanted – this is the “apply governance” stage. Without it, the business can only hope that they get what they need.