SOA (service-oriented architecture) creates enormous opportunities for improving business information handling, although it can also be a difficult subject to get your head around. No spokesperson
for SOA can compare with Ted Codd, who described data modelling in the early 1970s, or Michael Hammer, who has been a proponent of process-oriented business management since 1990. You don’t
need to agree with everything they say, but any conversation on the subject must always take these advocates into account. All leading IT players thus feel compelled to present their own view of
SOA. The risk is, however, that the more you read and learn, the more confused you’ll become.
This document seeks to give you about a solid grasp of the basic building blocks of SOA. It should, of course, be read with a critical eye, but it will give you a basis upon which to you can relate
the various viewpoints and propositions. This document builds on Thomas Erl’s brilliant book, SOA: Principles of Service Design and IRM’s work on SOA in
business. Erl comes at it from the technical side, while we at IRM approach the subject from the business side. Hopefully, it will be a fruitful meeting, one where we can make beautiful music
together.
It is difficult to find a common theme because for the first time it now spans all the way from the corporate vision to the technical solution. We all have certain areas about which we are
knowledgeable, more or less. This paper seeks to describe an interrelated process within which you can immerse yourself in the things that interest you most. As an SOA specialist, the bigger
picture is primary; more in-depth detailed knowledge is the next step.
You should also bear in mind that technical development, be it for car engines, mobile phones or SOA, doesn’t make things less complicated, but hopefully improves their functionality. SOA
requires more training, experience and competence than ever before, much in the same way that maintaining a car engine is today more demanding than it was previously.
Introduction
Any IT solution should support the business in the most appropriate way and should be easy to adapt as business needs change. One of the goals is increased flexibility. But at the same time this
requires increased traceability across the board so that we can ensure that an SOA solution provides the business with the right kind of support. This is akin to the way that urban planning
requires that buildings are built to the specifications in permits that are granted in accordance with an approved city plan. Without a trackable result, all plans are fairly meaningless.
Enterprise Architecture
The process of establishing SOA starts with the architecture, which provides a foundation upon which to successively develop SOA services. An enterprise architecture plan is developed in three main
steps. The first step is a process map of business operations, followed by an overview data model and, last, the architecture itself, which is described in an IRM matrix. The matrix describes how
the basic data building blocks are placed within object groups, as well as identifies the dependent variables existing between the various object groups. Our starting point is that each
object group can perform an SOA service.
Our basic premise is that the business has developed clear visions and strategies that describe how it should develop. We call this a business plan or operational plan. When we start building the
service-oriented architecture, those services that are most important for the business’ visions and strategies are a logical starting point. In this way we have a development project
supported by and actively involving management.
Hammer and Business Processes
Michael Hammer launched the idea of business processes in 1990. What makes increasing numbers of businesses work with their processes customer focus and the ambition for a high level of customer
value has led an increasing number of companies to concentrate on the development of business processes. Customer focus increases competitiveness and improves business efficiency.
We go through the business plan and examine which of the company’s processes are most important to develop in order to implement the visions and strategies that have been put into place. We
call these strategic processes, because they speak to the company’s underlying efficiency and competitiveness. These are important to identify and bring to the attention of
management. Projects that relate to these strategic processes are allocated resources and are on the radar of management.
Zachman’s Framework
This document is also related to Zachman’s Framework. He builds this on interrogatives, whereby we prioritise why (the business exists), how
(the processes) and what (data and information). This order is important when discussing with the company’s management. We will leave the other questions of
who,when and where aside for now.
When Zachman developed his framework while at IBM in the early 1980s, he prioritised the data (what) column, something that he found it lacking in his work at IBM. This meant that business systems
planning (BSP) at that time lacked stability and trackability.
Traceability is easiest to clarify in the data column. It has a mathematical stringency that is lacking in process models. Ted Codd, of course, was a mathematician. This is why traceability in the
“what” column can support trackability in the “how” and “why” columns. Everything so that the plan can be compared with the solution.
Codd and the Data Model
That is also why the next step is to decide what data is needed to manage strategic processes. The processes change from year to year, but the data are stable and therefore create a vital and
decisive foundation for ongoing work. We develop data models (sometimes called information models or object models), that show how the crucial business concepts relate to each other. Codd, who
developed the theory of normalization, continues to be a key figure in the development of high-quality data models. This is difficult and requires a great deal of training, yet it is crucial for
successful SOA activities.
SOA Services
We are now coming to the most central point in SOA – namely, the services (or service components). LEGO blocks provide a good comparison. The services must be stable and have clear interfaces
that enable the pieces to fit together. It is important that you do not confuse SOA services with the services that the business delivers to its customers. One of many points of confusion is when
we start to arrive at SOA. The interface is standardised and published. When you use the service, you are using the published interface.
The services have functions that manage data within the business. An example of a service is the customer. Here we manage all customer data, such as identity, name, address and telephone number,
and credit status. We often say that the services are CRUD-oriented. CRUD stands for Create, Read, Update and Delete. The service
has at least one interface for each CRUD function. But there can be several different interfaces, especially for reading. When you require information on a specific customer, you ask the question
in a standardised and previously determined manner. You also state that you are authorised to ask the question. The service then responds in a preset way with information such as name, address and
telephone number.
There are two options: either a large number of fairly minor services, or fewer but more comprehensive services. We recommend the latter option. The experiences published to date in books, for
instance Erl’s, indicate that this is the correct approach.
Thomas Erl
How we design the services is quite naturally the most fundamental question when setting up SOA, and so this is a good time to stop and spend some time on this issue. Of all of those who have
written books, articles and presentations about SOA, Thomas Erl remains the main proponent of SOA.
Thomas comes from the technical side of things, but has successively gravitated towards the business side. He lives in Vancouver and has written four major works on SOA, of which the latest,
SOA: Principles of Service Design, is an impressive attempt at transforming tacit knowledge into explicit knowledge. See footnote 1. Sort of like how a
baker writes his recipes. To understand and evaluate whether it is a good recipe, you have to actually try it. The same applies with Thomas’ explicit design rules – you can only
understand and challenge them if you know a fair bit about SOA. Because the rules are explicit, they can also be developed in the same way that a recipe can be refined and adapted in your own
kitchen to suit your personal tastes and needs. When they are explicit, they can also be distributed so that more people learn and use them.
The Eight Basic Principles
In the formulation of Erl’s eight basic principles on proper services design, some are easier to understand and others are more difficult. When designing your own services, it may be fair to
reflect upon these eight fundamental ideas. None of them are black or white and can be implemented 100% or left out altogether.
The four first basic principles relate to the business architecture and derive from an IRM architectural plan.
- Reusable services
- Service structure (service composition)
- Standardised interfaces (service contracts)
- Services catalogue (service discoverability)
The other four basic principles relate to the design phase:
- Loosely coupled services
- Limit information about the interface (service abstraction)
- Autonomic services
- Service statelessness
Basic Principle 1 – Reusable Services
Let us take a closer look at one of the basic principles, “reusability.” Ideally, this would mean that the entire business has only one service for handling, for instance, customer data. In this
case, the customer’s address would be located in only one place. The other extreme is that anyone can create and develop their own customer service and define customers entirely according to
their own specifications.
Most people would probably agree that a common customer database is desirable, but that this can be difficult to achieve. To have the same data in many different systems and databases can cripple
IT development. An increasing proportion of an IT company’s resources are used to maintain and integrate legacy systems. This trend must stop and the most important action is to avoid
allocating the same data several times into different systems. This, in turn, leads large volume of data being sent between different systems.
If we can increase reusability and create a common customer service, this could reap massive rewards such as reduced costs (certainly over 50% for the majority of larger
businesses), higher data quality (better results) and, perhaps most importantly, faster development of IT solutions that can keep up with the development of the
business.
With a large degree of reusability, we are talking about 25–50 more comprehensive services and with lesser degrees of reusability and smaller scale services we are talking about
perhaps thousands of services. Strategically, this is a critical business decision, and one which should be made by management. This decision impacts critically on future IT solutions and the
company’s ability to compete and manage future change.
We strongly recommend reusability and fewer, more comprehensive services. But what does it take to get to that point? Creating common solutions requires, of course, that everyone in the business
can use them. To date, this has often been hindered by different IT solutions being located on different technical platforms. This, in turn, has prevented the implementation of common solutions.
Now that this obstacle is disappearing, what challenges remain? We must base common services on the data structure because it is stable and based upon mathematical principles.
Processes are not as stable and do not have the same theoretical basis. Processes, which focus on customers and their values, are critical for the business’s competitiveness, but
processes must not be used to design services. We can regard this as a strategically decisive choice of the same magnitude as that regarding reusability, as described above.
If we accept that designing fewer, larger services is based on a stable data structure, how then should we proceed, and what are the pitfalls? It is possible, with a high-quality data
model that has well-defined terms. Since Codd launched his theory in 1970, development has trailed behind and the data model has not circulated as it should. It is easy to get caught in a
vicious circle, whereby poor data models do not deliver their promised benefits, and so we stop developing data models altogether. SOA needs a “positive circle,” where we produce good data models
that do deliver the benefits they promise.
Sweden leads the world in data model use. Ever since Linnaeus and Berzelius, structure and a certain degree of order have been accepted as sound basic principles. See footnote 2.
Driving Forces Behind SOA
Before we continue with the SOA process itself, this may be a good point to look at the driving forces that have made SOA such a big deal. To start with, we will limit ourselves to two of them.
Global Competition Based on Speed
Most large companies are now global and have customers across the world. They are constantly subject to faster changes; product life cycles shrink, production needs move elsewhere, sections of the
organisation are sold off and new ones are acquired. The most important changes are probably still ongoing development and changes in the business’ processes. A dinosaur of an IT system is
therefore unacceptable.
The fact that IT heads and CIOs change increasingly often is not helping the situation. Something more radical must be done. A well-implemented SOA initiative may be the solution. This is where
you, as an SOA specialist, play a critical role. This can come about partly by having a fundamental understanding of the entire SOA process and partly by being strong enough to resist the constant
opportunities to cut corners. Embrace the “good spiral” and avoid the bad one.
Avoid a “Big Bang” with Successive Implementation
Most large companies have a system portfolio that can often be spaghetti-like. This is largely self-inflicted by a passive IT business, but is also the result of the changes described above.
Getting an overarching grip on things and obtaining a better system structure right away is of course enticing, but almost always fails. Large projects have an implementation factor of around zero
and will scare off most top management.
Sometimes implementation of a business system succeeds and replaces several individual systems, after considerable effort. At the same time, the risk is that the business will be locked into
standardised solutions that obstruct development of new competitive processes.
SOA offers a successive and more risk-free transfer from spaghetti to something that American professor David Robertson in Lausanne describes as ravioli. Compare with the LEGO pieces above. This
means that we should use the relatively large services recommended above and create an interface with the legacy systems. This uses the functionality of the old systems and in this way the outlaid
investment. This is something which the management will fully support.
In this way, we encapsulate the system the whole system and all contact with the system can occur only via the new services we have created. It is okay if this process takes a long time. We do it
at the same pace that the business’s strategic processes require new solutions. We can manage the other standard core processes in peace and quiet in their existing and tried-and-tested
environments. This works as long as the technical platform on which the old system was built still exists and can be maintained.
More Basic Principles
Basic Principle 2 – Service Structure
The various services can, naturally, not be entirely independent of each other – but as far as possible, we would like to create a service structure that works as efficiently as possible.
We derive the part of the service structure that manages object services directly from the IRM matrix that we developed in the architecture plan. Here we have specified how the
different object groups or services are interdependent. We distinguish between master data (customer, product, etc) and event data (customer orders, staffing).
Master data are normally quite independent of other services; event data, on the other hand, is always dependent on one or more master data services.
In addition to object services, we have different infrastructure services that manage items such as logging, security, messages or deviations. These are sometimes called utility
services or technical services.
We will come back to service structure and how it is affected by the other basic principles once we have gone through them.
Basic Principle 3 – Standardised Interfaces
The interface (the contract) indicates how we communicate with our services. Each communication usually requires an interface that we send to the service and another interface when we receive a
response from the service. The published interface must be as stable as possible. This means that we prefer not to alter them when a new communication need arises.
If the service is based on a well-normalised data model, the interface can also be stabilised. We want to avoid a large number of interfaces for each service. We would like to achieve interfaces
that cover most communication needs. Ideally, we therefore also like to make the interfaces as big as possible, that is, when we ask to read some of the service information, we will get everything
that we are authorised to read. At the same time, this is a trade-off with response time and complexity where we want to increase the likelihood that the interface will be reusable.
Defining services is a part of the process of developing the enterprise architecture. The services are based on the business’ overview data model of the business. The interface should also be
considered to be a part of the enterprise architecture. This requires a greater overview than in the individual projects and their requirements.
To be forced to alter a published interface and create new versions is a sign that we have not done the groundwork properly. Version management is something that takes a lot of resources and easily
generates errors, so let’s avoid this by getting it right first time.
Basic Principle 4 – Services Catalogue
After services are established, it is important to document them and make them available. An important aim is of course that they will be reused and that no one should have to create a service that
already exists. One alternative (that is seldom mentioned) is, of course, to supplement an existing service with new functionality or a new interface. The services are related to the overarching
data model, so it is easy to see which service needs to be updated with a new function. The enterprise architect should be responsible for cataloguing each service and making it available for
everyone in the business.
A joint analysis will assess whether the existing service meets needs or should be supplemented. This should determine if the existing interface covers the need or new interfaces need to be
developed. Alternatively, it may be a question of creating an entirely new service.
A services catalogue should also list how the number of services and associated interfaces grow and how the degree of reusability develops.
Basic principles 5 – 8 below are based on previous basic principles and are managed during the design phase.
Basic Principle 5 – Loosely Coupled Services
This basic principle is that we can communicate with the service totally independent of the technical platform. If we stick to the published interface, the technical platform the service is managed
on does not matter. This is directly based on development of web services.
It is also important to minimise interdependence of different services. The simplest way to do this is to manage only the same data in a service, in the manner described above. Should we, on the
other hand, allow each project to create its own services and interfaces, we risk quickly ending up in a spaghetti-situation as bad as the one we are trying to release ourselves from in the first
place. We derive the optimal SOA structure with minimal service interdependency from the IRM matrix, where we indicate interdependency between different object groups. The IRM matrix is part of the
architecture process.
Basic Principle 6 – Limit Information about the Interface
When we describe the interface we should limit information about the interface as much as possible. This is to maintain as much freedom as possible in the service architecture. Information we
should avoid publishing includes technical information, functional information, program logic or service quality such as SLAs (Service Level Agreements).
Each time we want to change or develop something that is already published we run into problems.
Basic Principle 7– Autonomic Services
This basic principle is that we should strive for maximum independence. If we start by having built our service structure on an overarching high-quality data model, we can also attain a high degree
of independence.
There is a lot of independence for master data-type object services, such as customer and product-related services. Object services that are event services are dependent on affected master data.
Customer orders are, for example, dependent on customer and product. If we create process services, they are dependent on all services that are managed in the process.
The alternative, that each process or project builds its own completely independent services, has been shown to have poor reusability and creates a lot of maintenance. Several published practical
cases show that this should be avoided.
The encapsulation of legacy systems to make use of the investment made and successively phasing out the legacy system may cause problems for independence in affected services. It is important to
consider the services’ autonomy when encapsulating. Try different measures to retain the services’ independence.
Basic Principle 8 – Statelessness Services
When we manage a customer order, for example, it goes through several different steps, such as finding the customer, and creating the order main and the order rows required. Processing can freeze
if a certain product can not be found. Should we save the status and await assistance with the product that is being sought? No, if we follow the basic principle. Then it is easy to lose control of
the service and its performance. Try to solve it another way. Amazon and many other websites solve this by first creating order rows in a particular service and then proceeding to create the main
order when customers are satisfied with their order rows.
During a session, a service can be changed from a “loose” status to a more or less “full” status. When the service calls on another service, it is status full until it has received a response
from the queried service.
It is difficult to back-pedal in a process service if something has gone wrong. A travel process in which flights, hotel and hire car need to be booked and paid for separately becomes very tricky
to reverse if some aspect cannot be resolved.
More on Basic Principle 4 – Service Structure
The ideal service structure that we derive directly from the IRM matrix may require adjustment for design or operational reasons.
Erl has thoroughly reviewed how each basic principle affects the service structure during the design phase and while putting it into operation. Making these assessments is important and Erl
recommends thoroughly testing a new service structure before finalising the design or putting it into operation.
Advantages of SOA
SOA success conveys several advantages compared with a traditional way of managing IT solutions. Many overlap, but let us try to refine the vital ones.
Reduced IT Costs
The reuse of services creates considerable cost savings. Having many different systems that manage the same data is unnecessary. We can most easily follow this via the number of black dots in a
system matrix. We indicate here which data each system manages. On average, the same data is handled five times, based on our review of hundreds of larger Scandinavian companies. Reducing this to
an average of two saves each company millions of crowns. Calculate how much each “dot” costs to process – it could be SEK 1–10 million per year. Halving the number of dots by reusing
services translated into a saving of hundreds of millions of crowns. Don’t forget the initial investment cost!
Free Up IT Resources
Maintaining and integrating a lot of unnecessary IT solutions threaten to paralyse many IT businesses. All resources are spent on this useless task, one that creates no value for the business.
Nothing will be left over to develop new IT support for new business models and processes. IT operations block attempts to increase or at least maintain the business’ competitiveness.
Increased Competitiveness and Faster Development
Now that all data needs to be managed only once in a service, providing new processes with goal-focused IT support goes quickly. It speeds up process development. We can extend the processes
towards both the customer and the supplier. We can make customer-specific processes. We call this “orchestration” and it creates great flexibility and smoothness. The hot new term for this is
“agility.” This agility must not be confused with agile methods. These do not create agility, but can have their advantages when speed is more important than the long term.
IT as an Enabler
Many people today perceive IT to be a burden on a business and its development. It is time-consuming and costly to develop new IT solutions. SOA allows IT businesses to instead become the enablers
that can actively contribute to achieve the business’ goals and make the visions happen.
Standards Create Quality
When the same data is managed once in a service instead of several times in different IT systems, this creates a standard, which makes it possible to increase data quality in the business. This
creates confidence in the information and the decisions made by management. Today, different systems generate different bases for decisions, which generates uncertainty and demands extra and
resource-heavy investigation into which information is correct.
Reduced Risk When a Big Bang is Avoided
Business managers want to avoid unnecessary risks. Replacing an old legacy system in a large big-bang project where the whole transfer will take place on one single occasion is always very risky.
Experience shows that the likelihood of success is almost none. The idea of encapsulating legacy systems and step-by-step replacement is thus easy to sell. Especially because large companies often
have their own experience of failed big-bang projects. Make sure to do it in two steps, whereby the first step is creating the new service structure and in step two the service’s functions
are uncoupled from the legacy system. Everything to reduce risk and avoid poor implementation.
Rounding Off
The art of understanding what ties this document together requires patience and humility.
The chicken and the egg.
“The whole is difficult to understand before knowing about the different parts. At the same time the different parts are difficult to understand without seeing them in context.”
When we in retroactively go over the different parts, we should always try to understand what ties them together. We should also show the alternative approaches recommended by different players,
which may in part deviate from ours, but sometimes this depends mostly on different use of terminology or a different starting point for discussion.
A meta model that describes the different terms used clarifies the intended meaning. A meta model is a data model of a business, in this case the SOA business. Many SOA terms are historically
technical and may be difficult to understand for those from the business world.
IRM’s meta model is published at http://www.tdan.com/i039fe04.htm.
Footnotes
1. The Knowledge Creating Company
In 1995, Japanese professor Ikujiro Nonaka wrote a book on knowledge management that attracted a great deal of global attention and was much discussed. It covered how Japanese companies such as
Toyota have a highly ambitious approach to managing knowledge. The longest lasting model is how knowledge and information constantly transform from tacit knowledge (the baker
kneading dough) to explicit knowledge (a written-down recipe) and vice versa. Users of explicit knowledge in this way obtain tacit knowledge, and again can adapt and develop the
explicit knowledge.
Westerners often take it for granted how difficult it is to gain tacit knowledge. Once upon a time we had an advanced apprentice system, which we have dismantled to a large extent. But we are also
seeing signs of how we are trying to reinstate it. Different mentoring approaches, such as the Swedish group Ruter Dam (Queen of Hearts), are good examples of this. In this case, company leaders
take the time to support women who are aiming for management positions.
If you want to become proficient in SOA, strive to work near or learn from those who already have knowledge and experience.
2. DM Competence Network
A network (DM Competence Network) for developing data modelling was established in late 2007 with the participation of leading Swedish companies (Ericsson, H&M, IKEA, Saab,
etc). The network has also incorporated a number of selected global experts in this area, such as a Swedish professor in this field, Bo Sundgren. The aim is to develop and spread data modelling as
an important methodology and way of working.
The network is runs by DAMA Chapter Scandinavia, part of the global DAMA network (DAta MAnagement). John Zachman was one of the initiative takers in the late 1980s. Fifty or so chapters are now
established worldwide; the most recent, which is in China, is being established with support from Chapter Scandinavia.
Please send comments and suggestions to the author at eskil.swende@irm.se.
