One of the most common and important dialogues is when the enterprise data architect expresses the need to integrate and the project manager is completely focused on developing their specific application. The following type of conversation will often happen:
Enterprise Data architect for a large company:
“We have been asked to help on this project by offering common semantics for data so that when you design your data, it is integrated into the overall organization. This will help avoid creating a siloed solution.
If we can use the same vocabulary and data semantics across the organization when we design customer, product, order, and other constructs, we will be able maintain this information more consistently and with greater quality and it will also help us view and analyze integrated data across the enterprise.”
Project manager:
“We completely support your vision. However, we are an agile project and
are committed to delivering the first sprint in the next 4 weeks. Will this
slow down our effort at all?”
Enterprise Data architect:
“It may slow down this sprint, but subsequent sprints are likely to go
faster and there will be a much solid foundation and therefore a higher quality
solution.
There are things that we can do to help speed up efforts such as provide re-usable data designs, definitions, and standardized rules.
However, in our experience, it does take more time in the beginning to integrate the solution than to just build separation solutions that use their own semantics, vocabulary, and data designs. It takes time to understand the local view of the data for specific applications and synthesize it with common holistic views of the enterprise. Over time, momentum can build, similar to a manufacturing operation that takes time to build in processes that take more time up front but create efficiencies for future workloads. Once we start creating common data semantics, it becomes easier and easier for future efforts to re-use this work.”
Project manager:
“Sorry, but we don’t have any time budgeted for that on this sprint. We are
working for a specific business unit and to be frank, they just care about their
requirements for this solution.”
Enterprise Data architect:
“But key executives in our organization have said that there is a corporate
mandate to develop integrated data and system designs!”
Project manager:
“OK. Send us whatever communications that you have received and any standards
you have and we will see what we can do regarding following them. Of course,
I’m sure that you understand that our first priority needs to be satisfy our
customer’s specific needs.”
Many project results could occur after this conversation. Three types of possible ensuing scenarios are:
- The team develops a siloed, independent application. In my experience over decades, this is the most common and likely result that will follow after the above conversation. The team will usually give lip service to standards and will quickly realize that changing the vocabulary and semantics to an enterprise-wide standard can mean that they have to re-do much of the data conversion routines, and data conversion efforts can often be a large part of any systems development effort.
- The project team is convinced or coerced to focus and spent a great deal of time and effort on enterprise wide data standards and vocabulary, and above all, integrate the application. While integration is important, this approach that prioritizes integration may overemphasize adherence to standards and harmonization of the application, leading to a very lengthy and costly project.
- There is a balance between integration and independence. The title of the article uses the made-up word ‘indegrate’, meaning a middle path between integration and independence. There are many ways to accommodate both perspectives. For example, this may mean integrating only the most crucial aspects of the system such as customer and product information, and letting go of other integration aspects in order to allow for faster project delivery.
The Middle Path – Indegrate
A central tenant of Zen Buddhism is following ‘the middle path’. There’s a saying in Zen, ‘Not one. Not Two.’ ‘Not two’ means let’s not get caught in polarity and think we have to choose position A or position B. These positions could point to any type of dualism (i.e., two opposing views) such as male versus female tendencies, hardware versus software engineering principles, mind versus matter, or a focus on a specific application versus a focus on integration. ‘Not one’ means that we also have to consider opposing perspectives, goals, and interests, including what each can offer. While it is true that everything is connected (pointing to a concept that everything is ‘one’), it is also true that things are separate as well. Thus, this teaching can help us to transcend positions, judgements, and polarity.
We can apply this wisdom to data and systems projects and instead of focusing too much on the specific, independent application or too much on integrating the application, we can ‘indegrate’ and skillfully provide a balance of ‘integration’ and ‘independence’.
To me, the dialogue at the start of this article feels like the movie Ground Hog Day, where this type of situation has repeated itself over and over in different companies, at different times, and in different parts of the world.
From my experience, there seems to be some people more disposed towards working towards integration. Many data architects, data modelers, and data governance professionals often have a propensity towards looking at the perspective of integration.
In contrast, there also seems to be people more disposed towards building specific solutions. Integration is not a key consideration in their development efforts. It is about solving a specific problem, staying as clear as possible regarding trying to synthesize their effort with others, which could slow down and complicate the solution.
It is very important to note that both sides have merit. And there are times when more emphasis on one is clearly appropriate. If a bank is about to lose its license because it has run afoul of the Office of the Comptroller of the Currency, then a siloed agile approach may be what is needed at that time. One of the most important concepts in systems modeling is defining the boundaries between systems. Focusing on the specific application enables scope control and can help ensure that the specific application is optimized for its specific purpose. On the other hand, having siloed applications has often led to huge inefficiencies due to different processes for the same types of data, data quality issues, confusion regarding knowing which data source is correct, and suboptimal decision-making since there is inconsistent data.
Of course, there are many shades of gray between these two perspectives, for instance, some integrators will also give credence to the quick development of specific applications, and some that are focused on specific applications will also do their best to follow standards and integration practices.
It seems to me that what organizations are doing today is not working. To me, this is the problem. Trying something new is a good idea. Why not focus on developing a middle path where we skillfully employ both integration practices together with specific independent application development practices? In other words, let not just integrate and let’s not just develop independently – let’s indegrate.
How do we accomplish the middle path and balance these perspectives to develop effective systems?
Ways to Reach Middle Path
The following 3 suggestions offer possibilities for effectively navigating the above scenario and/or developing data structures and/or systems from a centered perspective versus a polarized one.
STOP!
The dialogue at the beginning of this article represents a conflict between the perspective of integration and the perspective of developing an independent system. If there is any one point that I would like you take away from this article, it is that when you are in a conflict such as the one in this article, stop and take some time and space!
What is the first thing to do when you are in a conflict?
I have asked this to many thousands of people at workshops. The vast majority of people do not answer it by saying to first stop and take time and space. However, many conflict management authors and professionals will advocate this as the first step.[i]
The practice of Zen also highly recommends this, and I have previously written about how one can effectively do this in our data world.[ii]
In a nutshell, when you find yourself on either side of this integration or separation equation, then try taking some time to simply stop thinking about it, take some deep breaths, put your attention on something else, or simply relax and let go.
This can lead to us moving closer towards the center and less polarized about our perspective.
Move to the Other Side
There is such as strong force in humans to want to be right!
However, if we can really stand in the opposing perspective, then this can have a powerful effect on having a balanced perspective. This can help take down barriers between people and reduce polarization.
I remember one project where I had engaged a consultant to work on the effort and his job was to ensure that the project built integrated, standardized data structures. The project team resisted this fiercely and said they didn’t have time for this. They were on an ‘agile’ sprint and needed to develop something right away. The consultant asked them, “What do you need?” They said the only thing they needed was performance tuning of their existing database structure. The consultant happened to have expertise in performance tuning and helped the project team with this need. After helping the project team, the project manager wanted to return the favor and they spent considerable time modifying the names of database objects to meet standards. It doesn’t always turn out like this, however, I have seen many positive results where someone has expanded their perspective by moving to the other side, and this led to a collaborative, effective solution.
Tyranny of ‘OR’ – Genius of ‘AND”
In Jim Collins’ book, Good to Great, he discusses that great leaders will often embrace opposing perspectives instead of choosing A or B.
Find solutions that accommodate both perspectives of how to integrate and, at the same time, develop high quality, specific applications quickly.
In our data world, three of the leading approaches to analytics employ a combination of integration and specific application perspectives. In Dan Linstedt’s Data Vault approach [iii], the development of ‘hubs’ and ‘links’ provides a means to integrate data from many different source systems, while the constructs of ‘satellites’ allow attributes to be closer in structure to specific source systems. Similarly, Bill Inmon’s approach to developing analytical reporting allows for an integration layer and then a very specific reporting layer that is more suitable for end user access.[iv] A third common approach is Ralph Kimball’s approach of allowing for analytical reporting using very specific ‘star schemas’, but then providing constructs for integration such as ‘conformed dimensions’.[v] Even with these various approaches that accommodate both the integration and specific application perspectives, there is a strong need to keep both perspectives balanced.
I think it is useful to have people acknowledge if they have a stronger bent towards the integration perspective or the delivery of specific systems. Our role may also determine where we are more focused. If we are an enterprise architect, our goal is probably focused on the integration aspect and if we are a project programmer, our goal is probably more focused on completing the specific application at hand.
Once we acknowledge our specific bent, it is important to appreciate other perspectives and if we take on the genius of ‘AND’, we can often see things that we didn’t before.
Bringing this back to Zen: Zen is the art of awareness. By continually working towards understanding different perspectives, for example, deeply realizing the benefits of both integration and developing separate systems, we can often eliminate blind spots and then be so much more effective in servicing needs through our data and IT development efforts.
Summary
My experiences, many of my colleagues’ experiences, and the track record of so many data and systems development efforts point out that:
- Neither too much focus on “enterprise integration” nor “application independent specific” have worked. It is time to try something new.
- Both offer advantages, so perhaps the solution lies in the middle.
- To move into the middle, do the following:
- Allow time and space when there is disagreement or conflict between these perspectives.
- Stretch yourself and if you find yourself with a stronger bent towards one of these perspectives, move to other perspective and really understand it.
- Find solutions that honor and balance all perspectives.
Let’s not just integrate and let’s not just focus on specific independent applications, let’s skillfully combine these and let’s ‘indegrate’!
Thanks to Tom Redman, ‘the Data Doc’, for his editorial
review, comments and suggestions that were included in this article.
[i] This principle is referred to in books such as “Getting Past No: Negotiating in Difficult Situations’ by William Ury, and “Crucial Conversations: Tools for Talking When Stakes Are High” by Joseph Grenny , Ron McMillan , Al Switzler , Kerry Patterson , Laura Roppe
[ii] https://tdan.com/zen-and-the-art-of-data-maintenance-stop-the-crucial-step-in-data/24791
[iii] https://datavaultalliance.com/
[iv] https://www.amazon.com/Corporate-Information-Factory-2E-Inmon/dp/0471399612
[v] https://www.amazon.com/Ralph-Kimball/e/B000AP5ULY%3Fref=dbs_a_mng_rwt_scns_share
