Several years ago I wrote in Agile Database Techniques [1] about the cultural impedance mismatch between the object community and the data community, which refers specifically to the difficulties
that object-oriented developers and data professionals experience when working together and generally to the dysfunctional politics between the two communities that occur within IT organizations and
even the IT industry itself. Worse yet, this impedance mismatch has become even more pronounced between the agile and data communities. This has been to the detriment of both, but the data community
has particularly suffered as a result. I believe that it doesn’t have to be this way.
History
To understand why our industry suffers from the object-data divide, you need to consider the history of the information technology industry. Object technology was first introduced in the late
1960s and adopted by the business community in the late 1980s and early 1990s, marking what I consider to be the first point of divergence between developers and data professionals. Up until then,
the data and developer communities pretty much worked to the same set of philosophies and strategies – it wasn’t perfect, but at least the two groups were reasonably in sync with each
other.
But the object revolution motivated a cultural gap between the two communities that has existed ever since. As with most other new technologies, there was spectacular hype surrounding objects at the
start: Everything is an object. Object technology is a silver bullet that solves all of our problems. Objects are easier to understand and to work with. Object technology is the only thing that
you’ll ever need. In time, reality prevailed and these claims were seen for what they were, wishful thinking at best. Unfortunately, one bit of hype did serious damage – the idea that the pure
approach supported by objectbases would quickly eclipse the “questionable” use of relational technologies. This mistaken belief, combined with the findings of several significant research studies
that showed that object techniques and structured techniques don’t mix well in practice, led many within the object community to proclaim that objects and relational databases shouldn’t be used
together.
At the same time, the data community was coming into its own. Already important in the traditional mainframe world, data modelers found their role in the two-tier client server world (the dominant
technology at the time for new application development) to be equally as critical. Development in both of these worlds worked similarly: the data professionals would develop the data schema, and the
application developers would write their program code. This worked because there wasn’t a lot of conceptual overlap between the two tasks; data models showed the data entities and their
relationships whereas the application/process models showed how the application worked with the data. From the point of view of data professionals, very little had changed in their world. Then object
technology came along. Some data professionals quickly recognized that the object paradigm was a completely new way to develop software. I was among them and joined the growing object crowd.
Unfortunately, many data professionals believed that either the object paradigm was another fad that was doomed to fail or merely another programming technology – therefore, they remained
content with what they mistakenly perceived to be the status quo.
Unfortunately, both communities got it wrong. To the dismay of object purists, objectbases never proved to be more than a niche technology, whereas relational databases have effectively become the de
facto standard for storing data. Furthermore, the studies of the late ‘80s and early ‘90s actually showed that you shouldn’t use structured models for object implementation languages
such as C++ or Smalltalk, or object models for structured implementation languages such as COBOL or BASIC. They didn’t address the idea of melding object and structured modeling techniques in order
to drive your efforts working with implementation technologies such as object programming languages and relational databases. In fact, practice has shown that it is reasonably straightforward to map
objects to relational databases [2].
When it came to process there was a significant difference between the two communities. Throughout the 1990s, the majority of new software development would use object and component-based technology
and follow evolutionary processes in the mold of Barry Boehm’s spiral life cycle. Where the data community for the most part stuck with what they knew to be tried and true, the developer
community started experimenting with new techniques and technologies, pushing the software process boundaries. Yet, although many promises were made and many case studies written, and even though
modeling languages unified under a single banner, the productivity gains proved to be much smaller than expected. Then, in 2001, seventeen thought leaders from the object community decided to go
skiing. In the evening, they gathered to discuss what works in practice rather than what they’d been told should work in theory when it comes to building systems. Surprisingly, they actually
agreed on a set of values and principles that were captured the publication of the Agile Manifesto [3] and the second divergence occurred. Evolutionary development was good, but doing so in a highly
collaborative and quality-driven manner was even better. The chasm between data professionals and developers was growing even wider, and the agile philosophies and techniques actually provided the
productivity gains that had been promised in the first age of divergence.
Yes, You Need to Care
Since 2001, adoption of agile strategies has grown steadily in organizations, with 76% of organizations reporting one or more agile projects and an average of one-third of all project teams now
applying agile techniques [4]. Furthermore, agile teams are now achieving measurably higher success rates than traditional application development teams and data warehousing projects, 72% compared
with 63% and 63% respectively [5], calling into question the approaches preferred by the traditional data community. In short, agile is not only growing, it appears to be more successful than the
strategies preferred by many data professionals.
Worse yet, a July 2006 survey into the current state of data management indicates that 66% of respondents indicated that development teams sometimes go around their data groups [6]. The reasons given
are varied, and both developers and data professionals share in the blame. One-quarter of the problem seems to be attributable to developers because 8% didn’t know that the data group existed and
17% didn’t know that they needed to work with them. Luckily, both of these problems can easily be addressed through improved training. The remaining three-quarters of the problem is attributable to
data professionals and will prove to be more difficult to address. Twenty percent of development teams felt that their data group was too difficult to work with, 36% believed that the data group was
too slow to respond to their requests, and 19% felt the data group offered too little value. The implication is that data professionals will need to change the way that they work, very likely
adopting agile data techniques such as database regression testing, database refactoring, and continuous integration [7].
On the one hand, this reluctance of development teams to interact effectively, if at all, with their data group potentially exacerbates the data quality problems in modern organizations. The
development teams are unable to leverage the knowledge, skills, and experiences of the data professionals available to them and thereby make common mistakes such as creating new data sources instead
of taking advantage of existing data sources, creating a substandard database design due to lack of skills in this area, and not conforming to corporate data naming and metadata conventions. The data
management group also suffers because they’re unaware of the new database development strategies being adopted by the agile community and thereby miss out on concrete, quality-focused
techniques.
On the other hand, agile strategies of seem to be working. Compared to teams taking a traditional approach to development, agile teams are producing higher quality systems, are delivering them in
less time, are more likely to build systems that meet their stakeholders’ actual needs, and are providing greater return on investment (ROI) [8]. What is important to data professionals is that
they’re doing so via what they call a “whole team” approach where the development team itself has sufficient skills to get the job done, including data skills. Sometimes this means
that a data professional, or perhaps several, is embedded within the team; although, more often than not, it means that the developers themselves are doing the data-oriented activities. Although the
jury is still out, the implication may be that development teams are more effective when they’re not working with data professionals.
Over the past two or three years, I and several of my colleagues have noticed a trend within the industry of moving data professionals out of development and enterprise modeling roles into
operational database administration roles. I suspect that this is happening for several reasons. First, this is very clearly an important role that needs to be filled by experienced professionals. In
short, somebody’s got to do it. Second, many senior IT executives are tired of the constant squabbling between developers and data professionals, and are finding it easier to reduce their
interaction as compared to helping them to smooth over their difficulties. Third, as noted earlier, development teams following agile approaches appear to be getting the job done, often without the
help of specialized data professionals.
Yes, There is a Solution
Overcoming the cultural impedance mismatch is much more difficult than overcoming the technical impedance mismatch. First and foremost, everyone needs to recognize that the problem exists and
needs to be overcome. Developers and data professionals have different skills, different backgrounds, different philosophies, and different ways that they prefer to work. Instead of finding ways to
work together that take advantage of these differences, many IT shops instead have chosen to erect communication and political barriers between the two groups of professionals. These barriers must be
removed, something that the agile data (AD) method [1] can help with.
Second, recognize that one process size does not fit all. Different projects require different approaches and you need to manage accordingly. A data warehousing project is different than a website
development project (although the data warehousing project can take an agile approach too [9]). A team of three people will work differently than a team of thirty, or three hundred. A team that is
co-located will work differently than a distributed team. A team working in a regulatory environment will work differently than one that does not. A team working with legacy systems will work
differently than a team which is developing a greenfield system. It isn’t sufficient for the data group to be right or the application group to be right; they need to be right together and both
need to be flexible enough to do so. We need to stop playing political games and instead find ways to work together.
Third, recognize that we need to consider the entire architectural picture. Too many application developers struggle to understand the fundamentals of data technology and thereby rely too much on
object-relational mapping frameworks such as Hibernate to address challenges with data sources. Similarly, too many data professionals struggle to understand architectural concepts beyond the narrow
confines of the data realm and thereby hobble their effectiveness. Data is one of many important architectural issues that we must take into consideration.
Fourth, everyone needs to expand their horizons. Developers should be trained in data skills. I’ve lost track of the number of times I’ve heard data professionals complain, usually
accurately, about developer’s lack of skill when it comes to database design and subsequent mistakes when they circumvent data professionals. Yet when I ask them how much training they’ve
given developers in these relatively simple skills, or how much mentoring they’ve done, and the conversation suddenly stops. Similarly, data professionals need to be trained in development
skills, particularly in agile philosophies and techniques – both object technology and the agile development paradigm are here to stay, and it’s time that the data community accepted
this.
References:
- Ambler, S. W. (2003). Agile Database Techniques: Effective Strategies for the Agile Development. New York: Wiley Publishing.
- Ambler, S. W. (2002). Mapping Objects
to Relational Databases: O/R Mapping In Detail. - Agile Alliance (2001). The Manifesto for Agile Software Development .
- Dr. Dobb’s Journal (2009). State of the IT Union: July 2009 Survey.
- Dr. Dobb’s Journal (2007). IT Project Success Rates Survey: August 2007.
- Dr. Dobb’s Journal (2006). The Current State of Data Management Survey: July 2006.
- Ambler, S.W. (2005). Evolutionary/Agile Database Best
Practices. - Dr. Dobb’s Journal (2008). IT Project Success Rates Survey: December 2008.
- Hughes, R. (2008). Agile Data Warehousing: Delivering World-Class Business Intelligence Systems Using Scrum and XP. New York: iUniverse, Inc.
