We communicate in “near approximations” at all times: everything is open to individual interpretation. How we receive and internalize any message depends on a number of factors: the delivery, the
conditions at the time of receipt, our previous experiences (which determine how we associate and “store” the message with everything else), our interest in the message, our state of being
(energy level, etc.). There is one message that appears to be “failing” in its delivery: data warehousing is not a technology.
Hardcoded Habits
We’ve heard the stories; we’ve lived the results. Solutions are “identified” at the point in time that they are needed – “Just start coding; we’ll figure out what they really want along the
way.” Even though this is a bad approach for systems development in general, it doesn’t apply to data warehousing at all, because its about data first – process and application are the end game.
Many data warehousing implementations mistakenly start with the end game. Certainly, knowing where you’re going is important. However, knowing that you need to get to Boston provides little
“added value” when deciding what kind of vehicle you might want to use to get there (please note the very important distinction that we’re talking about getting TO Boston, not driving IN Boston,
which introduces a whole different set of requirements). It’s the things that I’ll encounter on my way to Boston that should influence my vehicle decision; not Boston itself. Too many
implementations focus on the end results and not on optimizing the process of getting the results.
Many companies are “evolving” into data warehousing implementations. Using existing staff, they are leveraging the skillsets they have to help move their companies forward. This is good, but
there is a caveat. Implementing a data warehousing environment is not the same as our past experiences with systems engineering: the methodologies are different, the staffing skillsets are
different (certainly existing skillsets can be leveraged), and the planning/management is different. Ken Rudin makes a valid point in his comment: “We must forever rid ourselves of the notion that
building a data warehouse is a project.”
This message isn’t obvious and isn’t being heard. Having the opportunity recently to go through a number of interviews there is one disturbing trend I’m seeing: repeatedly I get asked to name
the tools I can put my hands on and manipulate. Less often do I get questioned about abilities to evaluate processes, determine needs, or identify appropriate data sources (relying on existing
domain experts – I have been asked to do it myself).
Since there are very few tools to accomplish the full breadth of what needs to get done, and even fewer people identifying the technology “gaps”, focusing on tool proficiency in this industry is
meaningless. Ok, so there are some implementations where a specific toolset is in place and you need those capabilities. What else are you investing in when you bring in that staff? What real
transferable skillsets do they possess that you will be able to utilize when the “tool of the hour” has exhausted its lifespan?
Wasted Resources
I’m a firm believer in the concept of reciprocal models in life: find a situation and the underlying corollary is likely true in other situations. As a race, we’re not real good at optimizing
resources. One of the by-products of generating anything is waste. We waste a lot of things: “In 1994, Americans generated 209 million tons of municipal solid waste…” That’s just solid waste. What about more intangible wastes, such as the misappropriation or misutilization of human
effort?
Granted, there are a lot of extenuating factors which “challenge” us in optimizing resources: time, money, information, etc. It is the latter that is justifying many data warehousing engagements:
the ability to provide wider access to data and allow for user-specific manipulation to derive personal information. Are we supplying better information merely to support processes that aren’t
effective? Who’s asking? Who’s evaluating? The technologists think that the business staff is responsible to assess the functioning of the business; the business staff is surrounded by alligators
and deservedly can’t change their focus. The bane of Business Process Reengineering was not in the concept but in the execution: the right question, the wrong answer.
Someone still needs to evaluate what we’re doing and why. Data warehousing can be the opportunity to accomplish this task. It is necessary to “qualify” each data element: what is it
intended to support, how it will be used, what rules will govern its use. As we question the data, we should also question the processes that the data is to support: process evaluation for
“free”.
Data warehousing implementations are often “tagged” as failures, in spite of technological brilliance, because they didn’t deliver on the dream of a better life. The organizational and
operational changes that are needed are not directly related to data warehousing, but treating them as being “outside of scope” is a mistake. If they’re related enough to cause failure, they’re
related, period. It’s time to cross the chasm (no, I haven’t read the book yet), define, and begin to dedicate resources to manage business from the middle: a perspective that doesn’t separate
technology and business, but balances them seamlessly.
The In-Between
In my last TDAN article, I introduced some “new” concepts and tried to suggest that the real “meat” of the matter is in the
“in-between”. Focusing too much on success without allowing for or learning from failure sterilizes the opportunities to be gained from the “in-between”: creativity. Even GartnerGroup recently
highlighted the downside of ignoring failures in an article focusing on Knowledge Management practices: “Worst Practices? Discovering the Knowledge in Failures”, by K. Harris, June 29, 1998.
Recently, I got phonecalls from two different headhunters on the same day. One had previous interactions with former co-workers of mine. He had engaged in dialog with them and was pleased to
announce that he finally figured out the “essence” of my skillset: I was non-technical. Another recruiter had just received my resume and enthusiastically thanked me for the detail I had
provided, for it clearly showed just how technical I was.
This is my personal dilemma. I have worked to fit my skillset to the “in-between”. I’ve done such a good job of it that I don’t fit into the structure of most organizations. Operating
predominately in “either, or” mode, they don’t know what to do with my skillset. Personal interests aside, this is not a good sign.
Thinking at all
We often hear the term “thinking outside of the box”. I’m on the campaign trail for “thinking at all”. The recruiting process employed by most companies focuses on finding people who DO
something specific – not on finding people who think. Ok, it’s easier to assess the former rather than the latter. So?
Companies operate like machines. They are looking for machine parts to “fit” into the machine. If you can’t figure out what kind of part it is – discard it. Once someone has molded themselves
into the shape needed for the part they get hired for, companies have trouble trying to recognize individuals as something else (especially a considerably “more expensive” part). So individuals
have to abandon one machine for another to optimize their potential contribution and corresponding worth as they grow and develop new skills.
It may take time, perhaps only 2 years with year 2000 looming large, but eventually the machines will start to fall into catastrophic disassembly. They’re not designed for rapid adaptation and the
demands for change keep coming. Some companies are so “finely tuned” in their ability to produce outdated products or methods of delivery (telemarketing is one example) that they can’t change
the process without threatening the survival of the rest of the machine. The only hope for change is to “kill the beast” through a takeover, or some other “life shattering” event. Year 2000
will provide the additional “stress” needed to tax marginal machines into obliteration.
It’s a Matter of Balance
Unapologetically borrowing this sub-title from my last article, it still applies: it is a matter of balance. Yes, there is a need to have people on staff who can DO certain types of things which
can be identified and categorized: these are hard skills. But there has to be a balance in the accumulation and utilization of soft skills: there is still a focus on deliverables, but the “end
product” is not necessarily tangible. “But,” you say, “I can’t sell intangibles to management.” Ask them if they have life insurance policies, or buy securities. Ask them if they’d consider
giving them up simply because they’re intangible products. These are as intangible as the types of results I’m suggesting that we begin to focus on.
Run some high-level statistics on all of the resources dedicated to operational systems in your organization: number of systems, number of staff, dollars for support, etc. Operational systems are
one-half of the equation; the other half is informational systems or data warehousing. The resources currently dedicated to operational systems will need to be matched for informational systems
(and perhaps be surpassed by). We did a lot of damage by not providing broad-range visions or cross-functional architectures for our operational systems. Let’s avoid the making the same mistake as
we embark on implementing informational systems. And let’s also make sure that we realize that a system is a collection of resources and processes of which a portion may be technically
implemented.
