Data warehouse development has a number of critical success factors. One of these deals with the team getting and sustaining the appropriate level of management support. Even with the most
qualified data warehouse staff, if it is placed in the wrong organizational structure, the seeds of failure begin immediately. A systems development group has a paradigm and supporting procedures
that are designed to help it build and support on-line transaction processing systems. These procedures are not very effective for building and supporting data warehouses.
Following is a letter that might be sent to the Data Warehouse Project Manager from a Director of Systems Development who does not understand what it takes to build a data warehouse.
Dear June:
It’s time for your performance appraisal, and I must say that I am struggling with it. On the one hand, the Vice President of Marketing tells me that his group attributes a 15% increase in the
company’s sales to the data warehouse you are managing, but on the other hand, you consistently violate virtually every standard in our department.
Let’s start at the beginning.
We hired you ten months ago. Even during the interview, I sensed that you would have problems conforming to our way of doing things. You convinced me, however, that you successfully built data
warehouses at two other companies, and that you understood all of the related issues.
Shortly after you arrived, you put together a project proposal for a marketing data warehouse. Contrary to our procedures, you did not submit it through the annual project evaluation process, and
you did not complete our ten-page cost – benefit analysis. Instead, you and a business analyst from Marketing put together a proposal and said that our standard cost – benefit analysis just does
not apply. The two of you theorized that if the Marketing people had a data warehouse to support them, they could more effectively cross-market the company’s products and increase sales by 10 – 20
percent. While you did a good job describing the scope of the effort, you didn’t provide any proof that the benefits would be attained. Not only did the Vice President of Marketing provide the
funds for the project, he put tremendous pressure on me to staff your group with some of my best people. How am I supposed to manage my department under these conditions?
As for the project itself, I am still trying to figure out which steps of our systems development life cycle were followed. That methodology exists for a reason and it has been very valuable in
helping us to develop quality systems. Granted, our systems are often late, over budget, and not responsive to the user needs at the time of implementation, but they absolutely meet the needs
defined in the two-binder requirements definition. Our requirements definition process is intentionally very detailed – it ensures that our team thoroughly understands the problem being addressed
and its proposed solution. Instead of taking the usual six months to produce the requirements package, you chose to have facilitated sessions with the business people who would use the data
warehouse. This is very unorthodox! You held the sessions, developed the data model, and said that you had enough information to proceed with design. You even had the gall to tell me that you
expected additional requirements would be discovered after implementation. Everyone knows that it’s much more expensive to correct requirements deficiencies after implementation. I gave you the
benefit of the doubt, and sure enough, we’re now working on the third release of the data warehouse.
Somehow, you convinced me to acquire an integrated tool set which includes facilities for moving data from the source systems to the data warehouse and for building and managing meta data, whatever
that is. With this tool set, your programmers worked very quickly, but I never saw them documenting their work. You told me that the tools did that for them. The support programmers have not lacked
for information to support the data warehouse, but I haven’t seen any of the documentation placed onto our standard templates.
The database administration group is up in arms! You convinced me to let you have your own DBA, and the group has absolutely no respect for him. His designs include raw data in a highly normalized
database, and he is physically storing a lot of summarized and derived data. In addition, he is creating another database in what he calls a star schema so that business people can have direct
ad-hoc access to the data. The design of this database is terrible – how can anyone expect to update it with sub-second response time?
When you first populated the data warehouse, the sales figures didn’t match those in our annual report. Without the detailed requirements analysis, this was a predictable result. Instead of owning
up to errors by your team, you set about to prove that the problems were in the source systems and in the procedures which govern them. We’re running the company with these systems – how can they
be in error? It turns out, as I could have told you, that the systems do exactly what they’re supposed to do. You just neglected to consider the special program John executes at the end of the
month to look at each transaction and to manually make necessary adjustments.
You seem to have formed a very tight relationship with the Marketing Department personnel. This relationship is so tight, in fact, that I almost never pass the conference room we’ve dedicated to
the data warehouse without seeing two or three of them there. Don’t you know that business users are to be interviewed, and then only involved during the review process? According to your roles
and responsibilities document, you even hold them responsible for ensuring that all the elements are correctly defined – don’t you know that this is your analyst’s job?
You implemented the first release of the data warehouse in six months, and Marketing began to immediately receive the anticipated benefits. You then proceeded to work on the second release, and it
took another three months. I can only summize that the this release was needed because the users changed their minds or discovered new uses for the data – and you are accommodating them! If the
users didn’t know what they wanted, we shouldn’t have gone beyond requirements definition. They should have been able to anticipate that once the data warehouse provided data across product
lines, it would also be useful to have the customer demographics data available. The second release helped Marketing identify needed changes in our advertising messages so that we could reach
specific customer sectors better. Why couldn’t the first release have been delayed to include this feature? Now you’re midway through the third release, again accommodating changing user
requirements. When is this going to end?
As you can see, I am very frustrated, and I really feel that you don’t fit within the application development organization I have created. Reluctantly, I will be giving you a superlative rating,
because, after all, you did deliver on all you commitments. It’s also become painfully obvious to me that you cannot continue working in my group because your approach and interaction with user
departments just don’t mesh with the ones that have served us quite adequately for the last twenty years. One of us is going to have to leave. I understand that the Marketing Vice President has
made you a job offer. I encourage you to accept that offer and wish you the best with your new career.
Yours truly,
Jim
Message to the Readers:
I hope that you found this article thought provoking. If you have any related anecdotes, please send them to me at jggeiger@geigerintl.com. People who provide me with contributions will receive
copies of those submitted by others.
