In the weeks leading up to The Data Warehousing Institute’s World Conference in San Diego, all hell had broken loose in southern California. Over 3,000 homes and 20 lives were lost to forest
fires, smoke was everywhere, and air traffic was a mess. In spite of it all, I wanted to go to San Diego. I wanted, if nothing else, to spend a few dollars to help out the local economy. However,
when I landed, rain had cleared the air, the sun had returned, and what I experienced at the conference was a ground-swell of energy and new vision in data warehousing and business intelligence.
The turnout was strong. TDWI had received a few cancellations due to the fires. However given the numbers that did show up, I would say that interest in data warehousing is recovering from the
recession quite nicely. In fact, data warehousing and business intelligence seem to be leading the charge out of the IT malaise of the past several years. TDWI appears to be well positioned for
this renewal.
Sunday
I planned to lead off the week with TDWI’s new course, “Business Intelligence Fundamentals.” Although I had covered data warehousing fundamentals in an earlier conference, TDWI claimed to have
added significant content and the instructor, Karolyn Duncan, has always brought a lot of insight into whatever she has taught in the past, so I decided to take a look.
When Howard Dresner coined the term he simply referred to BI as “a set of concepts and methodologies to improve decision making in business through the use of facts and fact based systems.” I
remember some vendors took this to mean that BI was a big tent and that a business could get along without a data warehouse if it had the right tools. I also remember that data warehousing
architects didn’t pay much attention to delivery. “Build it and they will come” had been a common phrase in the discipline for some time. I wondered how TDWI would bridge the two concepts – data
warehousing and BI.
Duncan started off by deftly reviewing the terms surrounding BI and data warehousing. She quoted Dresner, Inmon, and a number of other thought leaders – all the standard stuff. However, in no time
flat, she had a BI components framework in front of us with data warehousing at the center surrounded by business considerations. I guessed that some of the business types in the audience were
thinking, “It’s about time that we started out with the needs of the business, ended up with true business value, and considered all of the pieces in between within one integrated architecture.”
The beauty of TDWI’s BI architecture was that it could accommodate any one of the major data warehousing architectures – hub and spoke or bus – without modification. It also had the benefit that
managers and architects could gain an appreciation for the complementary roles of business analytical applications, OLAP tools, and enterprise reporting technologies.
Duncan finished the day by covering BI and data warehousing from several different perspectives including business value and process management. She also included a strong section on business
metrics and scorecarding. Even if you have been in the field for a number of years, I strongly recommend this course. It has a fresh perspective. It’s what continuing education is all about.
After class was over, Cary White and I headed for Old Town for dinner and a planning session for our peer networking session that was coming up on Thursday. Old Town is a section of San Diego that
contains a number of Mexican and Native American restaurants and shops. The Coyote Café had a special on lobster. It was excellent.
Monday
The conference got into full swing with Barry Devlin’s Monday morning breakfast keynote, “Future Trends: The Evolution of Data Warehousing and Business Intelligence.” Devlin certainly had the
credentials for the topic. He and Paul Murphy had written of “an integrated warehouse of company data” in a seminal article in 1988. Back then we were struggling to integrate a few of our core
business systems to use for enterprise decision support. Now, Devlin was throwing out the challenge for us to consider the data warehouse as a part of a wider integrated information picture that
included structured and trusted data as well as unstructured and un-trusted data, all in real time and facilitated by improved integration technology.
Devlin’s thesis did generate some controversy. I overheard one wag call it another run at the “virtual data warehouse.” However, Devlin’s framework was not a plea for circumventing the data
warehouse. In fact, he stressed the value of the data warehouse as giving depth to data available to the organization. The truth of the matter is that integration technology will be an ever more
essential key to feeding and accessing the data warehouse in future tactical decision support applications. Over the rest of the week, many instructors were to echo that same message.
After the keynote I headed off for Stephen Brobst’s session on real-time data warehousing (RTDW). I had become interested in the subject several years ago; however I had only scratched the surface
of the subject. Brobst took us to another level.
Brobst brings impressive credentials to the classroom. He has an MIT Ph.D. and has built real-time trading systems on Wall Street. He quickly dispelled any notion that RTDW would forever be a data
warehousing option. His slides on the evolution of data warehousing from batch, to ad hoc queries, to analytical modeling, to time-sensitive queries, and on through event based triggering had a
strong suggestion of inevitability about them. Those of us who have been around data warehousing for any length of time have already seen many companies progress through at least the early stages.
Later on, Brobst shared case histories where some organizations are now in stage 5 and reaping huge benefits. Can you say, “Wal*Mart?”
The challenges of low latency data technologies, i.e. getting data from source systems into the data warehouse as soon possible, turns out not to be the only thing to consider with RTDW. Brobst did
talk about messaging and enterprise application integration as ways to reduce latency, but he then went on to outline the implications of adopting RTDW architecture. He drew a number of
distinctions between tactical and strategic data warehouse deployment. The main business driver behind RTDW is support for the “in the field” decision maker, and his/her information delivery
needs are much different from those of strategic business planners. Brobst ended up recasting many of the tuning and support issues surrounding data warehousing. He suggested increasing performance
by an order of magnitude, deploying hot backups, and creating multiple data warehousing partitions – some tuned for strategic analysis and others tuned for tactical support.
Later, Bob Seiner – who taught a class on Data Governance – and Dan Linstadt – who taught a class on Very Large Data Warehouses – and I discussed data warehouse design issues over dinner at the
Pier. Although the crab cakes were average, the wine was good, and the conversation was stimulating. We talked well into the night.
Tuesday
I usually try to get around to different tracks at each conference, but one session that I always try to hit is TDWI’s BI forum. Instead of having one instructor take a day or two to drill down on
a subject, TDWI brings in different speakers for 40 minute overviews of different aspects of a given BI subject area. This time, the forum dealt with, Strategies, Emerging Trends, Techniques, and
Technologies in Business Intelligence. Most of the speakers were industry analysts and we covered a lot of high value ground. Richard Hackathorn’s session on alerts sounded interesting so I
dropped in.
Alerts in data warehousing are actions such as email notification that some parameter, in which the recipient is interested, is out of normal range. I remember a while back that a guy named Bill
Gates wrote in his book, Business @ The Speed of Thought, about the necessity of having good news travel fast and bad news even faster. All of the major BI tool suite vendors offer
alerting technology, but I have always been amazed at how few of their customers deploy it. Alerts can be tied into batch driven data warehouses, but Hackathorn focused on how the value of the
technology really goes up as data latency goes down and data volumes go up. Those that have passed on alerting in the past need to take another look at it.
The vendor area opened up later on and I went down to see what was up. Information Builders was showing off its newest version of WebFOCUS, which contained a new visual OLAP feature. I took in a
demo of Cognos’s new ReportNet, which integrates Power Play, their OLAP front end. Both vendors have great tools, but they will have to keep an eye on Microsoft’s Report Server. In fact, the
vendor area had a number of solid reporting and analysis technology vendors from Business Objects to Panorama. I didn’t have time to drill down on all of these tools, but I sure could have made up
a decent list of vendors to look into if I were in the market.
The analytics, database, ETL, data quality, and data modeling vendors also made good showings. I talked with Siebel about their predictive analytics capabilities. I had gained a new appreciation
for the unique role of analytical applications from the classes that I had attended earlier in the week. As organizations move towards real time data warehousing they will want to consider
deploying analytical apps along side their reporting and ad hoc analysis tools to gain maximum pay-back.
By the end of the day, I was starting to drag. I was planning to get a light snack at one of the hospitality suites and hit the sack early, but then I ran into:
Dr. Feelgood and The Interns of Love
Think Earth, Wind, and Fire, think Crusaders, think high energy, think great band with a serious grove. The folks at Teradata had really gone all out. They put on a excellent spread, had a great
band, and showed some really funny ‘Teradata Soaps’ – marketing take-offs on day-time TV shows. Teradata is really in the thick of real time data warehousing (they call it active data
warehousing), so they had substance to go with the style. Great show!
Wednesday
TDWI has implemented a new program built around special sessions targeted at specific industries. This time around it was higher education’s turn. Dave Wells and Laura Voss-Allen led off with
their class on the “Unique Challenges in Higher Education Data Warehousing.” The session was interactive and the participants were thoroughly engaged. I wouldn’t say that data warehousing has
been a roaring success in this sector for a lot of different reasons, and this class wasn’t going to produce any magic bullets. However, most of us felt that a common understanding of higher ed
issues and the connections we made that morning would have positive long term effects on our data warehousing programs.
That afternoon Steve Hoberman and Jeani Wells walked us through the basics of enterprise data modeling with a bent toward higher ed. Wells presented a well thought-out higher ed core concepts
model, which drew quite a bit of interest. Hoberman defined several different model types, core, conceptual, logical, and physical, and gave us a set of guidelines for choosing the ideal level for
a given purpose.
After the enterprise modeling session, the higher ed crowd migrated to Andrea Ballinger’s Night School class. Ballinger presented a case study on data warehousing at the University of Illinois,
which blew us away. She laid out a near text book approach to data warehousing that not only was applicable to higher ed, but also would work in most large conglomerates given that they had a
strong executive sponsor with a sound vision. She covered all of the major aspects of the program from planning to administration, and in the end gave us all hope for implementing top notch data
warehousing programs at our respective institutions. TDWI needs to book this woman for a keynote.
In spite of being in class since 8 that morning, a group of us from the higher ed crowd went to the Fish Market for dinner. This was one great seafood restaurant.
Thursday
Claudia Imhoff’s keynote, “Here’s to Your Success,” took us through what many find to be pretty common steps in data warehousing, where corners get cut on the first go-’round regardless of who
has what vision. She introduced the idea of the Information Workshop, which would integrate the data warehouse with the business. Imhoff stressed the need for BI sponsors, an understanding of the
differences between program and project, well identified funding, attention to data quality, and integration between technology and business. I always like Imhoff’s presentations. She does a good
job of launching them with the familiar and then extending the subject in some way or another. This keynote lived up to expectations.
After having been at it since Sunday morning, I took a break. A couple of friends and I went sailing on San Diego Bay. There was boat rental agency just a few feet away from the hotel, and by
mid-morning we had shoved off in a 22 foot Capri. It was a great day for a sail!
That evening Cary White and I hosted a peer networking session for data warehousing in higher ed. We focused on making the group into a loosely knit organization with a news letter and identified
projects such putting together the semantics of higher ed. TDWI is supporting the group by hosting a higher ed forum on their site. Dave Wells is to be congratulated for helping to get this all
started.
Afterwards, Cary and I headed off to Croce’s, San Diego’s best known jazz spot. We had an excellent dinner and singer David Patrone backed up by Mikan Zlatkovich, piano – Chris Mees, bass – and
Tim McMahon, drums belted out a number of tunes in the style of Frank Sinatra.
Friday
The debate over the benefits of real time data warehousing versus the operational data store (ODS) seems to be gaining strength. After attending a session on real time warehousing earlier in the
week, I decided to take Joyce Norris-Montanari’s class on the ODS.
Norris-Montanari provided an excellent ODS overview. People in a hurry often refer to a simple copy of their operational data as an ODS. However, the ODS has a more formal structure. It is designed
to be a subject oriented and integrated decision support database. Unlike the data warehouse, the ODS is intended to be volatile and it is designed for frequent updates in order to support tactical
versus strategic decision making. I thought that in this respect, the ODS is similar in purpose to the active portion of the real time data warehouse.
An operational update arriving in a real time data warehouse triggers an insert and then an update to mark the previous data as no longer active. An operational update arriving in an ODS overwrites
the existing data. The trade-off between these two approaches appears to me to be in the freedom of design. The real time data warehouse reflects all changes and has a more complex structure than
the ODS which is designed to reflect the current state of data. The real time data warehouse includes an active partition that has the same basic structure of the historical partition. The ODS is
not bound tightly to the historical data warehouse in this way. The ODS should be easier for the casual user to navigate for current information. However, the real time data warehouse will give the
tactical user access to history within the same environment. It would be fun to see this discussion in a regular session at a future TDWI conference.
Wow! What a great conference. I’m looking forward to Las Vegas in February.
