First off, two pieces of news: My book, Building The Agile Database should be appearing soon (it’s with the publisher and being edited as you read this). The book will be published by Technics Publications (the nice people who brought you The DAMA Guide to the Data Management Body of Knowledge). Second, I will be presenting a half-day workshop on Agile database development on Sunday, August 7, at TDWI in San Diego. If you’re planning to be there (and if you’re not, you should!), please plan to attend this workshop. You’ll find it very informative!
The major theme of TDWI San Diego this year is Agile BI, and I admit to having been (at first) a bit perplexed by this. BI projects have always seemed to me to be highly iterative, so the phrase “Agile BI” sounds almost like a tautology. However, iterative is not the same as Agile, and I have seen non-Agile BI projects fail. Coincidentally, I’ve just been assigned to a BI initiative at work, and this has proven to be a good opportunity to take the lessons I’ve learned about Agile from the software development projects I’ve worked on, and apply them in this new arena.
First of all, let’s examine what it actually means for a project to be “Agile.” Yes, Agile implies an iterative and rapid approach to development. But it’s more than that. The following are also essential characteristics of an Agile project:
-
There is continual, daily involvement in the project by the business sponsor and/or product owner. If the business doesn’t have daily input into the project requirements, it’s not an Agile project.
-
The work is done by a committed team, working together in constant communication and across organizational boundaries and areas of specialization. Work doesn’t just get thrown “over the wall” to one or more specialists; the entire team works together to solve problems and achieve goals in all areas of the project. If members of the team aren’t working together on a daily basis, committed to doing whatever it takes to make the project a success, it’s not an Agile project.
-
There is early and continual delivery of real-world functionality that the business users can look at, evaluate, and critique. Each iteration of the project provides the business with an opportunity to provide feedback into how well the product meets their needs and expectations. If business users have to wait more than a few weeks to see tangible progress being made, it’s not an Agile project.
These three essential characteristics of Agile also reflect (not surprisingly) the three biggest reasons I’ve seen BI projects fail. IT, traditionally, takes much too long to deliver solutions that aren’t really what the business needs right now. I remember one project in particular that was canceled by the business after six months had gone by, and the IT consultants were still trying to decide on a database design and data integration strategy!
There are a couple of other things that need to be said here: First, in my experience, BI projects are not given the same prioritization and resources, nor the same degree of commitment by management and the business, as other software development projects. BI projects are something that IT people are expected to do in their spare time, with no money. The BI project I’ve just been assigned is the third attempt at automating metrics and KPIs for our organization; the previous two attempts foundered for lack of resources and never made it past the POC stage. Needless to say, I haven’t been given any budget or resources for this effort; I’m supposed to get this done in my copious spare time!
Second, I think it’s important to remember that the end goal of any data management effort is not just to produce a point solution (that is, to build a single database or produce a single report); it’s to produce a data delivery infrastructure that allows data to be quickly, easily and seamlessly reused for multiple business purposes, thus greatly increasing the value of data to the business. This is a central theme of my book, and it applies both to traditional software development projects, and particularly to BI work. We may have to produce point solutions up front (i.e., the development team needs an application database right now; some business user needs a particular report right now), but we need to do this work in a way that iteratively moves us toward the goal of creating data repositories that can easily support ad-hoc reporting, analysis, and decision-making in any area of the business. As I say in my book, we are moving rapidly from a service economy to a self-service economy. Therefore, the goal of data management is not just to provide data (reports, applications, etc.) on an as-requested basis to the business (and make them wait for it); the goal is to enable the business to quickly and easily access whatever data they need, whenever they need it, in whatever form is most useful for their decision-making.
I’m taking this approach in my current BI effort. While giving management the point solutions they need right now (in the form of monthly spreadsheets), I am quickly and iteratively developing a dimensional database in Microsoft SQL Server that I can turn into one or more analytic “cubes” (using SQL Analysis Services) and deploy using Microsoft’s Performance Point KPI portal. This will give our managers direct access to all the data they need, in a form suitable for ad hoc querying, to monitor and evaluate the performance of our organization. In just a couple of months (of part-time work), I’m at the point where I can automatically update Excel worksheets from my database; these worksheets in turn automatically update graphs and charts that I can copy and paste into PowerPoint slides and post to our Operations SharePoint site for our management to review. This shows them that progress is being made, and gives them something to evaluate and provide feedback on. Soon, I’ll be able to show them how to query the data in Performance Point, where they will have “hands on” access to our operational data. This will, in turn, promote even more feedback, which I’ll use to improve the database and expand the range of data that’s available.
So yes, it is possible to do “Agile BI.” But keep in mind that, while you’re delivering the point solutions that the business needs right now, you should also be designing and building the data that the business will need tomorrow, and the infrastructure that enables the business to instantly get the data it needs, instead of having to wait for it.
And who knows, perhaps if our business users and managers can see the immediate, tangible benefits of our data management efforts, they may start taking data management more seriously.
NOTE: I’d like to make this a dialogue, so please feel free to email questions, comments and concerns to me. Thanks for reading!
