Our company is getting ready to begin another large Agile Development project, and I will be sharing with you some of the lessons I’ve learned about the work and role of the DBA in Agile projects. (Look for much more information on these and related topics in my upcoming book, Building The Agile Database.)
First of all, though, I need to say a few words about how Agile (and similar progressions in the field of software development) is changing the traditional role of the DBA. Traditionally, DBAs have performed more of an operational role: building and maintaining databases and database servers, addressing production problems and performance issues, and so on. DBAs have also worked with application developers in the past, but they were like the DJs at your local rock-and-roll radio station – if you called them and caught them at the right time, they might take a request. But several things have happened to start moving DBAs out of their traditional roles:
- Many of the traditional operational processes that DBAs have done are becoming standardized to a point where they can be done by (or outsourced to) lower-paid workers. Improvements in GUI-based tools (such as Microsoft’s SQL Management Studio) are helping to fuel this move. At our company, many of our operational DBA processes are being documented, standardized, scripted and turned over to our operations department. The DBA group thus becomes the second-level, rather than the front-line database support group.
- Agile software development practices require more direct, continuous involvement in the development process from all participants, including the business users, architects, QA and the DBAs. The days of “maybe we’ll have something for you in a year or two” development projects are over; now, business users expect to see tangible solutions in a matter of weeks. This necessitates a deeper commitment from everyone on the project team.
- Agile also requires that all participants be able to assist a project in whatever way is needed to solve a problem, overcome an obstacle or meet a tough deadline. Agile encourages people to develop skills outside their particular area of specialization, so they can help out with whatever is needed at any given stage of a project. In the last project I worked on, I not only did the DBA work (designing and building the database, doing performance tuning, and making incremental changes as needed) but also wrote a lot of the application SQL code; helped develop a SQL/CLR component used to perform calculations; wrote triggers to send database updates to a web service; wrote SSIS packages to integrate SQL and mainframe data; converted data from existing databases, files and spreadsheets to populate the database; and assisted with QA testing. This level of involvement is quickly becoming the expected norm for DBAs on Agile projects at our company.
- Another important driver is the desire of companies to leverage and reuse their data assets to provide increased value. Traditional approaches to application (and database) development have produced decades’ worth of stovepiped, application-specific, “islands of data” that cannot easily be integrated or reused. An entire multi-billion dollar industry (can you say “data warehouse”?) has sprung up to address this problem. Now, with businesses wanting to capitalize quickly on every available business opportunity and with the advent of easy-to-use software for creating online reports, multidimensional analytical “cubes,” dashboards, scorecards, and collaboration sites, there is a huge demand for people with the ability to design and create information systems quickly and populate them with high-quality data from disparate data sources. This is the role that Dan Sutherland, in his excellent TDAN article – DBAs Reinvented – refers to as the “information architect.” Dan correctly notes that in the future the role of the DBA will be much more closely tied to the creation of information systems than to the creation of databases per se.
I have frequently pointed out that DBAs perform at least two roles in an IT organization. They have a facilitation role, geared to meeting immediate needs (for example, the need of a development team for an application database they can start coding against, or for a stored procedure to perform a critical application function). They also have what might be called a control role, which helps ensure that the data generated from a given application is secure, reusable, high-quality and business-relevant. It is in the intersection (and the constant juggling) of these dual roles that the modern DBA provides the greatest value to the business.
Having said this, I have to admit that I am not happy with one of the roles I’ve been assigned on this new project: that of the Data Architect/Analyst (at our company, we call them Data Engineers, or DEs). I am filling this role because in our last round of layoffs some months ago, we lost our last DE, a very capable woman who was not only an excellent data analyst and modeler, but a close personal friend as well. Our senior management feels that the DE role can be capably handled by our few remaining DBAs, an idea I have been very much opposed to. I have tried, so far unsuccessfully, to make the following salient points:
- DEs work with the logical (requirements) view of data; DBAs work with the physical design of the database. As I’ve pointed out in many previous articles, nothing good comes from blurring the critical distinction between the logical and physical views of data. Having DBAs do logical modeling increases the likelihood that the logical data model will become the physical database design. In projects I’ve worked on in the past, this has resulted in numerous problems, resulting in much lost time and rework, and lots of frayed tempers within the project team.
- DEs possess a very specialized set of analysis and modeling skills, skills they have developed over many years of experience, study and hard work. These are skills that DBAs, for the most part, don’t have.
- DEs also have a requisite set of soft skills, which enable them to interact well with business users and management. This enables them to quickly and accurately elicit the necessary business data requirements for a given project. Again, these are skills that a lot of DBAs (being mostly technically oriented) don’t have.
- DEs are required to have a solid, in-depth understanding of the company’s business, and of its critical business processes. Most DBA work doesn’t require this depth of understanding of the business.
- The DE role takes a good deal of time and work. In an Agile project, the DE needs to be part of every requirements meeting, every analysis meeting (there is usually one for each user story, plus one at the beginning of each Sprint), and each planning meeting. The DE then needs to constantly update the logical data model from these continually evolving requirements, and feed the updated requirements to the DBA, for implementation in the database. This is more work (and time) than a DBA can spare in addition to his/her regular work.
To illustrate this last point, my work on this project will commence with six straight days of 8-to-5 project planning and requirements-gathering sessions. After each day’s meetings, I will have to document the business data requirements that have surfaced, update them in a first-cut of the logical data model (so that I can quickly produce a first-cut of the database in time for the initial development Sprint), and then do all my regular DBA work.
And on the seventh day, I shall rest.
NOTE: I’d like to make this a dialogue, so please feel free to email questions, comments and concerns to me. Thanks for reading!
