Published in TDAN.com January 2001
Successful delivery of data resources for an e-commerce development effort can best be accomplished with a sub-team of skilled and experienced data specialists who form an integral part of the
overall project team. The cumulative skill sets of this team clearly overlaps with that of a data warehousing team: data modeling, database administration, data transformation, middleware and data
presentation skills and experience are essential for both types of projects.
The allocation of skilled individuals to roles is, as always, dependent on the size of the project effort and the available staffing budget. On a large project, one or more individuals may be
designated for each role. On a smaller project, single individuals may perform multiple roles. The important point is that these or similar roles are indeed clearly defined, the listed deliverables
are clearly assigned to the roles, and individuals are clearly slotted to fill the roles.
Data skills and aptitudes are somewhat tangential to those required for function analysis or software development. Data skills and software skills are conflicting yet complimentary. Data
specialists tend to consider process as something that happens to data; function analysts and software developers tend to look at data as a byproduct of processes or programs. If both
“camps” are strongly represented on an e-commerce project, the resulting deployment is likely to be well-balanced and robust in construction.
As is the case with staffing for specialized skills and peaks in development effort, contracted staff, ranging from minimal staff supplementation to all or nearly all of an entire team, should be
considered as an option. Because of the similarity of much of the data-related work on e-commerce projects to that of data warehousing, many of the necessary skills required can be found in staff
with data warehousing experience.
Data Architect
The most visible generalist role throughout all phases of an e-commerce project is that of Lead Technical Architect. Not to be overly stereotypical, but…technical architects (designers) are,
by and large, more skilled, experienced and acclimated to working in detail with software and hardware–rather than with data in and of itself. For e-commerce projects, establishment of a
complimentary role of Data Architect is strongly advisable
- Because of the importance of the data components and interfaces throughout the project lifecycle
- As a means to assure that the ubiquitous and proprietary nature of the organization’s data assets are leveraged as the fundamental integrating component of the effort and the result.
Most of the staff of an e-commerce project will, of necessity, be specialists in either a business area or a technical area. In contrast, one of the major requirements of the Data Architect is the
ability to span both domains. Like the Lead Technical Architect, the Lead Data Architect is staffed full-time on the project throughout the life-cycle, from initiation through
implementation—and beyond. Doing so provides critical continuity across the phases.
Data Modeler
The framework for the entire data team (and dare we say for the entire project) is developed and maintained by the data modeler. New technologies, languages, architectures, development techniques,
and business channels have not eliminated the need for fundamentally sound database design practice.
Data modeling tasks include the analysis and design of extensions to e-commerce package database designs.
Data Administrator
While the data modeler provides the framework for the data team, the data administrator fills in all the details. The DA should have a background in metadata repository or data dictionary
development and administration, as well as data usage/lineage identification and tracking.
The data administrator oversees all data components within the project, as well as all interrelationships among data components, and between data and other components. The individual filling this
role should be detail-oriented and be willing and able to act as a catalyst among team members of various roles.
Information Architect
The information architect functions as a link between the creative-and-cognitive front-end designers and the data team. S/he must understand the full set of data available and required for the
site, and how to best present this data in a manner integrated well into the overall interactive design. An ideal candidate for this role would combine a graphics background with experience in the
design of decision-support output such as report screens and multidimensional displays.
Database Architect
The role of database architect is likely to be filled by a seasoned DBA (database administrator)—a practitioner who knows one or more database management system products inside and out. This
individual has been though all the hard knocks of installing, configuring, performance tuning, and debugging high-volume, high-performance and high-availability databases.
If a decision has been reached that the deployment will utilize a packaged e-commerce application product, it will also be beneficial if the database architect has experience in the pros and cons
and idiosyncrasies of deploying this specific solution.
The first assignment of the database architect, in the design phase, is to work closely with the data modeler and data administrator to transform the data requirements of the project–captured in
the logical data model–into a physical data model. The physical data model will in turn be used to create or enhance the production database or databases for the e-commerce site.
During the development phase, the database architect supports the software development team by creating and maintaining one or more test versions of the physical database. The database architect
also works closely with the data conversion specialist and the data access developer to assure that the test database is populated with data necessary for executing test cases for
- data integrity
- processing accuracy
- “stress testing” for peak concurrent accesses
- satisfactory response to accesses
The database architect is indispensable in the early deployment period. Regardless of the detail and extent of testing scenarios developed and executed before going live, and the robustness of
automated database error-handling and failover mechanisms, in early production there will be stress and exception situations encountered that are impossible to foresee. The database architect will
need to be on call to monitor, respond to, and correct these types of situations as they arise.
Data Access Designer/Developer
This team member knows all the ins and outs of data access development–and in today’s technical environment this undoubtedly means SQL and all of its acronymic flavors, including:
- ODBC (Open Database Connectivity)
- OLE DB (Object Linking and Embedding Database)
- JDBC (Java DataBase Connectivity)
- SQLJ (SQL for Java)
There may be several different ways of accomplishing any given database access requirement. In order to chose the most appropriate alternative, the data access designer/developer needs to be
intimately familiar with
- the physical data model for the project
- the performance requirements of the application
- the characteristics and configuration of the connectivity technologies with which the data access requests and responses will be communicated between the application and the DBMS
- the DBMS in which the data will be stored—especially how the DBMS internally interprets and executes data access requests
Data Transformation Specialist
Because an e-commerce application is typically assembled from multiple large component parts, the role of the Data Transformation Specialist is critically important.
The person filling this role requires experience with both commercial ETL (Extract, Transform, Load) tools, and in designing and developing custom software to provide these capabilities. The data
transformation specialist works closely with the data conversion/quality specialist to take advantage of opportunities to co-develop and reuse similar capabilities for data conversion and
transformation. The skill sets for these roles are similar, so in many cases they may be combined in a single person.
The team is likely to look to the data transformation specialist for skills and experience in messaging-type data formats and standards, most universal of which are likely to include XML and
related technologies such as
- XML meta-data formats: XML schemas and DTDs
- XSLT—eXtensible Stylesheet Language Transformations
- SOAP—Simple Object Access Protocol
- SAX—Simplified API for XML
- DOM—Document Object Model
Data Middleware Specialist
Just as the database architect is the more technical alter ego to the data modeler, the data middleware specialist is the technical counterpart to the data transformation specialist. This role
specializes in designing and implementing high-performance physical transport systems that put data in motion between software and data storage components of the e-commerce application. The
individual in this role is experienced in the connectivity options of several database management systems and middleware solutions such as EAI frameworks and message brokers.
Data Storage Hardware Specialist
This role is the hardware expert on the data team. The person filling this role has knowledge and experience in high-performance, high-volume data storage technologies, and high-bandwidth,
high-capacity data communication networks. Competencies should include:
- SANs (Storage Area Networks)
- RAID (Redundant Arrays of Independent Disks)
- “Hot-swappable” storage devices and arrays
- Offline and “near-line storage”
- Backup media
Data Conversion/Quality Specialist
An e-commerce application is never deployed with an empty database. The initial data needs to come from somewhere, and the starting places are existing databases and files, both internal and
external. It’s absolutely essential that the site starts with a quality set of data. The data conversion/quality specialist brings skills and experience in developing quality-measurement
metrics, data profiling, testing plans, and initial load/unload/reload procedures.
Even if the resources of your “E-Commerce Data Dream Team” do not allow for allocation of a full-time person to each of these roles, the unambiguous assignment of available skilled and
experienced staff to each of the roles will clearly increase the likelihood of successful e-commerce project delivery.
This article is an excerpt from the forthcoming book Data Warehousing and E-Commerce, Copyright 2001, Prentice-Hall PTR.
