Dimensions of Data Architecture

In this article, we focus on identifying the various dimensions of data architecture, their hierarchies and their levels.

Dimensional analysis can be defined as identifying the various facets of a problem and solution by measuring it along different dimensions that have an influence on it. It is useful in studying the structural and behavioral aspects of the problems and solutions both in isolation, and in matching them appropriately. The first step in dimensional analysis is identifying and extracting the different dimensions along which the enterprise problem can be classified. The next step would be to understand the organization of each classification in terms of any hierarchical relationship they exhibit, and how many levels exist in that hierarchy. It will also be interesting to note whether some of these dimensions co-exist or are mutually exclusive in a particular enterprise’s data architecture at any instant. Similar study can be done about the sub-dimensions whether they are overlapping or disjointed.

Multiple Dimensions of Data Architecture

Dimensions are orthogonally independent ways of looking at a particular measure. A business looks at sales by region, period, product and promotion, which are four dimensions that decide the sales figure during analysis. Similarly, we analyze data architecture by the requirements, technologies available, and the constraints that decide the choices, the domain that they relate to, etc. In this context, dimensions are also called as perspectives.

In Figure 1, three such perspectives (administrator’s view – vertically from top to bottom, user’s view – horizontally from left to right, and requirements’ view – as functional in the front, and non-functional behind it) have been depicted as a cube. Our visual cognition allows us to visualize only three orthogonal dimensions, hence the maximum limit is three and we call it a cube. But many other dimensions and sub-dimensions are also possible.

The other dimensions that could be added are data types, domains, classification of data, and security levels depending on the importance of data, accessibility requirements of differently abled users, geographical distribution or location of users and the life cycle time of the business process. Some dimensions could be combined like domains and data types, where there are some correlations. But they can remain and studied as independent relative to each other, as well.

Figure 1: An Example of Three Possible Orthogonal Perspectives of Data Architecture of an Enterprise

B. Life Cycle Stages of an Enterprise

The life cycle stages of an enterprise play an important role in the architecture. To start with, there is the birth of an enterprise. Next, the survival of the enterprise amidst its competitions nurtured by its customers. The birth and survival of the enterprise could happen in any of the life cycle stages of the Industry itself. Depending on it, the enterprise might be in a position to be a leader at birth of a nascent industry, or have to wrestle that position from its competition in a mature one.

C. Layering of Data Architecture

Data architecture layers can be of various types.

1. Horizontal data flow layer from data source to data users.
2. Vertical layered architectures that depict the services view of data.
3. Abstraction levels for administrators.
4. Abstraction levels for users.

Layering of data architecture is a best practice that could be used in various perspectives. Firstly, based on data flow from creators to consumers of data, it can be used to classify layers as online transaction processing (OLTP) or operational data store (ODS) where most of the consolidation, integration and data cleansing happens for operational reporting; data marts where individual business processes are analyzed with conformed dimensions across data marts drawn as dimensional models; online analytical processing (OLAP) cubes where multidimensional calculations are available for quick, ad hoc analytics on large data volumes; and finally the scorecards and dashboards that serve the visualization for the end users using the web medium. This is a horizontal flow architecture based on how data flows from its creation to its consumption – a cradle to grave kind of data life cycle lineage trace.

Secondly based on data service to data storage layering is used to classify the tiers (business objects services layer, master data services layer, analytics services layers, business intelligence services layer) that form the top layer. Below this there is a data as a service layer where any data is available as individual entity without the semantics of business objects, master data, analytics or business intelligence, but just as order master, vendor-part contract, etc. Below this are the application services that serve the application specific data like those from Oracle Financials or from SAP. Below that is the data integration layer where data is directly wrested out of the data stores, databases and files using custom written programs. Below this is the actual storage in the file internal physical representation. This is a vertical flow where same data is served in various levels of abstraction of semantics, depending on the sophistication of the customer of this data services.

Layering could also be based on administrators of data, from governance to data architects to designers to data management implementation.

Finally, we can use layering based on users of data, ranging from strategic decision makers, who use data with high leverage, to tactical managers who benefit analytical intelligence, down to the operational users who use it for routine transactions that propel the day-to-day functioning of the enterprise.

D. Stakeholders or People

People dimension forms two sub-dimensions. These are data administrator’s and data users’ perspectives. The data administrator’s perspective is similar to Zachman Framework’s [2] various perspectives as horizontal row headers. Each of the levels deal at a different level of abstraction. There is a semantic interface involved between the adjacent layers.

• Data stewardship view (planner’s / custodian’s / trustee’s) view: Governance view is the outlook of the guidelines and directives that establish the control and organizational structure of data architectural design and the operational data management activities for implementation of the architecture.Here the conceptual mental model (conception, evaluation, mental modeling, analysis, design, problem, and solution) takes shape, gets form and finally implemented. These could extend to data modelers and database administrators.
• Data users’ view: Data users view has sub-dimensions that are overlapping. First, we have strategic users, tactical users and operational users. Second, there are internal users and external users for the enterprise. Third are creators and consumers of data, denoting data flow architecture. Finally, there are owners (like an account manager for account master data) and users (sales assistant using the account master data for transactions data entry).

E. Requirements Classification View

Requirements are broadly classified into functional and non-functional requirements. Functional requirements involve reusing an industry accepted data model, compliance to laws and regulations by government, industry and other regulatory bodies that are mostly specific to that domain or the enterprise, and enhancing the data model to suit the particular enterprise’s requirement. Major portions of these activities are generally done by the enterprise data architect with the technology available.

Non-functional requirements are enabled by technology vendors through innovative products and partner offerings, and the choices offered are considered and selected by the enterprise data architect.

Here the major portion of the work is done by vendor as product offerings. This might involve choosing between shared-everything or shared-nothing architecture, scaling up or scaling out decisions.

F. Intent, Data and Knowledge

Intent is the metadata, rules and constraints as per the design of the system (intent to pay based on salary certificate, appraisal of co-laterals, title check of co-lateral, intent to drive well based on passing the driver license exam and vision test). Don’t forget to see the correlation between rules and data.

Data is the actual event and facts recorded from the real world experience (Check paid on 31st March 2009, value of property at 31st March 2009, title of property at 31st March 2009).

Knowledge is the belief that depends on the historical pattern of the above data, observed over a period of time, and inferences arrived with sufficient confidence, based on data rather than philosophical, intuitive, emotional or impulsive nature (will pay based on credit history, would have done this crime based on police records and criminal history, will drive safely based on driving history).

G. Data Types View

Data types can be extended for specific domains – for example, software requirements and software architecture – each of them having attributes defined (like IsFunctionalRequirements, IsAPerformanceRequirement, etc.). Such things could evolve into repositories or warehouses for requirements and architectures. A configurable matching engine based on relevance and the significant dimensions and constraints selected could rank the possible matching architectures. This is very similar to SQL doing the matching between the records stored in the database (in this case, the architecture solutions), against the selection criteria given in the query ( in this case, the requirement specifications). Cost parameters for the operators, query execution plans, and indexing have to be developed for such custom data types. They are called Data Cartridges (Oracle), Data Extenders (DB2), and Data Blades (Informix).

Data types can be classified broadly into structured, unstructured and semi-structured or extensible structured.

H. Security Classification View

Security aspect casts its influence on the data architecture heavily since security breach is all about data being compromised. Depending on the nature of classification of information involved, it would call for various special considerations. A good example would be master data management hubs. These are high risk and vulnerable targets for data thefts. This data is collated from all over the enterprise, cleaned and feeds the lifeline of an enterprise real time. So gaining access to this data is a very high value proposition for people on the lookout for such wrong opportunities. Also government and regulatory bodies mandate that such risky data be separated out and put behind multiple security filters. But the same may not be true for relatively low value ordinary transaction data. Similarly depending on the enterprise’s nature of business defense, national government, space-related projects might have a different framework that does not bother about monetary concerns but have their goal as mission critical. Next in line come business enterprises that are multinational, national and regional. Some of them differ by the industry, and the type of information they store like customer’s credit card number, or social security number. They are not only at risk of their information, but are liable for their customer’s information as well, as in personal information in credit card and banking-related applications or medical history of a person in the domain of healthcare and insurance industry.

I. Domain View

Various domains such as retail, healthcare and insurance, biological life sciences, weather, banking and capital markets exhibit different unique characteristics. As a data architect moves up his career ladder, his expertise in domain-specific technology practices assume larger importance. A principal Architect, for example, would be expected to know not just the technical aspects but also the unique aspects that characterize the domain. While some domains may require unconventional data types, others might follow different rules per country. Still others might require streaming analytics.

J. Product or Project

Data architectures for a product differ significantly from a program containing multiple projects or a single project. A product’s data architecture needs to be able to talk to various other components, supported by multiple vendor products. Multiple releases, backward compatibility, support for earlier versions and bug-fixes have to be planned. A sub-dimension of product could be the development  methodology as to proprietary, open source or community development. Project or an ongoing program-based enterprise data architecture might be dealing with problems similar to mergers and acquisitions, regulatory compliance, and deploying real time operational intelligence.

K. Volume and Nature of Data

Volume and nature of data have significant impact on the type of data architecture considered. Small volumes of static data like lookup or reference data may be good candidates for keeping in cache, while large volumes of static data might call for a data warehouse architecture. Streaming data like those of a network traffic or stock market ticker data might need stream-processing techniques and associated architectural components. Large volumes like explosive data on the web may not be amenable to data warehousing, simply because they continuously change and are so huge that they require analytics to be done from where and how they are.

L. Location of Data and Location of Processing

Data appliances come in varied architectures ranging from data-centered computing, conventional shared memory (random access memory) processing and stream processing. Based on where data gets referenced, many models exist including in-memory databases and conventional disk-based processing. Various other massively parallel processing architecture-based database machines exist that use shared-nothing principle of parallelizing storage and processing to enhance performance.

M. Source of Data

Based on the source of data, we have web farming external data into the database for 360 degree view of an enterprise from the point of view of suppliers, customers, competitors and regulators.

Enterprise data is also augmented by customer database marketing agencies and external address data providers.

N. Distribution & Replication of Data

Distribution of location, autonomy and heterogeneity of data and its model and representation affect the data architecture decisions. Replication techniques used varies between batch, synchronous and asynchronous types.

O. Nature of Data

1. Master Data: These are high value reusable assets that cater to both operational and analytical requirements. Their lifetime is high, also their usefulness. They are reusable.
2. Audit: This involves the audit data from monitoring privacy, security, reliability, compliance and safety-related activities.
3. Transaction: This consists of the key business and business support transactions that keep track of routine activities. The lifetime of transaction data is quite small compared to lookup, reference or master data.
4. Lookup/Reference: This consists of standard data available in the public domain like list of country names or the unique DUNS numbers for businesses.
5. External Data: This includes address data (US Postal Service codes).
6. Metadata: As an intent or a rule that defines what the business enterprise looks like. Later the actual data is analyzed statistically to align this intent as to how much of it actually works out in the real field.

This data serves many purposes:

• As a help desk or directory for the enterprise’s information assets.
• As a semantic translation layer interconnect between components that have impedance between them. As the dimensions grows, at the intersection of each dimension, there is a semantic layer, a mapping that is necessary for the translation.This can be visualized in case of:

These bridges are a rich source of metadata (like the Rosetta stone that acted as a translator device between Greek, Demotic and hieroglyphic languages, since it contained the same message in all these three languages – this was the earliest metadata repository semantic layer combine equivalent – in its function from the bygone era). Such universal metadata helps to trace from whatever domain in the cube of enterprise metadata to wherever else in the metadata cube.
Conclusion

We have described data architecture using various terms, dimensions and sub-dimensions. It has helped to appreciate the richness and variety of the complexity of the data architecture problems and solutions. This awareness will help get a better idea of where we’re headed and what might be the future challenges.
Acknowledgement

The first two authors Sundara_rajan and Anupama_Nithyanand are grateful to their mentor and third author, S V Subrahmanya, Vice President at E-Commerce Research Labs for seeding and nurturing this idea, and Dr.T.S.Mohan, Principal Researcher, Dr. Ravindra Babu Tallamraju, Principal Researcher, and Dr.Sakthi Balan Muthiah, Manager-Research at E-Commerce labs at Education & Research, Infosys Technologies Limited, for their extensive reviews and expert guidance in articulating these ideas. The authors would like to thank all their colleagues and participants of authors’ training and knowledge sharing sessions at Infosys Technologies Limited, and contributed positively to these ideas.

The authors would like to acknowledge and thank the authors and publishers of referenced papers and textbooks, which have been annotated at appropriate sections of this paper, for making available their invaluable work products that served as excellent reference to this paper. All trademarks and registered trademarks used in this paper are the properties of their respective owners / companies.

References

1. Sundararajan PA, Anupama Nithyanand and Subrahmanya SV, “Dimensional Analysis of Data Architecture,”
   communicated earlier to TDAN.
2. “The Zachman Framework for Enterprise Architecture,” Zachman Institute for Framework Architecture (www.zifa.com, www.zachmaninternational.com).
3. Ralph Kimball and Margy Ross, Data Warehouse Toolkit: The Complete Guide to Dimensional Modeling, 2nd Edition, John Wiley & Sons, 2002.
4. Dan Sullivan, Document Warehousing and Text Mining: Techniques for Improving Business Operations, Marketing and Sales, John Wiley & Sons, Inc., 2001.
5. David C. Hay, Data Model Patterns: A Metadata Map, Morgan Kaufmann, 2006.
6. Barry Devlin, Data Warehouse: From Architecture to Implementation, 1996.
7. W. H. Inmon, Building the Data Warehouse, Wiley, 2005.
8. Kamran Parsaye and Mark Chignell, Intelligent Database Tools & Applications, John Wiley & Sons, 1993.
9. Peter Cabena, Pablo Hadjinian, et. al., Discovering Data Mining: From Concept to Implementation, Prentice Hall, 1997.