“If information is really useful, our appetite for it is insatiable.” –Thomas H. Davenport, “Saving IT’s Soul: Human-Centered Information Management” Harvard Business Review March-April 1994,
p. 30
“The organization of knowledge is often more valuable than the information itself” — –Mark Watson, Intelligent Java Applications (San Francisco: Morgan Kaufmann Publishers, 1997), p.
121
“…this year industry projections are calling for customer Web interactions to increase by an astounding 840 percent.” — Geof Petch “Superman.com: CRM’s superhuman rush to the land of
dot com” Knowledge Management Magazine, July 1999, p. e4
“…natural language interfaces to computers would allow complex systems to be accessible to everyone.” — James Allen, Natural Language Understanding (Redwood City:
Benjamin/Cummings Publishing Co., 1995), p. 2
Introduction
There’s an enormous volume of information in corporate databases and less structured formats. The fact that this volume is rapidly growing (1) in both size and diversity means that there probably
exist facts and opinions that can be used to support almost any decision. A search for information in these wide-open spaces must point the decision-maker in the right direction regardless of what
format the information is ultimately stored in. It is time to combine natural language processing and software agent technology with online analytical processing (OLAP) to help every decision-maker
quickly find the relevant information needed to make timely, accurate decisions.
Jane’s Shelf Space Dilemma
Take Jane for example. Jane is a mid-level marketing manager at a department store. She was recently given the task of allocating shelf space for products in the sporting goods department. Her task
is not extensive; all she has to do is find the right nugget of information to help her decide which products get more space. Unfortunately, this information is so extensive and diverse (2) it can
no longer be effectively accessed via traditional information systems. Jane requires a new system that will humanize and intelligently sort through the ever-widening, constantly changing scope of
available information.
When looking for information to support her shelf-space decision, the first resource that Jane turns to is the point-of-sale system that logs customer purchases. Unfortunately, in the past Jane has
not met with success in her attempts to get valuable information from this system. First of all, combing through the enormous standard reports pushed out by the system never seems to provide
relevant information quickly. Secondly, her ad-hoc requests to the technical staff that support this system have been frustrating at best. As she puts it, they “give me either too much, too
little, or completely irrelevant data.” As a result Jane contacts Jim, a co-worker who recently moved from marketing to the information technology (IT) department. She knows that Jim has combined
his marketing know-how with knowledge of the company’s IT infrastructure. She believes he will be able to understand her request and translate it into “data speak.” By using Jim as her
intermediary, Jane hopes to get relevant information as quickly as possible.
Jim is now playing the role of Jane’s “information guide” (3). People like Jim are a valuable resource in today’s organizations. They are intermediaries who help to connect decision-makers with
the structured information required to make decisions. As stated before, Jim has the skills and knowledge to translate Jane’s request into a format the IT staff can use as the basis for a
successfully database query. This model can be very successful, as it uses Jim’s capacity to understand the context of Jane’s request for information. This means that the IT staff do not have to
recontextualize (4) the business decision that Jane is trying to solve. The downside is that the model has limited scalability and is costly to maintain. As more and more data is collected and
competitive pressure to leverage information increases, the demands placed on Jim become excessive.
Jim’s Helping Hand
The current solution to Jim’s problem is provided by business intelligence (BI). The combination of enterprise resource planning (ERP) systems with data warehousing and OLAP technology (5) places
detailed transaction data into a summarized multidimensional format. This format provides access to information in the full dimensional context that Jane uses to ask business questions e.g. “which
sporting goods product had the highest monthly turnover last year” (6). The fact that this dimensional context is already in place means that Jim can readily fulfill a request for information.
OLAP eliminates the intermediate step of defining the dimensional context of a business question for the purposes of creating a database query. Jim is now empowered with easier access to
information and has come to rely less on dedicated technical resources. As IT infrastructure improves, this model provides more scalable and cost-effective support for information guides than the
previous ad-hoc solution. In fact, improvements in infrastructure are a primary driver of the market for business intelligence tools: analysts predict that after 2000, ERP implementations will
create a “glut” of clean, integrated transaction data (7). In the short term, this glut will be dealt with by the implementation of BI reporting solutions.
If ERP systems and OLAP make it easier for information guides to get at structured organizational data, widespread dissemination of the benefits of these technologies represents a challenge.
Ideally, Jane should not require the assistance of an information guide like Jim. Unfortunately, interfaces to OLAP technology have not progressed so far as to be readily accessible to every
potential user. A recent Forrester Report points out that:
Transactional systems like SAP and Siebel don’t implement decision processes or present data so it can be analyzed. And decision support tools and servers from vendors like Cognos, Business
Objects, and Oracle are for experts only – those that can access data… and give it business meaning. (8)
In an environment where the information system must support every decision-maker, it is not practical for Jane to rely on the current OLAP interface for her analysis. We find that there is a huge
need for decision support systems to make sense of data for every decision-maker regardless of their level of technical expertise.
In her search for information, Jane is faced by another challenge: how to sort through the entire collection available Intranet documents, internal newsgroup postings, and various presentations.
She knows that somewhere in all of those pages lies a key insight that may help her with her decision. Unfortunately, the information contained in those pages is spread widely across many sources
beyond the company’s structured databases. On top of all this internal information, Jane has access to the information of her company’s suppliers, the World Wide Web and various industry
newsgroups. The term “information overload” is frequently applied to situations like Jane’s. So much diverse information without the ability to filter it down to the relevant nugget means that
Jane is confronted with a low level of “intelligence density” (9). Davenport states that “[i]f information is really useful, our appetite for it is insatiable.” As such, Jane’s information
overload is a qualitative problem. If she were able to easily find information relevant to her decision, regardless of the ultimate source, there would be no “overload”.
Information guides like Jim provide a valuable service by increasing the intelligence density of information. Unfortunately, Jim is only one person with a limited amount of time available to help
Jane find answers to her questions. It is not feasible or even possible for Jane to rely on Jim every time she requires information to support a decision. The skills and knowledge Jim possesses are
rare and expensive, so Jane’s company can’t just hire more people like him. Given this limited capacity, how can organizations cost-effectively increase the intelligence density of information
supplied to any person interacting with the organization’s information systems? How can this service be extended to any decision-maker be they an employee, supplier, or customer? How can all
benefits of these technologies be made accessible and intuitive to the most novice user? How can diverse sources of information, both structured and unstructured, be accessed intelligently and
easily from one point of entry?
What To Do
The answers to these questions lie in two complementary parts. The first part relies on natural language processing to intelligently connect every decision-maker both structured and unstructured
information. This answer is maturing as a mainstream solution. For example, the National Institute of Standards’ TREC 7 includes a new question answering track designed to evaluate information
rather than document retrieval (10). Microsoft spent a great deal of time at their recent Tech Ed conference (11) on the use of natural language understanding in applications that access structured
and document-based data. In fact, they announced plans to provide natural language queries into OLAP databases using Microsoft English Query. The combination of English Query with OLAP represents
an exciting prospect for accomplishing what one analyst describes as “moving business intelligence from the Fortune 1000 to the Fortune 75,000.” (12)
A decision support system that combines natural language understanding (such as English Query) with OLAP would make the full dimensional context of information readily available at levels ideal for
decision-making. The combination of a natural language interface with both OLAP data and unstructured information is a powerful tool. The availability of such an interface enables any
decision-maker who is able to ask a question to start finding answers no matter where the supporting information ultimately resides. Other vendors are planning to include the ability to access both
structured and unstructured data through “enterprise portals” (13). Companies such as Excalibur Technologies and Ask Jeeves are building successful businesses around the premise that the easiest
way to gather information is to ask a question. In fact, Excalibur’s fastest growing source of revenue is from a product called RetrievalWare that is marketed to organizations with large
Intranets. The product relies in part on natural language queries to allow any decision-maker to access diverse sources of information. (14)
The second part of the answer lies in software agent (15) technology. Application of this technology represents an enormous opportunity to intelligently guide the decision-making process. Software
agents enable more human-focused IT, as they allow the computer interface to be proactive, adaptive and smart. Technologies for distributed intelligent agents are just beginning to mature. In
addition, the economic justification for development of agents has only recently become a reality with the e-commerce boom. As e-commerce grows in complexity and business-to-business e-commerce
becomes more important, intelligent software agents will be necessary to maintain, speed and automate transactions. Individuals and businesses will employ agents that “enable the consumer to have
a virtual presence in the marketplace to further his or her interest, while freeing the consumer from constant monitoring of market progress.” (16)
After implementation of agent technology at Jane’s company, Jane has an available resource that specializes in finding marketing information. This agent learns Jane’s preferences for information
and actively monitors the databases and document repositories of the company for newly relevant information. With a natural language interface, built-in knowledge of relevant business logic
(rules), and intelligent search capabilities, a software agent is able to quickly provide Jane with the information she needs.
The software agent is not a replacement for Jim or Jane. In Jim’s case, the agent actually frees up Jim to focus on the value-added activities that are his primary work. In Jane’s case the agent
provides information but is not capable of making an actual decision on shelf space allocation. That role is best left to Jane, as her intuition and experience, along with her ability to summarize
and synthesize are uniquely valuable and can’t be replicated. What agents do is enable businesses to create a cost-effective personalized interface between the business and employees, customers,
and suppliers. This relationship can be used as a source of competitive advantage as it tames and humanizes the exploding complexity of today’s marketplace. (17)
Every decision-maker relies on information as input to guide the decision-making process. Whether it be shelf space allocation or the purchase of airline tickets, sifting through facts and opinions
to find those few dense nuggets is crucial to making the right decision. The next level of information system will draw on both natural language understanding and software agent technology to
enable Jane to intelligently interact with enormous volumes of both structured and unstructured data. (18) This proves that. The technology is now available to take advantage of existing IT
infrastructure and free up Jim to add strategic value to the business, not act as a tactical information guide.
1)Growth in the volume of structured information (data) is driven by pervasive implementation of Enterprise Resource Planning (ERP) systems. The most striking example of the growth of unstructured
information (documents) is the increase in size of the World Wide Web by a million pages a day. For ERP growth, see Stacie S. McCullough, “Frontline Decision Making” The Forrester Report June
1999. For Web growth, see Members of the Clever Project, “Hypersearching the Web” Scientific American June 1999, p 54.
2)The networking of information storage and retrieval technology with palm and other mobile computing devices is creating data (and information) in remarkably diverse places. See Keith Krenz,
“Converging Knowledge Devices” Knowledge Management Magazine June 1999, p. 98
3)Davenport, p. 25
4)When an IT department receives a request for information, that request is out of context. Dedicated technical staff cannot be expected to know the detailed context of every business decision that
requires information. Problems arise when a request is mapped to available data according to incorrect assumptions about how the information is to be used. For more on recontextualization, see Mark
Ackerman and Christine Halverson Organizational Memory: Processes, Boundary Objects, and Trajectories (last visited June 19, 1999) http://www.ics.uci.edu/CORPS/ackerman.html
5)Generally, an OLAP database, also known as a “cube” allows users to quickly view the intersection of a measure, in this case monthly turnover, with dimensions such as time and product. A cube
can contain these intersections at many levels (e.g. month and brand) within the dimensions, and will contain additional dimensions (e.g. location) as well. Placing information into this format
directly supports decision-makers, as they are able to quickly find the intersection that is most relevant.
6)Not only managers, but also every decision-maker thinks multidimensionally. The complex nature any decision requires data to be summarized and presented in it’s full dimensional context in order
for decision-makers to “eliminate information gathering as the limiting step” Richard Connelly, Roland Mosimann and Robin McNeill The Multidimensional Manager (Ottawa, Cognos Incorporated 1996)
p. 8
7)DLJ analyst report “The Data Warehouse and Business Intelligence Marketplace”, March 1999
8)McCullough, p. 9
9)Vasant Dhar and Roger Stein, Intelligent Decision Support Methods: The Science of Knowledge Work (New Jersey: Prentice Hall, 1997) pp. 5-6
10)Voorhees, et al Overview of the Seventh Text Retrieval Conference (TREC-7) (last updated June 23 1999) http://trec.nist.gov/pubs.html
11)Adam Blum Advanced English Query Development and Automatically Create Microsoft® English Query Interfaces to Databases or OLAP CUBES (last visited July 6, 1999) www.teched99.com
12)Teresa Wingfield of the GIGA Information Group quoted in a Cognos Inc. weekly competitive update, June 7 1999
13)Hummingbird, a leading business intelligence vendor recently acquired PC-Docs, a document management company. The stated goal of this acquisition is to “deliver a comprehensive view of all
relevant business information from all data sources”. Press Release Hummingbird Acquires PC Docs March, 1999 www.hummingbird.com
14)Press release, Excalibur Technologies Announces Improved First Quarter Financial Results, May 19, 1999. www.excalib.com
15)A software agent is software that is proactive, cooperative, smart, personalized and adaptive in its behavior when directly or indirectly manipulated by a user.
16)David Wang, description of the “Market Maker” agent (last visited http://ecommerce.media.mit.edu
17)Alper Caglayan and Colin Harrison, The Agent Sourcebook: A Complete Guide to Desktop, Internet, and Intranet Agents (New York: Wiley Computer Publishing, 1997)
18)An interesting flip side to the information gathering model is the concept of ‘intelligent publishing’ to data sources. The above descriptions of vendor strategies address only
‘subscription’. There is an opportunity to create the capability for a user to publish his or her output (in the form of a document, standard data reports, etc.) to the knowledge base on an
ongoing basis. This capability could be based on Extensible Markup Language (XML) applications that ensure a document or report is tagged and placed at the most relevant intersections in the
knowledgebase.