Web Mining

Published in TDAN.com January 2000


Increasingly the first point of contact between a customer and a company is at their website – where a staggering amount of consumer data can be aggregated and mined. The Web provides companies an
unprecedented opportunity to analyze their customers’ behavior and preferences. A company’s website, especially one involved in E-Commerce represents a tremendous source of consumer information.
A firm’s site is a perpetual engine of transactional data that can tell an enterprise who its customers are, what they want and how they want it.

Through a daily barrage of messages communicated with every click of their mouse customers are telling companies what products and services they want, in what format, at what rate and at what
price. Surprisingly, few companies are listening, a recent survey by Forrester Research of fifty of the largest United State corporations found that few were using their web data:

Web Data Applications

  • Marketing 18%
  • Customer Service 16%
  • Don’t Use Data 72%

In their frenzy to be the next Amazon, eBay, or Yahoo, companies are scrambling to set up their E-Commerce sites. Often concentrating on the mechanics of transactional processing, setting up their
inventory and shopping carts – but fail to plan for the vast amount of customer data their site will generate. Most companies fail to see that in E-Commerce, long term success depends on how this
web data is leveraged to convert visitors into customers and customers into loyal clients.

Customer Relationship Modeling Via Web Metrics

The web data that is generated with a single sale is of more value then the sale itself since it can lead to a long and profitable relationship with that customer. The goal of marketers today is
not to capture market share, but instead capture a share of a customer over a long period of time on a one-to-one basis: the Web provides an ideal marketplace for doing this.

Every visit to a retailing site generates important consumer behavioral data, regardless of whether a sale is made. Every visitor action is a digital gesture exhibiting habits, preferences and
tendencies. These interactions reveal important trends and patterns that can help a company design a website that effectively communicates and markets its products and services.

The beauty of E-Commerce is that you are able to create your own customer data. For example, your home page should quickly establish a dialog with your visitors in order to find out what their
needs are. Focus on interacting with your customers in order to learn what their needs are so that you can service them better over time, and subsequently retain them.

Data Collection

One key to compiling and capturing this shopper information is a unique identifier: a visitor ID number. A proven strategy is having your new customer register initially at your site by enticing
them with a special service or incentive. Offer access to a special section of your site. Have contests or door prizes. The point is that you need them to register in order to set a cookie, which
can be used as the unique ID number. A cookie is a header that servers pass to browsers.

From that point, the unique key can enable you to track every interaction with that visitor. This unique key will allow your site to link log files and forms database with your company’s data
warehouse and other third party demographic and household information, ad server networks or collaborative filtering engines.

Mining Web Data

So far, most analyzes of web data have involved log traffic reports; most of which provide cumulative accounts of server activity but do not provide any true business insight about customer
demographics and on-line behavior. Most of the current log analyzers provide pre-defined reports about server traffic activity. This limits the scope of these tools to statistics about domain
names, IP addresses, browsers and other TCP/IP specific machine-to-machine activity.

On the other hand, the mining of web data for an E-Commerce site yields visitor behavior analyses and profiles, rather than simple server statistics. An E-Commerce site needs to know about the
preferences and lifestyles of its visitors. Data mining in this context is about addressing such business questions as, “Who is buying what items and at what rates.”

You need to know how to sell and what incentives, offers and ads work, and how you should design your site to optimize your profits. The use of data mining algorithms can autonomously extract
relationships among web data to determine if patterns exists which can yield actionable business and marketing intelligence.

The Time is Now

The Web is a fast, competitive marketplace that doubles every 100 days. A marketplace where on-line shoppers browse retailing sites with their fingers poised over their mouse: ready to buy if they
find what they are looking for – or move on should the content, wording, incentive, promotion, product or service of that site not meet their preferences.

The Web is a marketplace where browsers are attracted and retained based on how well the retailer remembers the customers’ needs and whims. The goal is to know and serve every customer, one at a
time, and to build long-term, mutually beneficial relationships. Data mining is the key to customer knowledge and intimacy in this type of competitive and crowded marketplace.

The information that a merchant gathers from its site can reveal what products have cross-selling opportunities, or what information and incentives the merchant should provide to its visitors based
on their gender, age, demographics and life style interests.

The process involves capturing important visitor attributes from server logs, cookies and CGI forms, then appending to them household and demographic information. Using powerful pattern-recognition
technologies such as neural networks, machine-learning and genetic algorithms, customers may be profiled by their propensity to buy.


The act of retailing on the web is an interactive one in which the consumer can negotiate, exchange information, specify and customize the product and services they wants from the retailer. Web
data mining can serve retailers by providing them with the technology to segment, model and predict how to sell more, learn what’s working and what’s not, and quickly react by adjusting their
marketing, pricing, inventory and communications.

E-retailers (a.k.a. E-Tailers) need to use their web mining analysis to discover their visitors’ demographics, consumer preferences, values and lifestyles. They need to incorporate their knowledge
about their customers in the tone, manner and method by which they communicate with them. They need to look for similar attributes in their current customers and new prospects.

In a networked economy, the customer is in charge. As such, retailers must be adaptive and receptive to the needs of their clients. In this expansive, competitive, and volatile environment, web
mining will be a critical process impacting every retailer’s long-term success where failure to learn from customers can translate into “churn.”

Share this post

Jesus Mena

Jesus Mena

Jesus Mena holds 20 years experience in expert systems, rule induction, decision trees, neural networks, self-organizing maps, regression, visualization, machine learning, etc., and has worked on data mining projects involving clustering, segmentation, classification, profiling and personalization with government, web, retail, insurance, credit card, financial and healthcare data sets. He has authored five data mining and advanced analytics books (search by author on Amazon.com) as well as dozens of articles for Competitive Intelligence Review, New Architect, Intelligent Enterprise, PC AI, DM Review, Marketing Tools, Software Development Magazine, Washington Technology, WEB Techniques, etc., have consulted with NCR, SPSS, SAS, GAO, Raytheon, DHS, Boeing, Booz, Allen and Hamilton, DOJ, Northrop Grumman, Sandia National Laboratories, and the Terrorist Threat Integration Center.  Jesus is the curator of the topic “Web Mining” for Scholarpedia, an online encyclopedia. He has also conducted data mining boot camps for dozens of companies and government agencies.

scroll to top
We use technologies such as cookies to understand how you use our site and to provide a better user experience. This includes personalizing content, using analytics and improving site operations. We may share your information about your use of our site with third parties in accordance with our Privacy Policy. You can change your cookie settings as described here at any time, but parts of our site may not function correctly without them. By continuing to use our site, you agree that we can save cookies on your device, unless you have disabled cookies.
I Accept