Web mining

Web Mining
Presentation by : M.Hayeri
Supervisor : Dr. M.Ahangaran

2
Discovering Knowledge from and about WWW -
is one of the basic abilities of an intelligent agent
Knowledge
WWW

3
Contents
 Introduction
 Web content mining
 Web structure mining
 Evaluation of Web pages
 HITS algorithm
 Discovering cyber-communities on the Web
 Web usage mining
 Search engines for Web mining

5
Data Mining and Web Mining
 Data mining: turn data into knowledge.
 Web mining is to apply data mining
techniques to extract and uncover
knowledge from web documents and
services.

6
WWW Specifics
 Web: A huge, widely-distributed, highly
heterogeneous, semi-structured,
hypertext/hypermedia, interconnected
information repository
 Web is a huge collection of documents plus
 Hyper-link information
 Access and usage information

7
A Few Themes in Web Mining
 Some interesting problems on Web mining
 Mining what Web search engine finds
 Identification of authoritative Web pages
 Identification of Web communities
 Web document classification
 Warehousing a Meta-Web: Web yellow page service
 Weblog mining (usage, access, and evolution)
 Intelligent query answering in Web search

8
Web Mining taxonomy
 Web Content Mining
 Web Page Content Mining
 Web Structure Mining
 Search Result Mining
 Capturing Web’s structure using link
interconnections
 Web Usage Mining
 General Access Pattern Mining
 Customized Usage Tracking

10
What is text mining?
 Data mining in text: find something useful and
surprising from a text collection;
 text mining vs. information retrieval;
 data mining vs. database queries.

11
Types of text mining
 Keyword (or term) based association analysis
 automatic document (topic) classification
 similarity detection
 cluster documents by a common author
 cluster documents containing information from a
common source
 sequence analysis: predicting a recurring event,
discovering trends
 anomaly detection: find information that violates
usual patterns

12
Types of text mining (cont.)
 discovery of frequent phrases
 text segmentation (into logical chunks)
 event detection and tracking

13
Information Retrieval
 Given:
 A source of textual
documents
 A user query (text
based)
IR
System
Query
Documents
source
• Find:
• A set (ranked) of documents that
are relevant to the query
Ranked
Documents
Document
Document
Document

14
Intelligent Information Retrieval
 meaning of words
 Synonyms “buy” / “purchase”
 Ambiguity “bat” (baseball vs. mammal)
 order of words in the query
 hot dog stand in the amusement park
 hot amusement stand in the dog park
 user dependency for the data
 direct feedback
 indirect feedback
 authority of the source
 IBM is more likely to be an authorized source then my second
far cousin

15
 Combine the intelligent IR tools
 meaning of words
 order of words in the query
 user dependency for the data
 authority of the source
 With the unique web features
 retrieve Hyper-link information
 utilize Hyper-link as input
Intelligent Web Search

16
 Given:
 A source of textual documents
 A well defined limited query (text based)
 Find:
 Sentences with relevant information
 Extract the relevant information and
ignore non-relevant information (important!)
 Link related information and output in a
predetermined format
What is Information Extraction?

17
Querying Extracted Information
Extraction
System
Documents
source
Ranked
Documents
Relevant Info 1
Relevant Info 2
Relevant Info 3
Query 1
(E.g. job title)
Query 2
(E.g. salary)
Combine
Query Results

18
What is Clustering ?
 Given:
 A source of textual
documents
 Similarity measure
• e.g., how many words
are common in these
documents
Clustering
System
Similarity
measure
Documents
source
Doc
Do
c
Doc
Doc
Doc
DocDoc
Doc
Doc
Doc
• Find:
• Several clusters of documents
that are relevant to each other

19
 Given: a collection of labeled records (training set)
 Each record contains a set of features (attributes), and
the true class (label)
 Find: a model for the class as a function of the values of
the features
 Goal: previously unseen records should be assigned a
class as accurately as possible
 A test set is used to determine the accuracy of the
model. Usually, the given data set is divided into
training and test sets, with training set used to build the
model and test set used to validate it
Text Classification definition

20
Text Classification: An Example
Ex#
Hooligan
1
An English football fan
…
Yes
2
During a game in Italy
…
Yes
3
England has been
beating France …
Yes
4
Italian football fans were
cheering …
No
5
An average USA
salesman earns 75K
No
6
The game in London
was horrific
Yes
7
Manchester city is likely
to win the championship
Yes
8
Rome is taking the lead
in the football league
Yes
10
Training
Set
Model
Learn
Classifier
Test
Set
Hooligan
A Danish football fan ?
Turkey is playing vs. France.
The Turkish fans …
?
10

21
Discovery of frequent sequences
 Find all frequent maximal sequences of words
(=phrases) from a collection of documents
 frequent: frequency threshold is given; e.g. a phrase
has to occur in at least 15 documents
 maximal: a phrase is not included in another longer
frequent phrase
 other words are allowed between the words of a
sequence in text

22
Summary
 There are many scientific and statistical text mining methods
developed, see e.g.:
 http://www.cs.utexas.edu/users/pebronia/text-mining/
 http://filebox.vt.edu/users/wfan/text_mining.html
 Also, it is important to study theoretical foundations of data
mining.
 Data Mining Concepts and Techniques / J.Han & M.Kamber
 Machine Learning, / T.Mitchell

24
Web Structure Mining
 (1970) Researchers proposed methods of using
citations among journal articles to evaluate the quality
of reserach papers.
 Customer behavior – evaluate a quality of a product
based on the opinions of other customers (instead of
product’s description or advertisement)
 Unlike journal citations, the Web linkage has some
unique features:
 not every hiperlink represents the endorsement we seek
 one authority page will seldom have its Web page point to its
competitive authorities (CocaCola  Pepsi)
 authoritative pages are seldom descriptive (Yahoo! may not
contain the description „Web search engine”)

26
Web Search
 There are two approches:
 page rank: for discovering the most important
pages on the Web (as used in Google)
 hubs and authorities: a more detailed evaluation
of the importance of Web pages
 Basic definition of importance:
 A page is important if important pages link to it

27
Predecessors and Successors of a Web
Page
… …
Predecessors Successors

28
Page Rank (1)
Simple solution: create a stochastic
matrix of the Web:
– Each page i corresponds to row i and
column i of the matrix
– If page j has n successors (links) then
the ijth cell of the matrix is equal to 1/n if
page i is one of these n succesors of
page j, and 0 otherwise.

29
Page Rank (2)
The intuition behind this matrix:
 initially each page has 1 unit of importance. At each
round, each page shares importance it has among its
successors, and receives new importance from its
predecessors.
 The importance of each page reaches a limit after
some steps
 That importance is also the probability that a Web
surfer, starting at a random page, and following
random links from each page will be at the page in
question after a long series of links.

30
Problems with Real Web Graphs
 In the limit, the solution is a=b=6/5, c=3/5.
That is, a and b each have the same
importance, and twice of c.
 Problems with Real Web Graphs
 dead ends: a page that has no succesors has nowhere to
send its importance.
 spider traps: a group of one or more pages that have no
links out.

32
HITS Algorithm
--Topic Distillation on WWW
 Hyperlink-Induced Topic Search

33
Key Definitions
 Authorities
Relevant pages of the highest quality on
a broad topic
 Hubs
Pages that link to a collection of
authoritative pages on a broad topic

34
Hub-Authority Relations
Hubs Authorities

35
Hyperlink-Induced Topic Search (HITS)
The approach consists of two phases:
 It uses the query terms to collect a starting set of pages (200
pages) from an index-based search engine – root set of pages.
 The root set is expanded into a base set by including all the
pages that the root set pages link to, and all the pages that link
to a page in the root set, up to a designed size cutoff, such as
2000-5000.
 A weight-propagation phase is initiated. This is an iterative
process that determines numerical estimates of hub and
authority weights

Discovering cyber-communities
on the web
Based on link structure

37
What is cyber-community
 Definition : a community on the web is a
group of web pages sharing a common
interest
 E.g. A group of web pages talking about POP Music
 E.g.. A group of web pages interested in data-mining
 Main properties:
 Pages in the same community should be similar to
each other in contents
 The pages in one community should differ from the
pages in another community

38
Similarity of web pages
 Discovering web communities is similar to
clustering. For clustering, we must define the
similarity of two nodes
 A Method I:
 For page and page B, A is related to B if there is a
hyper-link from A to B, or from B to A
 Not so good. Consider the home page of IBM and
Microsoft.
Page A
Page B

39
Similarity of web pages
 Method II (from Bibliometric)
 Co-citation: the similarity of A and B is measured by the
number of pages cite both A and B
 Bibliographic coupling: the similarity of A and B is
measured by the number of pages cited by both A and B.
Page A Page B
Page A Page B The normalized degree of
overlap in outbound links
The normalized degree of
overlap in inbound links

41
Read More
 http://webselforganization.com/

43
What is Web Usage Mining?
 A Web is a collection of inter-related files on one or
more Web servers.
 Web Usage Mining.
 Discovery of meaningful patterns from data generated by client-server
transactions.
 Typical Sources of Data:
 automatically generated data stored in server access logs, referrer logs,
agent logs, and client-side cookies.
 user profiles.
 metadata: page attributes, content attributes, usage data.

44
Web Usage Mining (WUM)
The discovery of interesting user access patterns from Web
server logs
Generate simple statistical reports:
A summary report of hits and bytes transferred
A list of top requested URLs
A list of top referrers
A list of most common browsers used
Hits per hour/day/week/month reports
Hits per domain reports
Learn:
Who is visiting you site
The path visitors take through your pages
How much time visitors spend on each page
The most common starting page
Where visitors are leaving your site

45
Web Usage Mining – Three Phases
Pre-Processing Pattern Discovery Pattern Analysis
Raw
Sever log
User session
File Rules and Patterns Interesting
Knowledge

46
The Web Usage Mining Process
- General Architecture for the WEBMINER -

47
Web Server Access Logs
looney.cs.umn.edu han - [09/Aug/1996:09:53:52 -0500] "GET mobasher/courses/cs5106/cs5106l1.html HTTP/1.0" 200
mega.cs.umn.edu njain - [09/Aug/1996:09:53:52 -0500] "GET / HTTP/1.0" 200 3291
mega.cs.umn.edu njain - [09/Aug/1996:09:53:53 -0500] "GET /images/backgnds/paper.gif HTTP/1.0" 200 3014
mega.cs.umn.edu njain - [09/Aug/1996:09:54:12 -0500] "GET /cgi-bin/Count.cgi?df=CS home.dat&dd=C&ft=1 HTTP
mega.cs.umn.edu njain - [09/Aug/1996:09:54:18 -0500] "GET advisor HTTP/1.0" 302
mega.cs.umn.edu njain - [09/Aug/1996:09:54:19 -0500] "GET advisor/ HTTP/1.0" 200 487
looney.cs.umn.edu han - [09/Aug/1996:09:54:28 -0500] "GET mobasher/courses/cs5106/cs5106l2.html HTTP/1.0" 200
. . . . . . . . .
 Typical Data in a Server Access Log
 Access Log Format
IP address userid time method url protocol status size

48
Example: Session Inference with Referrer Log
IP Time URL Referrer Agent
1 www.aol.com 08:30:00 A # Mozillar/2.0; AIX 4.1.4
2 www.aol.com 08:30:01 B E Mozillar/2.0; AIX 4.1.4
3 www.aol.com 08:30:02 C B Mozillar/2.0; AIX 4.1.4
4 www.aol.com 08:30:01 B # Mozillar/2.0; Win 95
5 www.aol.com 08:30:03 C B Mozillar/2.0; Win 95
6 www.aol.com 08:30:04 F # Mozillar/2.0; Win 95
8 www.aol.com 08:30:05 G B Mozillar/2.0; AIX 4.1.4
7 www.aol.com 08:30:04 B A Mozillar/2.0; AIX 4.1.4
Identified Sessions:
S1: # ==> A ==> B ==> G from references 1, 7, 8
S2: E ==> B ==> C from references 2, 3
S3: # ==> B ==> C from references 4, 5
S4: # ==> F from reference 6

49
Data Mining on Web Transactions
 Association Rules:
 discovers similarity among sets of items across transactions
X =====> Y
where X, Y are sets of items, confidence or P(X v Y),
support or P(X^Y)
 Examples:
 60% of clients who accessed /products/, also accessed
/products/software/webminer.htm.
 30% of clients who accessed /special-offer.html, placed an online
order in /products/software/.
 (Actual Example from IBM official Olympics Site)
{Badminton, Diving} ===> {Table Tennis} (69.7%,.35%)


50
 Sequential Patterns:
 30% of clients who visited /products/software/, had done a search
in Yahoo using the keyword “software” before their visit
 60% of clients who placed an online order for WEBMINER, placed
another online order for software within 15 days
 Clustering and Classification
 clients who often access /products/software/webminer.html
tend to be from educational institutions.
 clients who placed an online order for software tend to be students in the
20-25 age group and live in the United States.
 75% of clients who download software from /products/software/demos/
visit between 7:00 and 11:00 pm on weekends.
Other Data Mining Techniques

51
Path and Usage Pattern Discovery
 Types of Path/Usage Information
 Most Frequent paths traversed by users
 Entry and Exit Points
 Distribution of user session duration
 Examples:
 60% of clients who accessed
/home/products/file1.html, followed the path
/home ==> /home/whatsnew ==> /home/products
==> /home/products/file1.html
 (Olympics Web site) 30% of clients who accessed sport
specific pages started from the Sneakpeek page.
 65% of clients left the site after 4 or less references.

52
Search Engines for Web
Mining

53
The number of Internet hosts exceeded...
 1.000 in 1984
 10.000 in 1987
 100.000 in 1989
 1.000.000 in 1992
 10.000.000 in 1996
 100.000.000 in 2000

54
Web search basics
The Web
Ad indexes
Web Results 1 - 10 of about 7,310,000 for miele. (0.12 seconds)
Miele, Inc -- Anything else is a compromise
At the heart of your home, Appliances by Miele. ... USA. to miele.com. Residential Appliances.
Vacuum Cleaners. Dishwashers. Cooking Appliances. Steam Oven. Coffee System ...
www.miele.com/ - 20k - Cached - Similar pages
Miele
Welcome to Miele, the home of the very best appliances and kitchens in the world.
www.miele.co.uk/ - 3k - Cached - Similar pages
Miele - Deutscher Hersteller von Einbaugeräten, Hausgeräten ... - [ Translate this
page ]
Das Portal zum Thema Essen & Geniessen online unter www.zu-tisch.de. Miele weltweit
...ein Leben lang. ... Wählen Sie die Miele Vertretung Ihres Landes.
www.miele.de/ - 10k - Cached - Similar pages
Herzlich willkommen bei Miele Österreich - [ Translate this page ]
Herzlich willkommen bei Miele Österreich Wenn Sie nicht automatisch
weitergeleitet werden, klicken Sie bitte hier! HAUSHALTSGERÄTE ...
www.miele.at/ - 3k - Cached - Similar pages
Sponsored Links
CG Appliance Express
Discount Appliances (650) 756-3931
Same Day Certified Installation
www.cgappliance.com
San Francisco-Oakland-San Jose,
CA
Miele Vacuum Cleaners
Miele Vacuums- Complete Selection
Free Shipping!
www.vacuums.com
Miele Vacuum Cleaners
Miele-Free Air shipping!
All models. Helpful advice.
www.best-vacuum.com
Web crawler
Indexer
Indexes
Search
User

55
Search engine components
 Spider (a.k.a. crawler/robot) – builds corpus
 Collects web pages recursively
• For each known URL, fetch the page, parse it, and extract new URLs
• Repeat
 Additional pages from direct submissions & other sources
 The indexer – creates inverted indexes
 Various policies wrt which words are indexed, capitalization,
support for Unicode, stemming, support for phrases, etc.
 Query processor – serves query results
 Front end – query reformulation, word stemming,
capitalization, optimization of Booleans, etc.
 Back end – finds matching documents and ranks them

56
Web Search Products and Services
 Alta Vista
 DB2 text extender
 Excite
 Fulcrum
 Glimpse (Academic)
 Google!
 Inforseek Internet
 Inforseek Intranet
 Inktomi (HotBot)
 Lycos
 PLS
 Smart (Academic)
 Oracle text extender
 Verity
 Yahoo!

57
Three examples of search strategies
 Rank web pages based on popularity
 Rank web pages based on word frequency
 Match query to an expert database
All the major search engines use a mixed
strategy in ranking web pages and
responding to queries

58
Best wines in France: AskJeeves

59
Best wines in France: HotBot

60
Best wines in France: Google

61
Conclusion
 Web Mining fills the information gap
between web users and web designers

Web mining

More Related Content

What's hot (20)

Similar to Web mining (20)

Recently uploaded (20)

Web mining