Web crawler

WEB CRAWLER

PRESENTED BY,
K.L.ANUSHA
(09E91A0523)

ABSTRACT
Today’s search engines are equipped with
specialized agents known as “web crawlers”(download
robots)dedicated to crawling large web contents online which
are analyzed and indexed and make available to users.crawlers
interact with thousands of web servers over periods extending
from weeks to several years.These crawlers visits several
thousands of pages every second, includes a high-performance
fault manager are platform independent or dependent and are
able to adapt transparently to a wide range of configurations
without incurring adittional hardware.In presentation we can
see the details of various crawling crawling strategies,crawling
policies and web crawling process which contain its
architecture and precedure.

WhAT iS A WEB CRAWLER?
 “A web crawler is a computer program
that browses the World Wide Web in a
methodical,automated manner.”
 Without crawlers, search engines would
not exist.
 It is also known as
WEB RoBoTS,
hARvESTER,BoTS,indExERS,
WEB AgEnT,WAndERER.
 Creates and repopulates search engines
data by navigating the web, downloading
documents and files.
CRAWLER
 Follows hyperlinks from a crawl list and
hyperlinks in the list.
 Without a crawler, there would be
nothing to search.

PREREQUiSTiES oF A CRAWLing SYSTEM
The minimum requirements for any large scale crawling system
are as follows:
 Flexibility:“Our system should be suitable for various
scenarios.”
 High Performance: “The system should be scalable with a minimum of
thousand pages to millions so the quality and disk assurance are crucial for
maintaining high performance.”
 Fault Tolerance: “The first goal is to identifying the problems like invalid
HTML,and having good communication protocols.secondly the system
should be persitent(eg:restart after failure)since the crawling process takes
about 2 t0 5 days.”
 Maintainability and Configurability: “There should be a appropriate
interface for the monitoring fo crawling process including download
speed,statistics and the administrator can adjust the speed of crawler.”

CRAWLING THE WEB
 A component
called the
“URL Frontier”
URLs crawled
for storing the list and parsed Unseen Web
of URLs to
download. z
SEED
PAGES

URL Frontier

CRAWLER(SPIDER) WEB

Given a set s of “seed” Uniform Resource Locators (URLs), the crawler
repeatedly removes one URL from s, downloads the corresponding page,
extracts all the URLs contained in it, and adds any previously unknown
URLs to s.

CRAWLING STRATEGIES

There are mainly five types of crawling strategies as below:

Breadth-First Crawling
Repetitive Crawling
 Targeted Crawling
Random Walks and Sampling
Deep Web Crawling

GRAPH TRAvERSAL(BFS oR DFS)?
 Breadth First Search
– Implemented with QUEUE (FIFO)
– Finds pages along shortest paths
– If we start with “good” pages, this
keeps us close; maybe other good
stuff…

 Depth First Search
– Implemented with STACK (LIFO)
– Wander away (“lost in cyberspace”)

 Repetitive Crawling: once page have been crawled,some systems requrie
the process to be repeated periodically so that indexes are kept
updated.which may be achieved by launching a second crawl in parallel,to
overcome this problem we should constantly update the “Index List.”

 Targeted Crawling: Here main objective is to retrieve the greatest number
of pages relating to a particular subject by using the “Minimum
Bandwidth.”most search engines use crawling process heuristics in order
to target certain type of page on specific topic.

 Random Walks and Samples: They focus on the effect of random walks
on web graphs or modified versions of these graphs via sampling to
estimate the size of documents in online.

 Deep Web Crawling: The data that which is present in the data base may
only be downloaded through the medium of appropriate request or forms
this Deep Web name is give to this category of data.

Web craWling architecture

FIG:This represents the High-Level Architecture of a
Standard Web Crawler

craWling POlicieS
The characteristics of web that make crawling difficult:
Its Large Volume
Its Fast Rate of Change
Dynamic Page Generation
To remove these dificulties the web crawler is having the following
policies.

A Selection Policy that states which page to download.
A Re-Visit Policy that states when to check for changes in pages.
A Politeness Policy that states how to avoid overloading web
sites.
A Parallelization Policy that states how to coordinate distributed
Web Crawlers.

SelectiOn POlicY
 For this selection policy the priority frontier is used.
 Designing a good selection policy has an added dificulty:it
must work with partial information,as the complete set of web
pages is not known during crawling.
1.“restricting followed links”used to request HTML resources
the crawler may make a HTTP HEAD request,then there is a
chance of occurrence of numerous HEAD’S.to avoid this the
crawler only request URL end with certain characters such as
“.html,.htm,.asp” etc,and remaining are skipped.
2. “Path-Ascending Crawling”to find the isolated resources.
3. “Crawling The Deep Web”multiples the number of web links
crawled.

re-ViSit POlicY
It contains
 Uniform Policy:This involves re-visiting all pages in the
collection with same frequency,regaurdless of their rates of
change.
 Proportional Policy:This involves re-visiting more often the
pages that change more frequently.

ParallelizatiOn POlicY

A parallel crawler is a crawler that runs multiple process in parallel.

The goal is to maximize the download rate.

CRAWLER IDENTIFICATION
 Web Crawlers typically identify themselves to a web server by
using user-agent field of an HTTP request.

EXAMPLES OF WEB CRAWLERS

 World Wide Web Worm
 Yahoo!slurp-yahoo search crawler
 Msnbot-microsoft bing web crawler
 FAST Crawler
 Googlebot
 Methabot
 PolyBot

CONCLUSION
Web Crawlers are the important aspect of the
search engines.web crawling process deemed high
performance are basic components of various web services.
It is not a trivial matter to set up such systems:
Data manipulation by these crawlers cover a wide area.
It is crucial to preserve a good balance between random
access memory and disk accessesss.

Web crawler

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Web crawler (20)

Web crawler