Web scraping
Download as PPTX, PDF19 likes23,637 views
Web scraping involves extracting data from human-readable web pages and converting it into structured data. There are several types of scraping including screen scraping, report mining, and web scraping. The process of web scraping typically involves using techniques like text pattern matching, HTML parsing, and DOM parsing to extract the desired data from web pages in an automated way. Common tools used for web scraping include Selenium, Import.io, Phantom.js, and Scrapy.
![Text Pattern Matching
This is a regular expression-matching technique using the UNIX grep
command, and clubbed with popular programming languages
message = 'Call me at 415-555-1011 tomorrow. 415-555-9999 is my office.'
for i in range(len(message)):
chunk = message[i:i+12]
if isPhoneNumber(chunk):
print('Phone number found: ' + chunk)](https://image.slidesharecdn.com/web-scraping-170522083556/85/Web-scraping-7-320.jpg)
Web scraping
- 2. - Data scraping - Types of data scraping - Web scraping - Process of web scraping
- 3. Data scraping Data scraping - is a technique in which a computer program extracts data from human-readable output coming from another program.
- 4. Types of data scraping Screen scraping is the method of collecting screen display data from one application and translating it so that another application is able to display it. Report mining is the extraction of data from human readable computer reports. Web scraping is a web technique of extracting data from the web, and turning unstructured data on the web (including HTML formats) into structured data that you can store to your local computer or a database.
- 5. Types of data scraping Screen scraping is the method of collecting screen display data from one application and translating it so that another application is able to display it. Report mining is the extraction of data from human readable computer reports. Web scraping is a web technique of extracting data from the web, and turning unstructured data on the web (including HTML formats) into structured data that you can store to your local computer or a database.
- 6. Manual scraping: Copy-paste technique
- 7. Text Pattern Matching This is a regular expression-matching technique using the UNIX grep command, and clubbed with popular programming languages message = 'Call me at 415-555-1011 tomorrow. 415-555-9999 is my office.' for i in range(len(message)): chunk = message[i:i+12] if isPhoneNumber(chunk): print('Phone number found: ' + chunk)
- 8. Computer vision web-page analysis There are efforts using machine learning and computer vision that attempt to identify and extract information from web pages by interpreting pages visually as a human being might.
- 9. Vertical Aggregation Vertical aggregation platforms are created by companies with huge computing power, targeting a specific verticals. Some even run these data harvesting platforms on the cloud. Creation and monitoring of bots for specific verticals is done by these platforms, with virtually no human intervention. Since the bots are created automatically based on the knowledge base for the specific vertical, the efficiency of the bots is measured by the quality of data extracted.
- 10. HTML Parsing HTML parsing is done using Java scripts, and targets linear or nested HTML pages. This fast and robust method is used for text extraction, link extraction (for example, nested links or email addresses), resource extraction, and so on.
- 11. DOM Parsing Document Object Model, or DOM, defines the style, structure and the contents contained within the XML files. DOM parsers are generally used by scrapers that want to get an in-depth view of the structure of the web page. One can use the DOM parser to get the nodes containing information, and then use a tool like XPath to scrape web pages.
- 14. Tools for web scraping - Selenium - Import.io - Phantom.js - Scrapy - etc.