Introduction to Web Scraping with Python
1 like102 views
The document is an introduction to web scraping using Python, focusing on essential topics such as importing files, using libraries like Beautiful Soup, and understanding web requests. It outlines the workflow for web scraping, including how to retrieve data from URLs and process HTML responses. Additionally, it provides practical steps for installation, importing modules, and performing specific functions in Python related to web scraping.
Introduction to Web Scraping with Python
- 1. An Introduction to Web Scraping with Python and DataCamp Olga Scrivner, Research Scientist, CNS, CEWIT WIM, February 23, 2018 0
- 2. Objectives Materials: DataCamp.com Review: Importing files Accessing Web Review: Processing text Practice, practice, practice! 1
- 3. Credits Hugo Bowne-Anderson - Importing Data in Python (Part 1 and Part 2) Jeri Wieringa - Intro to Beautiful Soup 2
- 5. File Types: Text Text files are structured as a sequence of lines Each line includes a sequence of characters Each line is terminated with a special character End of Line 4
- 8. Modes Reading Mode ◦ ‘r’ Writing Mode ◦ ‘w’ 7
- 9. Modes Reading Mode ◦ ‘r’ Writing Mode ◦ ‘w’ Quiz question: Why do we use quotes with ‘r’ and ‘w’? 7
- 10. Modes Reading Mode ◦ ‘r’ Writing Mode ◦ ‘w’ Quiz question: Why do we use quotes with ‘r’ and ‘w’? Answer: ‘r’ and ‘w’ are one-character strings 7
- 11. Open - Close Open File - open(name, mode) ◦ name = ’filename’ ◦ mode = ’r’ or mode = ’w’ 8
- 12. Open New File 9
- 13. Open New File 9
- 14. Read File Read the Entire File - filename.read() Read ONE Line - filename.readline() - Return the FIRST line - Return the THIRD line Read lines - filename.readlines() 10
- 15. Read File Read the Entire File - filename.read() Read ONE Line - filename.readline() - Return the FIRST line - Return the THIRD line Read lines - filename.readlines() What type of object and what is the length of this object? 10
- 16. Python Libraries
- 17. Import Modules (Libraries) Beautiful Soup urllib More in next slides ... For installation - https://programminghistorian.org/lessons/ intro-to-beautiful-soup 12
- 18. Review: Module I To use external functions (modules), we need to import them: 1. Declare it at the top of the code 2. Use import 3. Call the module 13
- 19. Review: Modules II To refer and import a specific function from the module 1. Declare it at the top pf the code 2. Use from import 3. Call the randint function from random module: random.randint() 14
- 20. How to Import Packages with Modules 1. Install via a terminal or console ◦ Type command prompt in window search ◦ Type terminal in Mac search 15
- 21. How to Import Packages with Modules 1. Install via a terminal or console ◦ Type command prompt in window search ◦ Type terminal in Mac search 2. Check your Python Version 3. Click return/enter 15
- 22. Python 2 (pip) or Python 3 (pip3) pip or pip3 - a tool for installing Python packages To check if pip is installed: https://packaging.python.org/tutorials/installing-packages/ 16
- 24. Web Concept 1. Import the necessary modules (functions) 2. Specify URL 3. Send a REQUEST 4. Catch RESPONSE 5. Return HTML as a STRING 6. Close the RESPONSE 18
- 25. URLs 19
- 26. URLs 1. URL - Uniform/Universal Resource Locator 2. A URL for web addresses consists of two parts: 2.1 Protocol identifier - http: or https: 2.2 Resource name - datacamp.com 19
- 27. URLs 1. URL - Uniform/Universal Resource Locator 2. A URL for web addresses consists of two parts: 2.1 Protocol identifier - http: or https: 2.2 Resource name - datacamp.com 3. HTTP - HyperText Transfer Protocol 4. HTTPS - more secure form of HTTP 5. Going to a website = sending HTTP request (GET request) 6. HTML - HyperText Markup Language 19
- 28. URLLIB package Provide interface for getting data across the web. Instead of file names we use URLS Step 1 Install the package urllib (pip install urllib) Step 2 Import the function urlretrieve - to RETRIEVE urls during the REQUEST Step 3 Create a variable url and provide the url link url = ‘https:somepage’ Step 4 Save the retrieved document locally Step 5 Read the file 20
- 29. Your Turn - DataCamp DataCamp.com - create a free account using IU email 1. Log in 2. Select Groups 3. Select RBootcampIU - see Jennifer if you do not see it 4. Go to Assignments and select Importing Data in Python 21
- 31. Importing Flat Files urlretrieve has two arguments: url (input) and file name (output) Example: urlretrieve(url, ‘file.name’) 23
- 33. Opening and Reading Files read_csv has two arguments: url and sep (separator) pd.head() 25
- 34. Opening and Reading Files read_csv has two arguments: url and sep (separator) pd.head() 26
- 35. Importing Non-flat Files read_excel has two arguments: url and sheetname To read all sheets, sheetname = None Let’s use a sheetname ’1700’ 27
- 37. HTTP Requests read_excel has two arguments: url and sheetname To read all sheets, sheetname = None Let’s use a sheetname ’1700’ 29
- 38. GET request Import request package 30
- 41. Print HTTP with urllib Use response.read() 33
- 42. Print HTTP with urllib 34
- 43. Return Web as a String Use r.text 35
- 44. Return Web as a String 36
- 45. Scraping Web - HTML 37
- 46. Scraping Web - HTML 37
- 47. Scraping Web - BeautifulSoup Workflow 38
- 49. Parsing HTML with BeautifulSoup 40
- 50. Parsing HTML with BeautifulSoup 41
- 51. Turning a Webpage into Data with BeautifulSoup soup.title soup.get_text() 42
- 52. Turning a Webpage into Data with BeautifulSoup 43
- 53. Turning a Webpage into Data - Hyperlinks HTML tag - <a> find_all(’a’) Collect all href: link.get(’href’) 44
- 54. Turning a Webpage into Data - Hyperlinks 45