Filip Podstavec - Get inside the head of a crawler
3 likes1,153 views
The document discusses strategies for optimizing web crawling, highlighting the importance of crawl budget and error management to boost organic traffic. It emphasizes the need for effective log analysis to understand crawler behavior and optimize website architecture for faster performance. Additionally, it provides insights into content management practices, such as managing filter combinations and disallowing unnecessary URLs to enhance SEO efficiency.
![155.62.100.122 - - [25/Aug/2017:08:22:55
-0400] “GET /category/mktfest/ HTTP/1.1” 200
“-” “Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)”](https://image.slidesharecdn.com/filippodstavec-171127154332/85/Filip-Podstavec-Get-inside-the-head-of-a-crawler-32-320.jpg)
![155.62.100.122
[25/Aug/2017:08:22:55 -0400]
GET /category/mktfest/ HTTP/1.1
200
Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)](https://image.slidesharecdn.com/filippodstavec-171127154332/85/Filip-Podstavec-Get-inside-the-head-of-a-crawler-33-320.jpg)
Filip Podstavec - Get inside the head of a crawler
- 1. GetINSIDE THE SEARCH ENGINE CRAWLER HEAD Filip Podstavec
- 4. Dr.
- 12. Goal: FASTEST
- 15. Why?
- 17. Higher crawl rate More indexed pages Benefits
- 18. Higher crawl rate More indexed pages Make website faster Benefits
- 19. Higher crawl rate More indexed pages Make website faster Cleaner architecture Benefits
- 20. Higher crawl rate More indexed pages Make website faster Cleaner architecture Fixed error pages Benefits
- 21. January 2016 April 2016 July 2016 October 2016 January 2017 Organic traffic boost Start of crawl budget optimization
- 22. Higher crawl rate ! Direct ranking signal =
- 25. How?
- 26. Make diagnosis
- 27. Medical History 12.6.2010 | Filip Podstavec | sick | height: 183cm | 120/60 17.9.2010 | Filip Podstavec | sick (again) | height: 184cm | 120/60 06.4.2014 | Filip Podstavec | diarrhea | height: 184cm | 120/60 Filip Podstavec | aneurisma| 0
- 30. Medical History 12.6.2010 | Filip Podstavec | sick | height: 183cm | 120/60 17.9.2010 | Filip Podstavec | sick (again) | height: 184cm | 120/60 06.4.2014 | Filip Podstavec | diarrhea | height: 184cm | 120/60 Filip Podstavec | aneurisma| 0
- 32. 155.62.100.122 - - [25/Aug/2017:08:22:55 -0400] “GET /category/mktfest/ HTTP/1.1” 200 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
- 33. 155.62.100.122 [25/Aug/2017:08:22:55 -0400] GET /category/mktfest/ HTTP/1.1 200 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- 34. IP (WHO) Timestamp (WHEN) Method and requested URL (WHAT) Status code (SUCCESSFULLY OR NOT) User-agent (DETAILS)
- 35. “The biggest source of website URLs”
- 36. Web Logs
- 37. Web Logs
- 38. Web Logs
- 39. The most accurate source about crawl budget
- 40. What is the crawl budget?
- 42. How does a Googlebot allocate the crawl budget for your website?
- 43. How much time it allocates: Authority * Amount of acquired new info
- 44. How do you use that time: Speed
- 45. How do you care about future crawling: Internal linking
- 46. “People choose the paths that grant them the greatest rewards for the least amount of effort. “ Dr. House / David Shore
- 47. “Crawlers choose the paths that grant them the greatest rewards for the least amount of effort.“ Filip Podstavec
- 48. “Crawlers choose the paths that grant them the greatest rewards for the least amount of effort.“ Filip Podstavec
- 49. Do I have logs? Yes, but I don’t know about that No Yes
- 50. mktfest_com.log (146.55 GB) podstavec_cz.log (11.07 GB)
- 51. Tools for log analysis
- 53. Static
- 54. < 50MB (Small Files)
- 55. 50MB - 10GB (Medium Files) Google BigQuery OpenRefine Screaming Frog Log Analyzer
- 56. > 10 GB (Large Files) Google BigQuery
- 57. Real-time
- 59. Real-time logs
- 60. = 20 minutes = 20$ / m + Server
- 62. What to check in logs: (diagnosis)
- 63. #1 Which search engine robots do crawl my website and how much?
- 64. Bot requests in last 7 days Googlebot Bingbot Slurp SeznamBot Other
- 65. Bot requests in last 7 days #1 Bot requests Googlebot Bingbot Slurp SeznamBot Other Googlebot Bingbot Slurp SeznamBot Other
- 66. DESKTOP : MOBILE : IMAGE Mozilla/5.0 (compatible; Googlebot/2.1; +http://www. google.com/bot.html) Googlebot-Image/1.0 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safa- ri/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- 67. #2 Googlebot agents Mozilla/5.0 (compatible; Googlebot/2.1; +http://www. google.com/bot.html) Googlebot-Image/1.0 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safa- ri/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- 68. #3 Which pages do they visit with the highest frequency?
- 69. Most visited URLs by Googlebot
- 70. Most visited URLs by Googlebot #3 Most visited URLs
- 72. .txt
- 73. #4 Which status codes and how much of them does a Googlebot crawl?
- 75. #4 Pie chart of Googlebot status codes 200 404 301 302 500 200 404 301 302 500
- 77. Where/when/why do they request error pages?
- 78. Googlebot hits with status code higher than 399
- 80. #6 User errors
- 81. Does somebody crawl my website?
- 82. Of your traffic are bots! 61.8%
- 87. #7 IP Requests #8 Googlebot IPs
- 89. What can you do? (content point of view)
- 90. Content
- 91. is
- 93. What can you do? (technical point of view)
- 98. “Be in touch with somebody responsible for infrastructure!” Filip Podstavec
- 101. DISALLOW
- 102. “Not all of your URLs should be crawled or indexed”
- 103. Links
- 105. <a href=”a/”> /a/ /a/a/ /a/a/a/ /a/a/a/a/
- 108. Rel=”prev” <link rel=”prev” href=”http://www.example.com/page1” /> Rel=”next” <link rel=”next” href=”http://www.example.com/page3” /> Noindex,follow <meta name=”robots” content=”noindex, follow”>
- 109. Sanitize the filter combinations
- 110. http://edition.cnn.com/videos/tech/2016/08/26/black-hole-breakthrough-lee-pkg.cnn
- 112. 95 367 431 640 625 Variants
- 114. 3.8 Years
- 115. How to fix that? Create rules like:
- 116. Disallow filters without search volume (price, etc.) Example: Eshop.tld/notebooks/price-0-1000/ Robots.txt block: Disallow: */price
- 117. Disallow more than one used filter from each segment Example: Eshop.tld/notebook/acer,apple,lenovo/ Robots.txt block: Disallow: /*,*,*
- 118. Delete unused filters by users
- 119. Use pseudo checkbox Example: <input><label><a>Select link</a></label> Example URL: https://notebooky.heureka.cz/
- 120. Avoid thin content crawling
- 125. Google Lighthouse
- 127. Keep your sitemap clean
- 129. Are you happy?
- 132. Thank you! Filip Podstavec THE MAIN CONSTRUCTER OF MARKETING MINER FILIP@MARKETINGMINER.COM