Using server logs to your advantage
Download as PPTX, PDF1 like163 views
The document discusses the importance of analyzing server logs to enhance website performance by identifying orphan pages and improving crawl efficiency. It explains that orphan pages, which are not linked from other pages, can waste Google's crawl budget and suggests strategies for detecting and managing them. Additionally, it highlights methods to improve crawl efficiency by addressing low-traffic pages and optimizing page load times.
Using server logs to your advantage
- 1. Alex Johnson alex (at) white.net @alex_cestrian
- 2. USING SERVER LOGS TO YOUR ADVANTAGE
- 3. @alex_cestrian #OptimiseOxford#OptimiseOxford What are server logs?
- 4. @alex_cestrian #OptimiseOxford#OptimiseOxford A server log is a simple text file which records activity on a server.
- 6. @alex_cestrian #OptimiseOxford#OptimiseOxford So why bother looking at server logs?
- 7. @alex_cestrian #OptimiseOxford#OptimiseOxford There is only one resource that tells you what search engines are looking for on a domain… These are web server logs. including stuff they found 13 years ago.
- 8. @alex_cestrian #OptimiseOxford#OptimiseOxford How do we analyse all that data?
- 14. @alex_cestrian #OptimiseOxford#OptimiseOxford Scenario 1 IDENTIFY ORPHAN PAGES
- 15. @alex_cestrian #OptimiseOxford#OptimiseOxford An orphan is a page that is not linked to by another page on the site. Homepage Dresses Skirts Our offers Summer 2016 offers
- 17. @alex_cestrian #OptimiseOxford Why are orphan pages bad? • There may be a lot of them, and they may be competing with your ‘live’ content • They waste GoogleBot’s crawl budget for your domain
- 18. @alex_cestrian #OptimiseOxford#OptimiseOxford So how do we find orphan pages using log files?
- 19. @alex_cestrian #OptimiseOxford Upload a crawl of your website (from SF, DeepCrawl etc) URLs that return a 200 ✅ status code… that don’t appear in the crawl of your site
- 20. @alex_cestrian #OptimiseOxford Redundant content, off little value 404/410 status code Relevant, valuable but out-of-date 301 redirect to relevant live page Useful content that orphaned accidentally Re-attach the page to the website
- 21. @alex_cestrian #OptimiseOxford If GoogleBot is wasting lots of time in a specific folder full of orphan pages that hold no value, block it via robots.txt
- 22. @alex_cestrian #OptimiseOxford#OptimiseOxford Scenario 2 IMPROVING CRAWL EFFICIENCY
- 23. @alex_cestrian #OptimiseOxford#OptimiseOxford Find where GoogleBot is wasting time Find parameter driven pages
- 25. @alex_cestrian #OptimiseOxford Block GoogleBot from crawling these URLs
- 26. @alex_cestrian #OptimiseOxford#OptimiseOxford Find infrequently visited pages Order by number of events: low to high
- 27. @alex_cestrian #OptimiseOxford#OptimiseOxford • Is this URL in the xml sitemap? • Is the page too deep within the architecture? • Is internal linking to this page optimal? • Are links to this page travelling through multiple redirects? • Can GoogleBot actually parse the links pointing to this page?
- 28. @alex_cestrian #OptimiseOxford#OptimiseOxford Look at all urls, and filter by average response time Find slow loading pages
- 29. @alex_cestrian #OptimiseOxford#OptimiseOxford If time taken is consistently high, you need to look at how you can reduce the load of the page
- 30. @alex_cestrian #OptimiseOxford#OptimiseOxford “See what GoogleBot is actually consuming. Improve GoogleBot’s diet.” Oliver Mason at Brighton SEO 2016
Editor's Notes
- #4: I’m going to talk you through 3 scenarios where logs files can help you.
- #5: I’m going to talk you through 3 scenarios where logs files can help you.
- #6: This is a raw server log file. Boring isn’t it? So what do you do with this?
- #10: Well there are a few options including tools like Botify and OnCrawl, but one of the most usable, affordable (and idiot-friendly ones) that has come onto the market in the past few years is Log Analyzer from Screaming Frog.
- #11: It’s really easy to use, you can drag and drop your raw log files (or a zip file) directly into the program, and it sorts them out into manageable sets of data.
- #12: By default the Log File Analyser only analyses search engine bot events, so the ‘Store Bot Events Only (Improves Performance)’ box is ticked. We recommend keeping this setting ticked, as it massively reduces time required to only have to store and compile search bots, rather than all event data from users and other User Agents.
- #13: And you end up with a pretty dashboard like this. Doing that alone isn’t going to solve anything, so I’m not going to show you….
- #14: 3 actionable scenarios where logs files can help you do your job….
- #15: Let’s start with what is an orphan page?
- #16: Some websites stop linking old content that is expired and do not deliver the right status code (like a 404 or a redirect to a newer version). The expired page is thus still available.
- #21: What do you do with orphan pages when you identify them?
- #22: What do you do with orphan pages when you identify them?
- #24: Look for large quantities of parameter driven pages, and combinations of parameters. These will often be areas where GoogleBot is losing time and wasting resource.
- #25: One common example of this is on Wordpress blogs. You’ll often find things like this in your log files/
- #27: If you see category pages or main service pages at the top of this list – further investigation is much needed.
- #28: Investigate why these pages haven’t been visited by search engines;
- #30: Review each bot event for these URLs.
- #31: Oliver Mason put this eloquently in his recent talk at the Brighton SEO conference:
- #32: That’s just an overview of a few things you can do with log files. Once you start playing around and analysing the data, it’s really rather interesting.