Evaluating URLs at Scale
0 likes152 views
This document discusses tools for evaluating URLs at scale. It provides an overview of commonly used tools for backlink analysis, site migrations, site structure visualization, extracting specific data, analyzing single-page applications, and conducting content audits. Examples of popular tools mentioned include Screaming Frog, Majestic, Ahrefs, SEMRush, DeepCrawl, and RStudio. The document also provides examples of how to use some of these tools to perform initial backlink audits, site migrations, and extract custom data.
Evaluating URLs at Scale
- 1. Evaluating URLs at scale 26th May 2019
- 2. What was I thinking? Maybe, just maybe, should have been called: A guide to an SEO toolkit
- 3. Quick BIO • Head of Digital Marketing at Bray Leino CX – 14 whole days • Head of SEO - 2 years • Senior SEO Manager – 2 years • Ashridge Trees – Marketing Manager – 1 year (was a bad idea) • Head of SEO – 3 years – SIFT / PracticeWEB • Run a team of 4 (we’re looking for a technical SEO exec to join us) covering Digital Marketing
- 4. How many URLs can you check? 10, 100, 1000, 10,0000, a million, 100 million? There comes a point where you want to look at them without having to look at individual URLs.
- 5. URL Tool Kit* * There are other tools. Many. Many other tools. Crawling Backlinks Site Speed SEMRush SEMRush Sitespeed.io SiteBulb Majestic Lighthouse Xenu Link Sleuth Ahrefs URL Profiler Screaming Frog URLProfiler Custom Built Botify Bright Local DeepCrawl
- 6. • Backlink analysis • Migrations • Site Structure (Internal linking visualisation) • Specific Data (authorship?) / Regex? Canonicals • Angular sites (SPA) • Content Audits What context?
- 8. Initial Audit Backlink Analysis • Get links from Search Console, Majestic, AHREFS, SEMRUSH, MOZ • Use list mode in Screaming Frog • Crawl the links • See which ones 404 or have long redirect chains • Fix them!
- 9. Initial Audit Backlink Analysis • Get links from Search Console, Majestic, AHREFS, SEMRUSH, MOZ • Use list mode in Screaming Frog • Crawl the links • See which ones 404 or have long redirect chains • Fix them!
- 10. Migration
- 11. Moving site Migration • Crawl the existing site in spider mode • Crawl the development site in spider mode • Export HTML pages from existing site • Change domain name in excel (R-Studio if there are too many) • Use List mode in SF • Look at 404’s (these are pages that haven’t been moved to the new site) • Set-up redirect mapping • Test (recrawl)
- 12. Site Structure
- 13. Site Structure
- 14. Site Structure All nodes with no- weight have been removed. L. New int. links: 312 R. Old int. links 374
- 15. Specific Data
- 16. Specific Data • Author Data • Tag data • Unencoded URLs • Anything not captured by default
- 17. Specific Data Configuration > Custom > Extraction
- 18. Specific Data Right click on the element you want to Select
- 19. Specific Data Paste in to the extractor and rename This data will be available at the URL level for any page that has it
- 20. JS and SPA’s
- 21. Problems Angular JS and SPA’s
- 22. Problems Angular JS and SPA’s
- 23. Problems Google is pretty good at crawling JS but not all links on a page will be crawled (Google I/O May 2018) It is possible that Google can render the links in a separate action after the initial crawl Other search engines are not as good and are more likely to have problems. Angular JS and SPA’s
- 24. Content Audits
- 25. • Readability scores • Sentiment • Broadcasted keywords • Page Speed (Desktop vs Mobile) Content Audits
- 26. Content Audits
- 27. Tool Overview
- 28. Tools • Three modes • Spider • List • SERP • Very configurable. Data focused • Speed of crawl – (can be a DDOS attack, true story.) • Can specify CDN’s • Crawler identifying name • Add GA and SC • Storage (RAM vs Hard Disk) • £149 a year
- 29. Tools • Report, not crawl focused • Hints are great • Runs and reports on multiple tests • Keeps all the audits in a central place • £27.50 a month
- 30. URL Profiler Tools • Swiss army knife of tools • Great at bringing together metrics in one place per URL • £24 a month (2 devices)
- 31. R Studio • Steep learning curve • Can handle large datasets • Can use machine learning • Lots of online tutorials • Keyword mapping (comp analysis) • Link visualization • Keyword clustering • Free
- 32. Any Questions?