What I learned from analysing thousands of robots.txt files | BrightonSEO 2020
- 1. What I learned from analysing thousands of robots.txt files samgipson # brightonSEO 2020
- 2. Of all things...why robots.txt? samgipson # brightonSEO
- 3. 2nd July 2019. samgipson # brightonSEO
- 6. samgipson # brightonSEO How many top performing sites still use unsupported or incorrect rules?
- 7. What are the most common mistakes within robots.txt? samgipson # brightonSEO
- 8. Robots.txt: The history samgipson # brightonSEO
- 9. Based on Robots Exclusion Protocol (REP) samgipson # brightonSEO
- 10. Millions of sites use a robots.txt file samgipson # brightonSEO
- 11. Despite not an official internet standard samgipson # brightonSEO
- 12. samgipson # brightonSEO Control the content crawlers can and can’t access
- 13. It’s hugely powerful. Mistakes can cost you big. samgipson # brightonSEO
- 14. Did you guess the year? samgipson # brightonSEO
- 16. In 2019 Google submitted a revised REP draft to try to make it an official standard samgipson # brightonSEO FACT
- 17. Robots.txt: The basics samgipson # brightonSEO
- 18. samgipson # brightonSEO User-agent: * Disallow: /checkout/
- 19. samgipson # brightonSEO User-agent: * Disallow: /checkout/ {field}
- 20. samgipson # brightonSEO User-agent: * Disallow: /checkout/
- 21. samgipson # brightonSEO User-agent: googlebot Disallow: /checkout/
- 22. samgipson # brightonSEO User-agent: googlebot Disallow: /checkout/ {value}
- 23. samgipson # brightonSEO User-agent: * Allow:Dis /checkout/
- 24. samgipson # brightonSEO User-agent: * Disallow: /checkout/
- 25. samgipson # brightonSEO User-agent: * Disallow: /checkout/ {directive or rule}
- 26. samgipson # brightonSEO User-agent: * Disallow: /checkout/ {path}
- 27. samgipson # brightonSEO User-agent: * Disallow: /checkout/ {group}
- 28. samgipson # brightonSEO User-agent: * Disallow: /checkout/ User-agent: googlebot Disallow: /checkout/ Disallow: /basket/ {group A} {group B}
- 29. robots.txt controls crawling not indexation samgipson # brightonSEO FACT
- 31. Here’s where it got confusing... samgipson # brightonSEO
- 32. Google used to support unofficial directives samgipson # brightonSEO
- 33. samgipson # brightonSEO HTML <head> <meta name=”robots” content=”noindex, nofollow”>
- 34. samgipson # brightonSEO HTTP header X-Robots-Tag: googlebot: noindex, nofollow
- 35. samgipson # brightonSEO robots.txt User-agent: googlebot Noindex: /checkout/ Nofollow: /checkout/
- 36. Webmasters / SEOs realised that noindex: worked samgipson # brightonSEO
- 40. Identified top traffic driving sites across a range of sectors samgipson # brightonSEO
- 41. Automotive Computing Cooking/Recipes Electronics Fashion Gambling Hardware samgipson # brightonSEO
- 44. Extracted the robots.txt files for 40,000 unique domains samgipson # brightonSEO
- 48. Results: Unsupported rules samgipson # brightonSEO
- 49. samgipson # brightonSEO Out of the 40,000 site analysed, 0.5% used unsupported rules
- 50. Nofollow: samgipson # brightonSEO 1 Gambling 40,000 domains analysed
- 52. Crawl-delay: samgipson # brightonSEO Real Estate Hardware/DIY Fashion 2,600 40,000 domains analysed
- 53. Noindex: samgipson # brightonSEO 220 40,000 domains analysed
- 55. Brands using outdated rules samgipson # brightonSEO
- 56. Results: Basic Mistakes samgipson # brightonSEO
- 57. Issue 1 samgipson # brightonSEO <field> name spelt incorrectly
- 58. samgipson # brightonSEO <field> name is case insensitive FACT
- 59. This is ok: samgipson # brightonSEO User-Agent user-agent USER-AGENT UsEr-AgEnt
- 60. This ISN’T: samgipson # brightonSEO useragent user agent er-agent ser-agent user-agnet
- 61. <field> name errors samgipson # brightonSEO Telecoms30 40,000 domains analysed
- 62. Issue 2 samgipson # brightonSEO Incorrect user-agent <value>
- 63. samgipson # brightonSEO User-agent <value> is case insensitive FACT
- 64. This is ok: samgipson # brightonSEO Googlebot googlebot GOOGLEBOT Bingbot bingbot
- 65. This is a grey area: samgipson # brightonSEO Googlebotrandomtext Google bot goglebot Google
- 66. Issue 3 samgipson # brightonSEO Incorrect directives
- 67. samgipson # brightonSEO <directives> are case insensitive FACT
- 68. This is ok: samgipson # brightonSEO allow: ALLOW: Allow: disallow: DISALLOW: Disallow:
- 70. <directive> errors samgipson # brightonSEO All18 40,000 domains analysed
- 71. Issue 3 samgipson # brightonSEO Invalid <path> format
- 72. samgipson # brightonSEO URL <path> should start with a / FACT
- 73. This is ok: samgipson # brightonSEO Disallow: /checkout/ Disallow: /*?delivery_type Disallow: *?delivery_type
- 74. This ISN’T: samgipson # brightonSEO Disallow: .js Disallow: .css Disallow: WebResource.axd Disallow: ScriptResource.axd Disallow: js/ Disallow: http://site.com/page
- 75. Incorrect <path> samgipson # brightonSEO Equal spread across sectors 231 40,000 domains analysed
- 76. Brands using incorrect <path> samgipson # brightonSEO
- 77. samgipson # brightonSEO URL <path> IS case sensitive FACT
- 79. samgipson # brightonSEO A specific user-agent overrules a catchall FACT
- 80. samgipson # brightonSEO User-agent: * Disallow: /checkout/ Disallow: /*?delivery_type
- 81. samgipson # brightonSEO User-agent: * Disallow: /checkout/ Disallow: /*?delivery_type User-agent: googlebot Disallow: /another-folder/
- 82. samgipson # brightonSEO User-agent: * Disallow: /checkout/ Disallow: /*?delivery_type User-agent: googlebot Disallow: /another-folder/
- 83. samgipson # brightonSEO User-agent: * Disallow: /checkout/ Disallow: /*?delivery_type User-agent: googlebot Disallow: /checkout/ Disallow: /*?delivery_type Disallow: /another-folder/
- 84. samgipson # brightonSEO The order of <directives> doesn’t matter for most bots FACT
- 85. samgipson # brightonSEO Specificity (length) of the matching rule wins FACT
- 90. samgipson # brightonSEO You can group user-agents together FACT
- 91. samgipson # brightonSEO User-agent: googlebot Disallow: /checkout/ Disallow: /*?delivery_type User-agent: bingbot Disallow: /checkout/ Disallow: /*?delivery_type
- 92. samgipson # brightonSEO User-agent: googlebot User-agent: bingbot Disallow: /checkout/ Disallow: /*?delivery_type
- 94. samgipson # brightonSEO Google are pushing for REP to become an Internet standard
- 95. samgipson # brightonSEO We should all be pushing for a best practice robots.txt
- 96. samgipson # brightonSEO Avoid Google having to make allowances for inaccuracies
- 97. samgipson # brightonSEO Who knows… they may suddenly stop
- 98. samgipson # brightonSEO Get the basics right. Many big brands aren’t.
- 103. samgipson # brightonSEO Tools Articles Chrome Extension: Robots Exclusion Checker samgipson.com/robots/ ContentKing: Robots.txt for SEO contentkingapp.com/academy/robotstxt/ Ayima: Robots.txt Parser ayima.com/robots/ Builtvisible: An SEO Guide to Robots.txt builtvisible.com/wildcards-in-robots-txt/ Google’s Webmaster Robots.txt Testing Tool google.com/webmasters/tools/robots-testing-tool Original Robots.txt Draft (1996) robotstxt.org/norobots-rfc.txt Google’s C++ robots.txt parser and matcher github.com/google/robotstxt Google’s Robot Exclusion Protocol Draft (2019) ietf.org/archive/id/draft-rep-wg-topic-00.txt