SearchLove Boston 2018 - Tom Anthony - Hacking Google: what you can learn from ethical vulnerability research
4 likes2,381 views
The document discusses insights from ethical vulnerability research and its application to SEO techniques, highlighting the potential for manipulating Googlebot and Google Search Console through undocumented functionalities and open redirects. It includes examples of using XML sitemaps and cross-domain redirects to exploit vulnerabilities for ranking improvements, resulting in significant visibility increases. The author stresses the importance of securing websites against such attacks and the necessity of maintaining a hacker mindset for effective SEO strategies.
SearchLove Boston 2018 - Tom Anthony - Hacking Google: what you can learn from ethical vulnerability research
- 1. HACKING GOOGLE Learning from Ethical Vulnerability Research @TomAnthonySEO
- 3. I’d hacked into a corporate network…
- 4. 287 Years
- 5. Bug Bounty programs didn’t exist yet…
- 6. Turns out, SEO required same mindset…
- 7. Social Network Login Status Googlebot Experiments Google Manual Actions XML Sitemaps Manipulation
- 8. Turns out you can combine SEO + Vulnerability Research… • 1st page of results • 6 days old domain • 0 links
- 9. Allowing for SEO results like this, one day to the next…
- 11. DETECT WHICH SOCIAL NETWORKS PEOPLE ARE LOGGED INTO http://www.tomanthony.co.uk/tools/detect-social-network-logins/
- 13. facebook.com/tomsprofile facebook.com/login?continue=/tomsprofile 302 redirect to: Page served (200): facebook.com/tomsprofile
- 14. facebook.com/login?continue=/tomsprofile facebook.com/tomsprofile facebook.com/login?continue=/tomsprofile Page served (200): 302 redirect to:
- 15. facebook.com/login?continue=/tomsprofile facebook.com/tomsprofile facebook.com/login?continue=/tomsprofile Page served (200): 302 redirect to: Already logged in, so just redirect to the intended page.
- 16. facebook.com/login?continue=/logo.png facebook.com/logo.png facebook.com/login?continue=/logo.png Page served (200): 302 redirect to:
- 17. facebook.com/login?continue=/logo.png facebook.com/logo.png facebook.com/login?continue=/logo.png Page served (200): 302 redirect to: Already logged in, so redirect and serve the image.
- 20. OTHERS HAVE EXTENDED IT https://robinlinus.github.io/socialmedia-leak/
- 21. PUMP IT INTO GA
- 23. LOGGED IN TO A COMPETITOR?
- 24. Social Network preferences can be recorded*/used. TAKEAWAY * something something GDPR
- 25. Redirects can be abused to get unexpected behaviours OBSERVATION
- 29. GOOGLEBOT JAVASCRIPT - RANDOM IS NOT RANDOM! Math.random() = 0.19426893815398216
- 30. SCRIPTS THAT SHOULD BE RANDOM CAN DETECT GOOGLEBOT
- 31. GOOGLEBOT JAVASCRIPT - SECONDS ARE NOT SECONDS setTimeout(doSomething, 5000) Browsers will wait 5000 milliseconds (5 seconds). GoogleBot fast forwards (dates become wrong).
- 32. GoogleBot does accept cookies in certain scenarios. TAKEAWAY
- 33. GoogleBot uses heavily optimised Javascript. TAKEAWAY
- 34. There are undocumented functionalities in GoogleBot OBSERVATION
- 38. The Snag
- 44. GOOGLE HAPPY TO SHARE
- 45. POSSIBLE TO SIMPLE CRAWL LIST OF MANUAL ACTIONS
- 46. SOME SITES HAD MORE PROBLEMS THAN OTHERS…
- 47. At this point, I’d confirmed there was a definite security issue, and reported it to Google.
- 49. OMINOUS MATT CUTTS EMAILS… :D * Matt was actually great
- 50. GOOGLE RESPONSE ✓ Acknowledged report in only 11 minutes! ✓ Triaged in a couple of hours. ✓ Fixed and back online in 4 days. ✓ $5000 bounty.
- 51. A lot more sites have manual penalties than you may think! TAKEAWAY
- 52. Google Search Console also has security gaps OBSERVATION
- 53. So far not shown any direct manipulation of rankings…
- 54. Disclaimer: Distilled don’t condone blackhat. Blackhat is naughty & bad.
- 56. Redirects can be abused to get unexpected behaviours OBSERVATION
- 57. There are undocumented functionalities in GoogleBot OBSERVATION
- 58. Google Search Console also has security gaps OBSERVATION
- 59. Can we put all that together?
- 60. SUBMITTING AN XML SITEMAP ✓ Search Console ✓ robots.txt
- 61. SUBMITTING AN XML SITEMAP
- 62. SUBMITTING AN XML SITEMAP Not entirely true…
- 63. CAN SUBMIT NEW SITEMAP FILES VIA THE PING URL ✓ Typically crawled within seconds ✓ No auth - ping sitemaps for any domain ✓ Google follows redirects
- 64. CAN SUBMIT NEW SITEMAP FILES VIA THE PING URL ✓ Typically crawled within seconds ✓ No auth - ping sitemaps for any domain ✓ Google follows redirects Interesting…
- 65. GOOGLE’S CHECKLIST FOR A VALID XML SITEMAP ✓ Sitemap must be correctly formatted ✓ The URLs must exist ✓ Site containing the URLs must be in GSC ✓ Site hosting the sitemap must be in GSC
- 66. GOOGLE’S CHECKLIST FOR A VALID XML SITEMAP ✓ Sitemap must be correctly formatted ✓ The URLs must exist ✓ Site containing the URLs must be in GSC ✓ Site hosting the sitemap must be in GSC Interesting…
- 67. GOOGLE’S CHECKLIST FOR A VALID XML SITEMAP ✓ Site hosting the sitemap must be in GSC
- 68. OBSERVATIONS ✓ Google follows redirects ✓ Site hosting sitemap must be in GSC
- 69. OBSERVATIONS ✓ Google follows redirects ✓ Site hosting sitemap must be in GSC ✓ Will Google follow a x-domain sitemap redirect? ✓ Will they ‘trust' it? QUESTIONS
- 70. Will Google follow a cross domain redirect for a sitemap? QUESTION ?
- 71. SIMPLE TEST 1. Hosted a sitemap.xml on blue.com 2. Setup a redirect script on green.com 3. Ping green.com?next=blue.com/sitemap.xml https://www.google.com/webmasters/sitemaps/ping? sitemap=http://green.com/next/blue.com/sitemap.xml
- 72. Will Google follow a cross domain redirect for a sitemap? YES
- 73. Will they ‘trust' it? (if submitted via ping url) QUESTION ?
- 74. Will they ‘trust' it? Let’s assume…
- 76. JONO IS OUR INNOCENT VICTIM jono.com
- 77. REDIRECT URLS STRIKE BACK! jono.com/logout?continue=/page.html
- 78. VICTIM & ATTACKER jono.com tom.com
- 79. OPEN REDIRECTS (CROSS DOMAIN) jono.com/logout?continue=tom.com/page.html
- 80. WHAT HAPPENS IF WE DO THIS? jono.com/logout?continue=tom.com/evil.xml
- 81. WHAT HAPPENS IF WE DO THIS? jono.com/logout?continue=tom.com/evil.xml URL on jono.com, but serves XML Sitemap from tom.com.
- 82. WHAT HAPPENS IF WE DO THIS??? https://www.google.com/webmasters/sitemaps/ping? sitemap=http://jono.com/logout?continue=tom.com/evil.xml
- 83. WHAT HAPPENS IF WE DO THIS??? https://www.google.com/webmasters/sitemaps/ping? sitemap=http://jono.com/logout?continue=tom.com/evil.xml Ping the URL to submit the sitemap. Will Google think the evil sitemap belongs to jono.com?
- 84. PINGING SITEMAPS CROSS-DOMAIN ✓ Google follows the redirect, and crawls it. ✓ Google trusts it as canonical to the originating domain. https://www.google.com/webmasters/sitemaps/ping? sitemap=http://jono.com/logout?continue=tom.com/evil.xml
- 85. WE CAN NOW SUBMIT TRUSTED SITEMAPS FOR OTHER SITES
- 86. We can now submit hreflang entries for other sites…
- 87. Lets try it in the wild…
- 88. DISCLAIMER I’m showing real results, but an alternative (similarly sized) UK retailer in the screenshots.
- 89. EXPERIMENT: HIJACK TESCO.COM INTERNATIONAL EQUITY
- 90. UK PRESENCE, BUT NO US PRESENCE
- 91. HIJACK UK EQUITY TO RANK IN US
- 92. HIJACK UK EQUITY TO RANK IN US
- 93. STEP 1: FIND A REDIRECT
- 94. STEP 1: FIND A REDIRECT
- 95. STEP 2: REGISTER A DOMAIN ($12) TESCOGLOBAL.COM
- 96. STEP 3: SETUP A NEW SITE ✓ Scrape contents for products/categories ✓ Mirror the URL structure
- 97. STEP 4: CREATE AN EVIL SITEMAP
- 98. STEP 5: PING OUR EVIL SITEMAP (HOSTED ON OUR FAKE SITE) https://www.google.com/webmasters/sitemaps/ping? sitemap=http://www.tesco.com/logout? continue=http://tescoglobal.com/sitemap_global.xml
- 99. RESULTS: CRAWL ACTIVITY APPEARS IN SEARCH CONSOLE
- 100. RESULTS: SEARCH VISIBILITY GROWS RAPIDLY
- 101. RESULTS: TRAFFIC APPEARING IN GA
- 102. RESULTS: BRITISH TERMS RANKING 1ST FOR MANY QUERIES
- 103. Still only submitted a sitemap, nothing else.
- 104. RESULTS: TRAFFIC KEEPS ON INCREASING…
- 105. RESULTS: SEARCH VISIBILITY GROWS MORE
- 106. RESULTS: I HIT FIRST PAGE FOR COMPETITIVE MONEY TERMS… • 1st page of results • 6 days old domain • 0 links
- 107. RESULTS: MILLIONS OF SEARCH IMPRESSIONS
- 108. ‘LINKS’ APPEAR IN GSC — SHOWING GOOGLE TRUSTS THE SITEMAP
- 109. EARLIER: CAN’T SUBMIT SITEMAPS IN GSC WHEN NOT PERMITTED SITEMAP NOT PERMITTED EXAMPLE:
- 110. NOW: CROSS SUBMITTED THE SITEMAP TO MY GSC, AND IT WAS ALLOWED SITEMAP NOT PERMITTED EXAMPLE: SITEMAP FOR "TESCO.COM" URLS WAS ALLOWED IN “TESCOGLOBAL.COM" GSC:
- 111. EVEN TRACKS INDEXATION… SITEMAP NOT PERMITTED EXAMPLE: SITEMAP FOR "TESCO.COM" URLS WAS ALLOWED IN “TESCOGLOBAL.COM" GSC:
- 112. SUMMARY ✓ Budget: $12 ✓ Setup time: ~4 hours ✓ Other activity: nothing ✓ Links: 0 ✓ Impressions: > 1.5 million ✓ Clicks: > 12,000
- 113. ALMOST UNDETECTABLE
- 114. DEFENCE ✓ No open redirects ✓ If you have them - block in robots.txt ✓ Have a sitemap, with hreflang & media entries ✓ Hide your sitemaps ✓ Check 302s in logs
- 115. FINDING OPEN REDIRECTS ✓ Look for redirect parameter (e.g continue= or next=) ✓ Check login & logout URLs ✓ Site searches, e.g site:www.foo.com inurl:=http ✓ Better with specific sections: site:www.foo.com/bar inurl:=http ✓ Check not blocked in robots.txt ✓ Check openbugbounty.org
- 116. GOOGLE OFFICIAL RESPONSE ✓ I reported it in September 2017 ✓ March 2018 - Google award a bug bounty ✓ March 2018 - Google confirm it is fixed. ✓ April 2018 - Google increase the bug bounty ($5000)
- 117. hreflang entries are ignored if your sitemaps are unverified TAKEAWAY
- 118. Ensure you do not have open redirects on your site! TAKEAWAY (robots.txt block them if you can’t remove them)
- 119. Not seen this attack in wild. Check your logs for 302s. TAKEAWAY
- 120. Be aware, there are these types of potential attacks out there TAKEAWAY (but don’t blame everything on them!)
- 121. Bring back the Hacker Mindset TAKEAWAY