Fun Learning Web Scraping - QueensJS - 9/2/2015
Download as PPTX, PDF1 like564 views
The document discusses different methods of screen scraping, including simple scraping using regex and XML parsing, advanced scraping using layouts and AJAX support, and browser emulation scraping. It also provides tips for website owners to make scraping more difficult, such as avoiding IDs, randomizing layouts, using AJAX, and implementing authentication. While scraping may still occur, the document suggests integrating potential attack vectors as new features and not viewing individual scraping as an attack.
Fun Learning Web Scraping - QueensJS - 9/2/2015
- 3. • Director of Engineering/Operations at ClassPass • Full stack engineer, DevOps day to day • First love is Javascript
- 4. Screen Scraping You should know about it
- 7. • Layout dependent • Difficult to traverse • Error prone • RegEx not made for XML • Doesn’t support AJAX
- 9. • Layout dependent • Doesn’t support AJAX
- 11. • Layout dependent • Slow
- 13. Screen Scraping You can beat it
- 14. Don’t make it easy • Are you using IDs? • Are you HTML templating? • How identifiable are your components? • Randomly break your layout to stop scripts
- 15. All about AJAX • Defeats most RegEx scrapers • Defeats most XMLParsers • Browsers can render AJAX • Don’t use easy to access AJAX routes • https://queensjs.com/page/2 • use https://queensjs.com/_ajax/pagination&?q=1
- 16. Know your enemy! • CSRF tokens for fields • User authentication required • Use a paywall to discourage anonymity • Trace IP addresses • DMCA takedown request
- 18. Screen Scraping …it’ll still happen
- 19. Why? • Your data is amazing • You don’t have an API • Backdoor feature request • Hackers!
- 20. Scraping happens but who does it?
- 28. Is it worth it?
- 29. Costs • Development time • Anchoring bad features • Never stops • Alienating good engineers
- 30. What can you do? • No, really don’t use IDs. • Layout changes do break scraping • Integrate attack vectors as new product features • Don’t panic
- 31. • Scraping on an individual level is a query • Scraping in a cluster is an attack
Editor's Notes
- #11: What’s different here? The previous example recreates the DOM from an XMLHTTPRequest and parses it using JSDOM. This example uses X-ray, which by default uses Request, but can be backed by PhantomJS driver. This spawns a headless browser, then drives it remotely. This is SLOW.
- #26: Site crawling
- #27: Well meaning individuals
- #28: Hackers!