apidays LIVE LONDON - API scrapping: how to protect your API against something which is not necessarily an attack by Artem Demchenkov
0 likes52 views
API scraping involves automated extraction of content from APIs for purposes outside of their intended use. While not necessarily malicious, it can harm API owners through lost revenue and intellectual property. To recognize scraping, API owners monitor for abnormal usage patterns and set limits on requests per user. Methods to prevent scraping include rate limits, firewall rules blocking suspicious traffic, IP databases, and machine learning models to detect bots. However, completely blocking scraping may not be possible due to the dynamic nature of the problem.
apidays LIVE LONDON - API scrapping: how to protect your API against something which is not necessarily an attack by Artem Demchenkov
- 1. API Scrapping. How to protect your API against something that is not necessarily an attack Artem Demchenkov Illustration © Caterina Carraro/Billie
- 2. 2 Artem Demchenkov ● CTO & Co-Founder at Billie ● ardemchenkov@gmail.com ● https://www.linkedin.com/in/artem-demchenkov-76b69934/
- 3. Billie We are an innovative financial platform, focused on working capital financing. Some numbers: ● Start: 01.12.2016 ● Closed Beta: 01.05.2017 ● Public Launch: 01.06.2017 ● Team: 100+ ● Engineering Crew: 35
- 4. Agenda 4 ● API scraping may not look like an attack ● Why should we worry then? ● KYCB ● How to recognize that you’re being scrapped ● How to stop API scraping. It is even possible? ● Bonus
- 5. 5 Let’s begin with numbers
- 6. Some numbers 6 ● 46% of web traffic are scraping bots (Distil Networks, 2016) ● 2% of online revenue is lost due to web scraping (Distil Networks, 2016) ● Global e-commerce sales are 3.53 trillion dollars (Statista, 2019) ● 2% of it is >70 billion dollars
- 8. API Attacks 8 ● DDoS ● Injections ● Data Exposure ● Authentication Hijacking ● “Man in the Middle” ● Unencrypted Communication ● Application Abuse ● Parameter Tampering }Unexpected API usage
- 9. 9 API Scraping is different
- 10. API Scraping 10 ● Is a pretty traditional and expected user behaviour ● It doesn’t try to hijack or break anything ● Content - is the main point of interest ● That’s why it is not always easy to identify that you are being scraped
- 11. Traditional communication flow 11 USER UI API DATA SOURCE
- 12. Traditional communication flow 12 USER UI API DATA SOURCE
- 13. Scraping communication flow 13 USER UI API DATA SOURCE
- 14. 14 Scraping is not allowed
- 15. Google Maps Terms of Service 15 3.2.3 Restrictions Against Misusing the Services. (a) No Scraping. Customer will not export, extract, or otherwise scrape Google Maps Content for use outside the Services. For example, Customer will not: (i) pre-fetch, index, store, reshare, or rehost Google Maps Content outside the services; (ii) bulk download Google Maps tiles, Street View images, geocodes, directions, distance matrix results, roads information, places information, elevation values, and time zone details; (iii) copy and save business names, addresses, or user reviews; or (iv) use Google Maps Content with text-to-speech services.
- 16. 16 But not everyone is Google, isn’t it?
- 17. Scraping communication flow 17 USER UI API DATA SOURCEBOT
- 18. Scraping communication flow 18 USER UI API EXTERNAL APIBOT
- 19. What makes API Scraping dangerous for API Owners 19 ● Unpredictable expenses ● Income losses ● Loss of intellectual property ownership ● Loss of competitive advantage ● Risk of being reverse engineered
- 20. 20 KYC Know Your Customers KYCB Know Your Customers’ Behavior
- 21. How to recognize that your API is being scrapped? 21 Logging Thresholds Monitoring alerts
- 22. How to set up a proper Data Monitoring Smart Data-driven Alerts with Prometheus and Grafana https://youtu.be/GbwyF6xZwwc
- 23. How to prevent API Scraping 23 HTTP-Request limits Per user, Per time period Basic Firewall rules Headers, Content-Size, IPs More extended rules Geography, Pattern Detection IP databases https://www.abuseipdb.com/ Real-time ML based bot detection Specific services (e.g. Cloudflare) Data thresholds Num or registrations, num of API calls...
- 24. 24 They say: no one can block API scraping completely
- 25. 25 We say: it is a race and one who stops first loses
- 26. 26 Bonus Track
- 27. Scraping communication flow 27 USER UI API DATA SOURCEBOTHUMAN This one leaves traces
- 28. 28 Prior to building a bot, a human needs to explore your API
- 29. Thank you ● ardemchenkov@gmail.com ● https://www.linkedin.com/in/artem-demchenkov-76b69934/
- 30. Check it out. There’s interesting More in our “Engineering Corner” on Medium 30 https://medium.com/billie-finanzratgeber