Case Study on Property Portal Data Security
Download as PPTX, PDF1 like668 views
This document discusses how advanced persistent bots pose a threat to real estate portals by scraping listing data and spamming contact forms. Traditional homegrown solutions are ineffective against sophisticated bots. Lamudi, a global property portal, was experiencing performance issues and data scraping due to bot traffic 15x higher than human traffic. Lamudi implemented Distil's bot detection and mitigation solution, which uses techniques like device fingerprinting and browser validation to accurately identify and block bots without impacting users. This resulted in the elimination of data scraping, reduced form spam, improved performance, and saved engineering resources.
Case Study on Property Portal Data Security
- 1. Securing Property Portals Lamudi Case Study
- 2. Platforms Portals Brokerages MLS Customers Premium Brands Distil in Real Estate and Premium Brands
- 3. The New Threat Landscape of APBs Advanced Persistent Bots (APBs)... Advanced Mimick human behavior Load JavaScript Load external resources Support cookies Browser automation (Selenium, PhantomJS) Persistent Dynamic IP rotation Distribute attacks across IP addresses Hide behind anonymous and peer-to-peer proxies 2015 Distil Bad Bot Report
- 4. Homegrown Solutions Are Ineffective Creates a poor user experience Bots appear human in logs Defeated by distributed IP attacks Defeated by advanced bots Labor intensive Defeated by low and slow crawlers Defeated by CAPTCHA farms Distributed attacks hard to pinpoint Defeated by peer-to-peer / proxies Reduces conversions by up to 27% Reactive in nature Reactive in nature
- 5. Web App Security Requires Complementary Solutions l DDoS Mitigation Firewall WAF Distil Bot Protection Core Competency Volumetric attacks on infrastructure Network layer attacks Application coding exploits Automated abuse, misuse, and attacks (scraping, fraud, account takeover, etc.) Techniques Scrubbing centers, Large pipes Access Control Lists (ACLs), Rules-Based App layer understanding, ACLs, Rules-Based Real-time Analysis, Fingerprinting, Honeypotting, Machine learning, Behavioral modeling
- 6. Survey Respondents 100 real estate executives representing over 600,000 realtors 14 real estate portal operators running 400,000 real estate websites 2015 Real Estate Web Scraping Survey
- 7. β 50% - 75% of bot traffic is from Consumer ISPs β 7 of top 10 sources of bad bots are Consumer ISPs β Most Consumer ISPs had 1,500+ IPs with bots Highlights of Bot Sources on Real Estate Websites The Facts on Scraping Real Estate Data Top 7 Consumer ISPs with Bot Traffic 1 Comcast 2 Time Warner Cable 3 Verizon FIOS 4 Charter 5 Cox 6 CenturyLink 7 AT&T Uverse Highlights of Bot Sophistication β 18-45% Automated browsers - mimicking humans β 14-25% Already in bot database - fingerprinted, known bots β 16-42% Slow crawlers - recycling IPs and user agents
- 8. About Lamudi 30+ Countries 900,000+ Listings 660+ Employees Property portal focused exclusively on emerging markets
- 9. Lamudi Bad Bot Challenges Bad Bot Challenges Bad guys scraped listing data to duplicate listings, impact SEO, and compete w/Lamudi Bots are spamming listing agent/owner contact forms & reducing agent retention & satisfaction 15,000 bad bot requests per minute (15x human traffic) caused slowdowns WAF-based IP blocking system used enginering time and was ineffective
- 10. Lamudi Selection Criteria Bot Detection and Mitigation Solution Requirements Support a complex deployment across several AWS instances with Akamai Block web scrapers and spammers without impacting human visitors Accurately identify good bots vs. bad bots Increase website availability and speed Detect automated browsing tools Simple setup for 30+ domains Little or no maintenance, βself-optimizingβ solution
- 11. Lamudi Results with Distil Results with ROI No more scraping data β unique listings = better SEO No more form spam to agents β higher value leads = $$ Less time addressing agent complaints β Rev. Retention = $$ Increased website performance β Faster site = better SEO Save 100 engineering hours/mo. β More resources! Save $$ βDistil is the best anti-bot and anti-scraper protection solution available, hands down.β Oliver Fiege, CTO, Lamudi
- 12. How the Distil Bot Detection Solution Works As web traffic passes through Distil, the system 1. Fingerprints each incoming connection and compares it to our Known Violators Database 1. If itβs a new fingerprint, validates the browser to determine if itβs a Bot or Not 1. βNo Silver Bulletβ - Distil randomizes a battery of challenges to find bots and remain spoof- proof from the bot coders 1. Based on your settings, Distil automatically tags, challenges, or blocks the bot
- 13. Sticky Bot Tracking With No Impact On Real Users Device Fingerprinting Fingerprints stick to the bot even if it attempts to reconnect from random IP addresses or hide behind an anonymous proxy or peer-to-peer network Tracks distributed attacks that would normally fly under the radar Without Distil With Distil Without Impacting Users Sharing the Same IP Avoids blocking residential users or organizations that might share the same NAT as the bot or botnet
- 14. Browser Validation Detects all known browser automation tools, such as Selenium and Phantom JS Protects against browser spoofing by validating each incoming request as self reported Advanced Bot Detection Increases Accuracy Behavioral Modeling and Machine Learning Machine-learning algorithms pinpoint behavioral anomalies specific to your siteβs unique traffic patterns Self optimizing algorithms improve bot detection and mitigation without manual configuration
- 15. Awards and Analyst Recognition βAnalyzing behavior provides the best chance of detecting and blocking bot- driven attacks.β 5 Stars across the board.β Verdict: For monitoring the impact of bots on a network this is the tool one needs.β The only anti-bot solution to be included in Gartnerβs Online Fraud Detection Market Guide Ovum puts Distil Networks On The Radar. βClear innovation compared to similar services.β
- 16. www.distilnetworks.com QUESTIONSβ¦.COMMENTS? C H A R L I E @ D I S T I L N E T W O R K S . C O M 1.703.962.1614 OR CALL CHARLIE ON