Big Data LDN 2018: SCALING A PLATFORM FOR REAL-TIME FRAUD DETECTION WITHOUT BREAKING THE BANK
- 1. BERLIN • NEW YORK • SAN FRANCISCO • SÃO PAULO • PARIS • LONDON • MOSCOW • ISTANBUL SEOUL • SHANGHAI • BEIJING • TOKYO • MUMBAI • SINGAPORE Scaling a platform for real-time fraud detection
- 2. 2 Why is no one prepared for success?
- 3. 3 ‣ Focus on idea ‣ Focus on execution is limited in time to market ‣ Underestimating importance of scalability and profitability Startup Zeitgeist
- 4. 4 ‣ Director of Engineering ‣ Consulted with early design decisions ‣ Joined Adjust 2012 as Head of IT Operations Who am I?
- 5. Who is Adjust? 1B+ Daily active users tracked 400+ Billion Data points tracked monthly 22K+ Apps tracked
- 6. 6 ‣ Automatically reject fraudulent data before it gets paid ‣ Sends rejection reason callbacks to all parties ‣ Customers and their networks have full transparency ‣ Real-time statistical analysis of all ad engagements and app activity Fraud Prevention Suite
- 7. 7 How did we do that?
- 8. 8 Bootstrapping a product Current approach ‣ Using cloud services ‣ Base product on 3rd party IP ‣ Faster development
- 9. 9 Bootstrapping a product Current approach Our approach ‣ Using cloud services ‣ Base product on 3rd party IP ‣ Faster development ‣ Own infrastructure ‣ Develop own IP ‣ Slower ramp up
- 10. 10 Bootstrapping a product Current approach Our approach ‣ Using cloud services ‣ Base product on 3rd party IP ‣ Faster development ‣ Potential lock in ‣ More cost efficient in the beginning ‣ Shared environment ‣ Own infrastructure ‣ Develop own IP ‣ Slower ramp up
- 11. 11 Bootstrapping a product Current approach Our approach ‣ Using cloud services ‣ Base product on 3rd party IP ‣ Faster development ‣ Potential lock in ‣ More cost efficient in the beginning ‣ Shared environment ‣ Own infrastructure ‣ Develop own IP ‣ Slower ramp up ‣ Keep flexibility ‣ More cost efficient in the long run ‣ Dedicated environment
- 13. 13 General Overview Realtime Callbacks Raw Data Upload Realtime Fraud Prevention Realtime Aggregation SDK Traffic Tracking Server Device Data Store
- 14. 14 So it begins…
- 15. 15 Redis Small haproxy Redis Leader Redis Replica
- 16. 16 A little more please?
- 17. 17 Redis Big haproxy Redis Leader Redis Replica
- 18. 18 We need to go further
- 19. 19 It’s all connected :) Leader 1 Replica 1 Leader n Replica n Sentinel 1 Sentinel n twemproxy twemproxy
- 20. 20 And we felt like…
- 21. 21
- 22. 22 ‣ Automation is tricky Smoke in the machine room
- 23. 23 ‣ Automation is tricky ‣ Weird latency spikes Smoke in the machine room
- 24. 24 ‣ Automation is tricky ‣ Weird latency spikes ‣ Increased average response time Smoke in the machine room
- 25. 25 ‣ Automation is tricky ‣ Weird latency spikes ‣ Increased average response time ‣ Long failover times ‣ Disruptive failovers Smoke in the machine room
- 26. 26 And we felt like…
- 27. 27
- 28. 28 Sometimes you have to let go…
- 29. 29 ‣ Only index is in RAM ‣ Data is on SSD ‣ Cost reduced by 85% ‣ Server count reduced from 40 to 6 Migrating to Aerospike
- 30. 30 ‣ 1,1 million reads per second ‣ 500.000 writes per second ‣ 180TB of data ‣ 3 locations Using Aerospike
- 31. 31 What if?
- 32. 32 ‣ Easy to scale ‣ Redis interface ‣ Online resizing ‣ All dirty work is done by Amazon Elasticache
- 33. 33 Q: Can I use Amazon ElastiCache for use cases other than caching? A: Yes. ElastiCache for Redis can be used as a primary in-memory key-value data store, providing fast, sub millisecond data performance, high availability and scalability up to 15 nodes plus up to 5 read replicas, each of up to 9.5 TiB of in-memory data.
- 34. 34 “Once we are successful, we will take care of scalability and profitability”
- 35. 35 What will happen to my infrastructure when I will be successful?
- 37. New York Paris São Paulo San Francisco London Berlin Istanbul Moscow Mumbai Beijing Seoul Tokyo Shanghai Singapore Robert Abraham DIRECTOR OF ENGINEERING robert@adjust.com ADJUST HQ Saarbrücker Str. 37a 10405 Berlin Germany