Challenges Encountered by Scaling Up Recommendation Services at Gravity R&D
1 like1,673 views
Gravity R&D is a global recommender system vendor that faces challenges in scaling its services, particularly in managing high request volumes and optimizing technology. The company's infrastructure includes over 200 hosts and utilizes multiple advanced algorithms to ensure real-time response and personalization for businesses. Key areas of focus include data security, efficient resource management, and flexibility in meeting various business requirements.
Challenges Encountered by Scaling Up Recommendation Services at Gravity R&D
- 1. Scaling UP Challenges Encountered Scaling Up Recommendation Services @Gravity R&D Bottyán Németh
- 2. Who we are and what we do Gravity R&D is a recommender system vendor company. We provide recommendation as a service since 2009 for our customers all around the globe. 2
- 3. How we imagine growth? 3 ?
- 4. How we imagine growth? 4
- 5. How it actually happens? 5 ?
- 6. How it actually happens? 6
- 7. # of requests 7 Vatera.hu largest online marketplace in Hungary served by one “server” Alexa TOP100 video chat webpage (~40M recommendation requests / day): Served by 5 application servers and 1 DB Too many events to store in MySQL using Cassandra (v0.6) Training time for IALS too long speedup by IALS1 Max. 5 sec latency in “product” availability
- 8. Using new/beta technologies 8 Cassandra (v0.6) Nginx (v0.5) (22% of top 1M sites) Kafka (v0.8) MySQL auto. failover
- 9. Reaching the limits 9 Even if the technology is widely used if you reach it’s limits the optimization is very costly / time consuming. Java GC – service collapsed because increased minor GC times due to a JVM bug (26th of January 2013) Maintaining MySQL with lots of data (optimize table, slave replication lag, faster storage device)
- 10. Complexity increases 10 There is always a business request or an algorithmic development which requires more resources.
- 11. Optimizations 11
- 12. Infrastructure 12 Currently 200+ hosts and 3500+ services monitored 0 50 100 150 200 250 2008 2009 2010 2011 2012 2013 2014 2015 2016 Number of servers
- 13. # of items 13 How to store item model / metadata in memory to serve requests fast?
- 14. # of items 14 How to store item model / metadata in memory to serve requests fast? VS. Auto increment IDs for the items? 231 not enough
- 15. Preconceptions 15 More data better results. If the CTR of a new algorithm is low than the old algorithm is better. Daily retrain is enough.
- 16. Training frequency 16 CTR decreased in the morning
- 17. 100+ Algorithms 17 0 10 20 30 40 50 60 0 20 40 60 80 100 120 Number of times an algorithm is used
- 18. Now 18 • Performance: Gravity’s performance oriented architecture enables real-time response to the always changing environment and user behavior • Algorithms: more than 100 different recommendation algorithm enables true personalization and to reach the highest KPIs in different domains • Infrastructure: fast response times all around the globe and data security thanks to the private cloud infrastructure located in 4 different data centers • Flexibility: the advanced business rule engine with intuitive user interface allows to satisfy various business requirements Performance 140M requests served daily Algorithms 30 man-years invested Infrastructure 4 data centers globally Flexibility 100s of logics configurable
- 19. Cross the river when you come to it 19
- 20. Thank you! 20