Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale
6 likes1,148 views
The document discusses the architectural overhaul of Spotify's ad serving technology, outlining the evolution from a legacy system to a modern full-stack solution to enhance performance and scalability. Key improvements included refactoring the ad server, optimizing delivery, and addressing technical debt to support growth and new client platforms. The author shares lessons learned through the process, emphasizing the importance of reliable data and a clear division of responsibilities in system design.
Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale
- 1. Ad Serving at Spotify Scale A journey of incremental full stack overhaul Kinshuk Mishra, Director of Engineering kinshuk@spotify.com @_kinshukmishra
- 6. The unintended consequence Artist engagement for exposed users went up
- 7. The unintended consequence Promising insights about content promotion use-case
- 8. The unintended consequence Confirmation that the ad server is a powerful messaging platform
- 9. Why should you care?
- 12. What I do ● Founded ads engineering team at Spotify in 2011 ● Build all things ads engineering - team & software ● Major focus areas : ○ Ad delivery (Backend and Web) ○ Multi-platform native ads (Client Platform) ○ Ad performance (ML and Data)
- 14. Full stack refactor Evolution at scale Pragmatic choices
- 15. 100,000,000+ MAU
- 19. $5,000,000,000+ Revenue paid to rightsholders
- 20. 60 Markets
- 23. Ad
- 28. Beauty of Ad Server Relevancy Pacing Unique View Sequence Optimization
- 29. Complexity of Ad tech ecosystem
- 30. In essence it is pretty simple Client User Profile database Ad Server Campaign Management Portal Billing/ Reporting Ad campaign database Data Collection System
- 31. Spotify Ads infrastructure in 2011 Edge Service Desktop Log Delivery HDFS User Profile Batch Basic Ad Server Campaign Management Billing/ Reporting
- 32. Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service Log Delivery GCS User Profile Targeting Service DMP Stream Batch Ad Server Decision Delivery Ad Exchanges Campaign Management Optimization Modeling Self-Serve Portal Creative Generation Payments Billing/ Reporting
- 33. Multi-platform clients iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service Log Delivery GCS User Profile Targeting Service DMP Stream Batch Ad Server Decision Delivery Ad Exchanges Campaign Management Optimization Modeling Self-Serve Portal Creative Generation Payments Billing/ Reporting
- 34. Data collection iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service Log Delivery GCS User Profile Targeting Service DMP Stream Batch Ad Server Decision Delivery Ad Exchanges Campaign Management Optimization Modeling Self-Serve Portal Creative Generation Payments Billing/ Reporting
- 35. Intelligence iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service Log Delivery GCS User Profile Targeting Service DMP Stream Batch Ad Server Decision Delivery Ad Exchanges Campaign Management Optimization Modeling Self-Serve Portal Creative Generation Payments Billing/ Reporting
- 36. Ad Delivery iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service Log Delivery GCS User Profile Targeting Service DMP Stream Batch Ad Server Decision Delivery Ad Exchanges Campaign Management Optimization Modeling Self-Serve Portal Creative Generation Payments Billing/ Reporting
- 37. Demand fulfillment iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service Log Delivery GCS User Profile Targeting Service DMP Stream Batch Ad Server Decision Delivery Ad Exchanges Campaign Management Optimization Modeling Self-Serve Portal Creative Generation Payments Billing/ Reporting
- 38. Now you know too Ad server is a powerful messaging platform
- 40. Architecture overhaul is hard ● While keeping the business running ● While innovating on new products ● When you should have done it yesterday
- 41. Why did Spotify evolve Ads architecture?
- 42. Future needs ● ● Growth in scale ● Emergence of new client platforms ● Cheap cloud computing ● New products to meet business objectives ● Technical debt
- 43. The 3 stories
- 44. Fixing the legacy mess Story 1
- 45. Original ad server design Edge Service Router hash(userid) Ad server ring with partitions Ad server instance Memcache Memcache Memcache Memcache Campaign DB User DB Desktop Rendering Ad trigger decisioning Ads Ranking Ads Caching Ad batching & fetch communication
- 46. Problems
- 47. Stateful service with faulty persistence
- 48. Cache as a data store
- 49. Service cluster as a hashed ring
- 50. Ad decisioning in Client
- 52. Fix strategy
- 54. Isolate refactor to one system at a time
- 55. The ad server transition Edge Service Log Delivery HDFS User Profile Batch Smart Ad Server Campaign Management Billing/ Reporting Ad Server Proxy (routing) Basic Ad Server Gradual transition from basic to smart ad serving Desktop Rendering Ad trigger decisioning Ads Ranking Ads Caching Ad batching & fetch communication
- 56. After the ad server transition Proxy Service Log Delivery HDFS User Profile Batch Campaign Management Billing/ Reporting Smart Ad Server Desktop Rendering Ad trigger decisioning Ads Ranking Ads Caching Ad batching & fetch communication
- 57. Lean, mean and fast Story 2
- 58. Division of responsibilities Desktop iOS Android Ads SDK Desktop Web Rendering Ad trigger decisioning Ads Ranking Ads Caching Ad batching & fetch communication Ad decisioning Ad fetch orchestration Client context Ad Trigger & Render Before After
- 59. Problems
- 60. Thick Clients
- 63. Fix strategy
- 65. Break monolith into services
- 66. Isolate platform independent logic into a lib
- 67. Fix tactic
- 68. Design your systems to be master of one thing
- 69. Remember division of responsibilities? Desktop iOS Android Ads SDK Desktop Web Rendering Ad trigger decisioning Ads Ranking Ads Caching Ad batching & fetch communication Ad decisioning Ad fetch orchestration Client context Ad Trigger & Render BAD GOOD
- 70. Multiplatform Client design iOS Proxy Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service Log Delivery GCS User Profile Targeting Service DMP Stream Batch Ad Server Decision Delivery Ad Exchanges Campaign Management Modeling Self-Serve Service Creative Generation Payments Billing/ Reporting
- 71. Knowledge is power, Unreliable data is your enemy Story 3
- 72. Event Stream Historical ETL1 ETL2 ETL3 UserEntity1(attribute1, attribute2) UserEntity1(attribute1, attribute3) UserEntity1(attribute1, attribute3’)
- 74. Problems
- 75. Duplicate, undiscoverable and fragmented datasets
- 78. Fix strategy
- 79. Focus on reliable and timely log delivery
- 81. Data engineering with SLA
- 83. Some useful lessons learnt from architectural overhaul
- 84. Test with minimal impact radius
- 86. Speed up build decisions
- 87. Think for tomorrow, Solve for today