MongoDB @ Fiverr: The Road to Atlas
0 likes411 views
Fiverr migrated their database from a self-hosted MongoDB deployment to MongoDB Atlas to reduce maintenance overhead and increase autonomy. They prepared by auditing their existing databases, applications, and infrastructure. The migration process involved automating the data transfer, addressing failures, and monitoring performance after switching to Atlas. Now Fiverr is consolidating monitoring, creating new clusters, and planning further migrations to take advantage of Atlas' features and management.
MongoDB @ Fiverr: The Road to Atlas
- 1. MongoDB @ Fiverr: The Road to Atlas 14 May 2019
- 2. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Sergei Makarevich Moshe Belostotsky Hi! We are,
- 3. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. What is Fiverr?
- 4. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Fiverr's mission is to change how the world works together. Fiver is a global marketplace connecting freelancers and businesses for their digital services needs. Meet Fiverr.
- 5. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
- 6. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Background: Fiverr development DNA Some facts... ● Task Forces - Business Domain oriented development teams ● Very fast MVP development cycle ● Many AB tests at once (80~ in parallel, 25~ new per week) ● Large amount of rollouts (52~ at once) ● High deploy rate (100~ per day) ● Ruby as the main language, Golang and Nodejs ● Micro-services architecture with CQRS (command query responsibility segregation) and event sourcing
- 7. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Once upon a time There was a tiny startup That had to get to market- FAST
- 8. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Architecture V1 – The quick way Main incentive: get to market - FAST SQL Monolith: Ruby on Rails ● Fastest to iterate ● Single SQL Database ● Memcached as caching layer Memcached READ/WRITE READ/WRITE
- 9. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Advantages of Relational Schema backed Database Why do we want to use it? ● Fast and simple integration with RoR ORM ● “Protection” from data inconsistency ● Common knowledge and practice ● Reliability and resilience
- 10. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. But then… The small startup became bigger...
- 11. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Relational Database Bottleneck The collapse of the naive approach ● Table locks on multiple insert/updates ● Severe impact on large tables with high write rate due to the locks above ● Schema changes are a “pain”
- 12. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Architecture V2 - Async writes Main incentive: Fiverr at scale SQL Ruby on Rails Monolith ● Async writes to alleviate stress from web ● RabbitMQ for passing command events RabbitMQ WorkerREAD/WRITE WRITE Memcached READ
- 13. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. And bigger...
- 14. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Problem: Relational Database slows us down We need to find another solution... ● Databases became too big with too many indexes to alter schema without downtime ● Database schema changes require “maintenance windows” ● Transactions are not “MUST” on most of the features ● The need to change schema slows the development ● Code reverts should drive schema updates as well
- 15. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Solution: Document store Database Why do we want to use it? ● Faster than Relational database on large sets ● No schema ○ No need of infra changes with new features ○ Also when reverting a feature… ● Have good enough ORM for RoR (Mongoid) ● Can create databases/collections without DevOps as bottle-neck
- 16. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Architecture V3 - Microservices Main incentive: Scale for productivity Ruby on Rails Monolith ● Microservices ● Use new tech outside the monolith ● Teams get more ownership and autonomy ● Schemaless and fast storage - Introduced MongoDB RabbitMQ Worker AMQP Consumer MongoDB Service HTTP API Worker AMQP Consumer MongoDB Service HTTP API READ/WRITE Gigs serviceOrders service Memcached READ/WRITE READ/WRITE
- 17. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. MongoDB replica-sets in Fiverr mongo-rs fiverr_prod RoR app1 app2app3 app4 app5 mongo-rs2 fiverr_prod app4 app5 app6 mongo-rs3 fiverr_prod app7 app8 app9 mongo-rs4 fiverr_prod app10 app11 app12
- 18. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. And even bigger...
- 19. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Architecture V4 - LOSA Data flow Worker AMQP&Kafka Consumer Dataview Service HTTP API Gig authoring service RoR (API Gateway) Worker Kafka Consumer Dataview Service HTTP API Controller READ WRITE Buyer dashboard Kafka PRODUCE/CONSUME View (Server side rendered) CONSUME ● Micro-frontends ● Kubernetes ● Backend, view oriented, microservices serving data over REST ● Read optimized private DBs ● Kafka as event broker
- 20. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. MongoDB replica-sets in fiverr mongo-rs RoR app1 app2app3 app4 app5 mongo-rs2 fiverr_prod app4 app5 app6 mongo-rs3 fiverr_prod app7 app8 app9 mongo-rs4 app10 app11 app12 mongo-rs5 app... app... app... mongo-rs6 app... app... app... mongo-rs7 app... app... app... fiverr_prod mongo-rs8 app... app... app...
- 21. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Problem: Cross-domain self-hosted databases ● Database sizing by the highest consumer and buffer ● Failure causes cross domain interruptions - Up to full service denial ● No ownership on the data and performance: DevOps as bottleneck ● Takes a lot of time and resources to maintain/backup/support/optimize the databases
- 22. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Solution: Domain Dedicated databases ● More accurate size: smaller databases ● Failures cause small extent of damage (mostly one domain) ● Co-ownership between Domain (Development Team) ownership and DevOps ● Higher Development team autonomy
- 23. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. MongoDB as a service Do you really want to manage so many clusters on your own??
- 24. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. MongoDB Atlas And so, our road to Atlas begins...
- 25. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. MongoDB Atlas First, we need to make some lists...
- 26. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Knowing Yourself Making checklists and preparing the existing environment ● Our source databases (versions, storage engines) ● Client specs (their types and versions) ● Application classification (migration readiness) ● Source infra specifications (VPC) ● New database scale and sizing ● Activating new features after the cutoff
- 27. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Knowing Yourself More preparations and coordinations ● How and when to migrate? ● Who should check the process? ● A new staging environment? ● Breaking changes that might pop-up?
- 28. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. MongoDB Atlas Once we were all set up, we took on the road
- 29. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. The Migration Process Automating the process ● Running the migration process itself ● Pitfalls and how to overcome them ● Failures along the road ● System impact during the migration
- 30. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. The Migration Process Switching over to Atlas ● Finishing the migration process ● Defining successful migration ● Post migration actions ● Monitoring application and cluster behavior
- 31. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. MongoDB Atlas Taking first steps, and it is a whole new world
- 32. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. New Environment and Features First technical impressions and perks ● The alerts and activity systems ● Real time and metrics monitoring ● The performance advisor tab ● Users and teams access control ● On-demand cluster configuration update
- 33. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. Knowing the Road The ongoing work and migrations ● Monitoring systems consolidation ● Building new and scaling existing clusters ● Working with the Atlas API ● Setting up proper alerts and signals ● Refreshing the staging environment
- 34. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. So what’s next? Things we are planning ● Fully automate cluster creation and scaling ● Continue data migrations to Atlas ● Align all clients with Atlas prerequisites ● Test new Atlas infra features (NVM)
- 35. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential. moshe.b@fiverr.com sergei.m@fiverr.com Thank You We’re hiring :)