MongoDB World 2016: Do What Matters: Migrating to Self-Managed Infrastructure
0 likes437 views
Brian Scanlan is an engineering manager at Intercom. Intercom allows teams to communicate with customers at scale through websites, apps, and email. The document discusses Intercom migrating their MongoDB infrastructure to self-managed servers on AWS for cost savings of 62% while maintaining security, availability, and performance. It describes the process of migrating Intercom's app and tags to the new infrastructure through replica sets and custom configurations to minimize downtime.
MongoDB World 2016: Do What Matters: Migrating to Self-Managed Infrastructure
- 2. Brian Scanlan Engineering Manager at Intercom @brian_scanlan
- 3. What is Intercom? Intercom is one place for every team in an internet business to communicate with customers, personally, at scale—on your website, inside web and mobile apps, and by email.
- 5. Do What Matters: Migrating to Self- Managed Infrastructure
- 7. Things that actually matter 🔒Security🔒 Making 💰💰💰
- 12. 💩👖
- 13. Team Infrastructure core values 1. Security, Availability, Performance, Scalability, Cost, Efficiency - prioritize for maximum impact 2. Faster, Safer, Easier, Shipping 3. Zero Touch Ops 4. Run Less Software
- 14. “Let’s run more software”
- 19. Intercom App and Intercom tags required a custom migration: 1. Create “DMZ” hosts with identically named replica-sets 2. Ask nicely for the keyfile from third party, install on DMZ hosts 3. Using rs.add(), add DMZ hosts to MongoHQ replica-sets and sync data over 4. Restart DMZ replica set hosts with MMS replica set key, detach from MongoHQ replica set 5. Low level hackery to make replica-set as MMS expects: db.system.replset.update() to manipulate replica-set member IDs, and quickly removing/inserting admin users (Almost) downtime-free migration 👍
- 20. “We were paying $x a month, now we’re paying AWS/MongoDB $y a month for our MongoDB hosting, a cost saving of 62%”
- 22. Interesting discoveries at scale Ruby MongoDB driver defaults /etc/security/limits.d/999-intercom.conf: mongod - nofile 640000 mongod - nproc 640000 /etc/sysctl.conf kernel.pid_max = 655360 net.ipv4.tcp_max_syn_backlog = 204800
- 23. Wrote a document identifying problem Cost estimates Operations estimates Started playing around with MMS Wrote “Intermission” Benchmarked available host types Moved test accounts to AWS hosted MongoDB Tuned drivers, hosts Verified backups work well Verified rollback plan Pointed canary monitoring at new infrastructure Setup new alarming Broke a server to see what happened Moved real customers over More even more customers over Finished moving customers over Changed host type to i2.8xlarge Rotated credentials More host tuning …
- 24. 🙀 problems are worth solving when the impact matters
- 30. 🙀 problems are worth solving when the impact matters