Migrating to MongoDB Atlas

We recently completed a major migration journey of our database infrastructure at Vyrill. The goal was to move from a standalone self-hosted database to a managed service called MongoDB Atlas.

The motivation behind this move was to allow the engineering team focus on the application innovation rather than spending cycles configuring, patching, and monitoring of the database infrastructure.

What is MongoDB Atlas?

MongoDB Atlas is a fully-managed cloud database that handles all the complexity of deploying, managing, and healing your deployments on the cloud service provider of your choice (in our case AWS).

We were already using a standalone instance of MongoDB on EC2. I talk about the database version upgrade - 

https://www.linkedin.com/feed/update/urn:li:activity:7120101981074952192/

There are two options to migrate to Atlas.

1. Atlas Live migration

2. Backup and restore

We chose Atlas Live migration as it does not require downtime. The other option allows you to select specific databases to migrate but it takes a backup and restore approach with some downtime.

Following is our detailed step-by-step process for migrating to Atlas:

Step 1. Create Atlas cluster

Step 2. Make sure your MongoDB version is correct

We realized that our standalone MongoDB version was 6.0.11, but Atlas needed it to be 6.0.12.

You must run the following commands on your existing EC2 hosting the standalone MongoDB cluster.

1. Update the MongoDB Repository

2. Upgrade MongoDB 

3. Restart MongoDB Service 

4. Verify the upgrade

Step 3.  Create users on Atlas

To perform migration, we need a admin user. You need to create a admin user with certain privileges as shown below

Execute the command to verify the user privileges

Step 4. Convert standalone to replica set (if needed)

With the Atlas live migration approach, only replica sets can be migrated, not standalone instances.

We had a standalone version, so the next task was to convert this to a replica set.

The conversion is accomplished using these steps.

Run these commands on your EC2 instance hosting the standalone MongoDB cluster.

a. Connect to to your mongod instance 

b. Shutdown the standalone instance

c. Name the replica set 

d. In the mongod.conf file add the following 

Usually this file is located at /etc/mongod.conf.

As a sidenote, the way to start mongo server using this conf file is the following command

e. Initialize and verify the replica set

The mongod.conf should be updated to include replication.replSetName as mentioned here

https://www.mongodb.com/docs/manual/reference/configuration-options/#replication-options

mongod process should be executed with the –replSet and security.keyFile

Step 5. Start migration

i. In the newly created Atlas cluster, select this option

ii. Next screen presents a list of IPs to be whitelisted at the source.

In our case we added these to the security group associated with the EC2 instance hosting source MongoDB cluster. Click on Start Migration.

iii. Once the migration is complete, Atlas will show this screen. You can click on Prepare to cutover once you determine the live upgrades are no longer needed.

iv. Once you complete the cutover procedure and confirm the applications are working as desired, click on “Cut Over”.

That’s it!

Here are a few recommendations based on our experience: 💡

- Confirm version compatibility  🤝

- Plan replica set conversion ♻️

- Monitor migration before final cutover 🔍

The ongoing tasks include monitoring which queries are frequently made and the ones that are also taking the most time. This will help identify opportunities for creating indexes and query optimization.

Our database infrastructure is now well positioned to scale for millions of users with Atlas.