Overview of Migration from Mongo DB to AWS DynamoDB

March 11, 2022 | Comments(1) |

In recent times, we have seen a rise in the volume of data used by the IT industry for analytics, processing, etc. The traditional way of storing the data is using a relational database and an on-premises server. However, this type of setup will handle limited requests because of limited infrastructure capabilities.

Data nowadays is becoming dynamic, unstructured, and rapidly changing. So, to fulfill this kind of data required, we can use a no-SQL database. We use programmable languages to query your data instead of using SQL queries for the CRUD (Create, read, update, delete) actions. Hence the No-SQL Database is in sudden demand due to its lightness and capabilities required to overcome the rapidly changing data and handle the vast amount of complex unstructured data.

Two extensive services helping us today are Mongo DB and AWS DynamoDB. Both are No-SQL databases with similar capabilities, and their advantages like MongoDB is platform-agnostic, but AWS DynamoDB is a managed AWS service. On the other side, MongoDB is limited to the infrastructure it is hosted on. In contrast, AWS DynamoDB is highly scalable, provides high throughput with low latency, and supports global tables.

Some of the key terms which are similar but differently named are shown below.

If you are seeking to improve your data handling capability you might be interested in migrating from MongoDB to AWS DynamoDB to enjoy a highly scalable managed infrastructure.

Today we will be using an AWS service known as AWS Data Migration Service to migrate the data.

AWS Database Migration Service is beneficial for migration from either on-premises, EC2 instance, Standalone servers, or AWS RDS to an AWS backed Database service with almost negligible downtime.

I have used a ubuntu 18.04 OS server to deploy my source on-premises MongoDB Database. I assume you have some data in your collection in the MongoDB database node, which is now ready to be sent to Amazon DynamoDB.

Let us get Started

AWS DMS offers two modes for migration

Document Mode: In this mode, all the data in the collection is transferred as a single column in the DynamoDB table
Table Mode: Here, DMS scan some specified documents in the table to create a schema which then helps the DMS to put them as fields in DynamoDB

We will be using the ‘table’ mode for the migration

Let us start by creating a role in your AWS account which will allow the DMS service to have access to DynamoDB

Go to IAM, then create a role
Choose DMS
Click next permissions and add DynamoDB full access policy to the role
Use tags if required and name the role as DMS-dynamo-role
Click Save

2. Let us go to DMS (Database migration service) in AWS Console

We will be required to create a replication server in a VPC that will perform the migration
Choose the name of your choice
Choose VPC where you want to launch
Choose the instance type: here, we have chosen t3.micro
Choose Dev workload (Single AZ) for Demo purpose for production workload you can opt for multi-AZ

3. Jump to your MongoDB database and provide the permissions

These permissions are required for the Replication server to access the DB

Use Admin to create the below role
Use your Database then to create a user with the role above

4. Now, we will be creating Source and Target endpoints

These endpoints will be required for the DMS task to transfer the data from source DB to DynamoDB

Create Source endpoint
Give Source endpoint name
Server Name as public Ip of MongoDB machine
Name the DB which has the required permissions we created in the earlier step
The endpoint will be created as below
Create Target endpoint
Give Target endpoint name
Choose Target engine as DynamoDB
The endpoint will be created as below

5. Test the endpoint connection for the Source DB connection. It will show as Successful if configured correctly

6. Create a Database migration task in the DMS console

Create Task
Give the task a name
Select the replication server we created
Choose Source and target endpoints
Choose table mode
Just as we complete creating task
Click on create and Database migration will take place. Time consumed for the task to complete will depend on the size of the data migrated

You can now go to DynamoDB in the AWS console and see your tables created

Conclusion

Migrating your No-SQL data to AWS DynamoDB is an up-gradation of infrastructure that will help provide single-digit millisecond performance. In addition, your data is now globally available for better throughput, which also supports encryption of your data at rest. For your CRUD operations, you can also follow up AWS documentation to merge your current application with AWS DynamoDB using APIs, SDKs, etc.

Big enterprises like Disney, Zoom, Dropbox are taking advantage of the seamless experience provided by AWS DynamoDB.

About CloudThat

We at CloudThat, are AWS (Amazon Web Services) Advanced Consulting Partner and Training partner and Microsoft gold partner, helping people to develop knowledge on cloud and help their businesses to aim for higher goals using best in industry cloud computing practices and knowledge.

Feel free to drop a comment or any queries that you have regarding AWS services, cloud and data migration, consulting and we will get back to you quickly. To get started, go through our Expert Advisory page and Managed Services Package that is CloudThat‘s offerings.