Test

Challenge

The customer housed their applications in a standard data center that placed limits on their ability to scale on-demand, a problem that became increasingly relevant with the new COVID-19 pandemic and the expected enormous amount of traffic to the business’s website.

The physical design of the infrastructure also contributed to the incorrect simulation of the production environment by non-production environments. This left the operating teams struggling with the amount of effort needed for each release of their key business application coping with and fixing outages as they did while the developer’s pace was slowed due to a lack of confidence in implementing change.

Other limitations included restricted internal environment management tools and a content management system (CMS) platform that was unable to offer key capabilities.

Why AWS DMS ?

Converting relational data structures to documents can be complex. With AWS Database Migration Service (AWS DMS) you can perform minimal downtime migrations, and can replicate ongoing changes to keep sources and targets in sync.

Why using multiple tasks ?

If you want to improve the performance when migrating a large table, you can break the migration into more than one task. For example, if you have an integer primary key ID from 1 to 8,000,000, you can create eight tasks using row filtering to migrate 1 million records each.

Why using Limited LOB mode ?

When using Limited LOB mode, you specify the maximum size of LOB column data. If the size of the LOB columns exceeds the size that you specified in the task, then AWS DMS truncates the data. AWS DMS then sends warnings to the AWS DMS log file. Using Limited LOB mode improves performance, it’s a best practice to be sure that you have enough memory allocated to the replication instance to handle the task.

Why using Amazon CloudWatch ?

In some cases, AWS DMS can encounter issues for which warnings or error messages appear only in the task log. In particular, data truncation issues or row rejections due to foreign key violations are only written in the task log. Therefore, in order to review the task log when migrating a database, we configure Amazon Cloudwatch.

The CloudLink Solution

Architecture

In this architecture, we suppose that you have your SQL Server Database on Amazon EC2 instance, and you want to migrate it to your Amazon DocumentDB cluster. In order to ensure the highest performance possible, we included 3 best practices as follows :

The migration Task is divided into multiple tasks. Each task migrate a predefined limited LOB size from the source database to your Amazon DocumentDB cluster. During the migration, the warning and error logs are saved on Amazon CloudWatch.

Prerequisites
  • An active AWS account
What you will do

Sign in to your AWS account, now you are on the AWS Management Console.

These are the different steps to do :

  1. Create and Configure a VPC

  2. Create SQL Server Database on EC2 instance

  3. Create an Amazon DocumentDB cluster

  4. Create and configure the AWS DMS replication instance

  5. Create the Multiple tasks

    1. Use Limited LOB mode

    2. Enable Logging using Amazon CloudWatch

  6. Start the migration

 
  • Note : The steps from 1 to 4 are described in detail in this Wiki Page
5- Create the multiple tasks

In the DMS console, go to Database Migration Tasks, you will create for this example 2 tasks to migrate your SQL Server Database to your Amazon DocumentDB cluster. In this example we have a SQL Server Database

with only one Table called “Categories”. This table contains 8 rows, so the first task will migrate the first 4 rows while the second one will migrate the last 4 rows to our target database.

To create the first task :

  • Click on Create task
  • You specify a name for the task (migration-bp), the DMS instance name (i.e dms-cdc), as well as the source and the target endpoint that you created. For the Migration type you choose “Migrating existing data’
a- Use the Limited LOB mode

In the Task Settings, you configure your task to use the Limited LOB mode, by choosing Limited LOB mode in Include LOB columns in replication section.

In this case we choose Maximum LOB size of 32 KB.

b- Enable Logging using Amazon CloudWatch

To enable Cloudwatch log :

  • Check Enable CloudWatch logs

In this example we use row filtering to create multiple tasks.

To do so we specify the selection rules in the Table mappings :

This task will contain the rows from the table “Categories” with CategoryID from 1 to 4.

  • You do the same configuration for the second task (migration-bp-02) except for Target table preparation mode :
  • And in the Selection rules, you specify rows with CategoryID is between 5 and 8.
  • Finally click on Create task

In Database Migration tasks, you will find taht both of your tasks are complete and that your migration is done successfully.

Here are the CloudWatch metrics :

As well as the Log Events saved during the migration by CloudWatch.

You can verify the results on your target database using show collections command and db.Categories.count() to count the number of rows of the Categories table migrated from the source database as follows :

Taking advantage of previous experience in the financial services sector, CloudLink, in close collaboration with the Operations team of the customer, first introduced a lightweight deployment framework that allowed the customer to deliver managed, structured, and repeatable environments and applications. A migration of the customer’s production environments from their traditional hosting platform to Amazon Web Services ( AWS) was then planned and carried out by the both teams;

The data migration from an on-site database to Amazon RDS was a key component of the customer’s transition process. This migration involved a 12-hour outage and included an upgrade from SQL Server 2008 Enterprise to SQL Server 2012 Standard Version.

A blue / green deployment approach for the entire migration with the environment and application teams adopting release control services, such as DNS, was adopted to allow the company to thoroughly test the new Production AWS environment prior to migration. This ensured that none of the current infrastructure architecture was affected, thereby facilitating a simple roll-back process should the migration fail by simply removing the ‘Sorry’ page and moving the database out of read mode.

The Benefits

Some components of the customer were hosted on AWS during the early days of the pandemic to offload page view requests on their public site. This allowed the increased traffic from the pandemic to be handled by the customer, but not with a high degree of confidence. Out of the top 10 “money converter” web results, nationally was one of the few currency pages that were available throughout the pandemic.

In addition to a few legacy systems which were due to be decommissioned / replaced, the majority of the customer components were on AWS. By taking advantage of the scaling capabilities of AWS, amid a significant rise in site traffic, the operations team was clearly watching the graphs during the events of the customer.

AWS environment post-migration testing revealed that the customer is now capable of serving 120 times the load faced at the early days of the pandemic, a load that nearly took down their conventional hosting provider.

CloudLink takes pride in procuring the “Next Gen” cloud experience and IT consultation to its customers. We are a team of highly skilled professionals that will guide and assist you in efficiently planning, designing, building, mitigating, running, and optimizing your cloud environments.

Our dedicated experts offer a range of services to our valuable customers varying from automation, proactive monitoring, management, and maintenance of our customer’s cloud environment. Our expert team has accomplished third-party audits and is highly skilled in dealing with application migration and cloud infrastructure.

Drive innovation and boost up the performance of your company by simply signing up to our full spectrum of digital, analytics, and enterprise cloud services. The initial procedures will comprise a detailed evaluation of the organizational goals and growth strategy of your company, followed by the design phase. With our technical expertise, we will design, implement, and manage your customized cloud infrastructure to help you attain your business objectives swiftly and systematically.

CloudLink is your right cloud partner if you are looking to start your cloud migration as it guarantees to augment your organization’s digital transformation, and performance to make your company stand out amongst the competitors.

A cloud in mind ?
Contact us !

How can we help ?