{"id":60376,"date":"2024-02-22T10:22:47","date_gmt":"2024-02-22T04:52:47","guid":{"rendered":"https:\/\/www.tothenew.com\/blog\/?p=60376"},"modified":"2024-03-01T10:27:44","modified_gmt":"2024-03-01T04:57:44","slug":"data-center-to-aws-cloud-migration","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/data-center-to-aws-cloud-migration\/","title":{"rendered":"Data Center to AWS Cloud Migration"},"content":{"rendered":"<h2><b>Introduction<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">For many organizations, the journey to the cloud begins with an important decision: migrating from their traditional on-premises data centers to Amazon Web Services (AWS). This transition demands courage, dedication, and extensive planning, as it involves moving their already operational infrastructure to the cloud without downtime.<\/span><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-60366 size-full\" src=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0.png\" alt=\"\" width=\"626\" height=\"352\" srcset=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0.png 626w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-300x169.png 300w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-624x351.png 624w\" sizes=\"(max-width: 626px) 100vw, 626px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">One such great journey is the story of one <\/span><b>Global Advertising Management Platform<\/b><span style=\"font-weight: 400;\"> client, a powerhouse in the world of <\/span><b>advertising<\/b><span style=\"font-weight: 400;\"> and <\/span><b>connected TVs<\/b><span style=\"font-weight: 400;\">. With a state-of-the-art Connected TV Advertising Management Platform, they needed a trusted partner to migrate and manage their critical AWS applications. And that&#8217;s where our journey with them began.<br \/>\n<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The key highlights of the proposed solution were:<\/span><\/p>\n<p><b>1. Re-platform and Re-architect<\/b><span style=\"font-weight: 400;\">:<\/span><span style=\"font-weight: 400;\"> To ensure compatibility with AWS, our team re-platformed and re-architected the solution, paving the way for a seamless transition.<\/span><\/p>\n<p><b>2. Petabyte Data Migration<\/b><span style=\"font-weight: 400;\">:<\/span><span style=\"font-weight: 400;\"> Moving petabytes of data from on-premises infrastructure to AWS requires a skilled hand.<\/span><\/p>\n<p><b>3. Active-Standby Setup<\/b><span style=\"font-weight: 400;\">:<\/span><span style=\"font-weight: 400;\"> We designed an active-standby setup for uninterrupted service, ready to take the reins as the primary system after the cutover.<\/span><\/p>\n<p><b>4. Enhanced Monitoring and CI\/CD Setup<\/b><span style=\"font-weight: 400;\">:<\/span><span style=\"font-weight: 400;\"> We modified their infrastructure with enhanced monitoring and a robust Continuous Integration and Continuous Deployment (CI\/CD) pipeline, ensuring performance and stability.<\/span><\/p>\n<p><b>5. Post-Go-Live Support:<\/b><span style=\"font-weight: 400;\"> Our commitment extended beyond the migration. We provided steadfast post-Go-Live support to ensure their operations ran smoothly on AWS.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Whether you&#8217;re an expert IT professional or just beginning to explore the world of cloud computing, this blog is designed to be your compass as you embark on the exciting journey from on-premises data centers to the boundless skies of AWS. <\/span><span style=\"font-weight: 400;\">Welcome aboard, and let&#8217;s explore the world of AWS migration together.<\/span><\/p>\n<h2><b>Why migrate from DC to AWS?<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">Migrating to a cloud platform like Amazon Web Services (AWS) from a traditional data center can offer numerous benefits. Still, it&#8217;s essential to understand the limitations of data centers and the reasons why we choose to migrate to the cloud. Our client was dealing with the problems mentioned below, and in general, here are some limitations of traditional data centers that drive the need for migration to AWS:-<\/span><\/p>\n<p><b>1. Scalability<\/b><span style=\"font-weight: 400;\">: Challenges in data centers due to the need for <\/span>upfront investments<span style=\"font-weight: 400;\"> and limited physical space for expansion. AWS provides unmatched scalability. You can easily adjust your resources up or down to handle changing workloads.<\/span><\/p>\n<p><b>2. Cross Availability Zones (AZs) for High Availability<\/b><span style=\"font-weight: 400;\">: AWS offers multiple AZs within regions. This means we can run applications with high availability and negligible downtime, even if one AZ faces issues. This becomes difficult in data centers because achieving geographic redundancy often involves high costs &amp; complex setups.<\/span><\/p>\n<p><b>3. OS and Software Upgrades<\/b><span style=\"font-weight: 400;\">: AWS handles the underlying infrastructure, including OS and software updates. This reduces the operational burden on your IT team, allowing them to focus on higher-value tasks. This is complex in data centers as it requires <\/span>manual updates, leading to downtime and operational overhead.<\/p>\n<p><span style=\"font-weight: 400;\"><strong>\u00a04.<\/strong> <\/span><b>Monolithic to Microservices-based Architecture<\/b><span style=\"font-weight: 400;\">: Breaking applications into smaller, modular components enables greater flexibility, faster development, agility, and easier maintenance. It is pretty tough in data centers due to hardware constraints and legacy architecture limitations.<\/span><\/p>\n<p><span style=\"font-weight: 400;\"><strong>5.<\/strong> <\/span><b>Enhanced Observability<\/b><span style=\"font-weight: 400;\">: Real-time monitoring is crucial for quickly finding and addressing issues, which is challenging in a data center. On the other hand, AWS\u00a0 has a wide variety of observability tools, like Amazon <\/span>CloudWatch<span style=\"font-weight: 400;\"> and <\/span>AWS X-Ray<span style=\"font-weight: 400;\">, that provide deep insights into the performance of your applications and infrastructure.\u00a0<\/span><\/p>\n<p><b>6. Security:<\/b><span style=\"font-weight: 400;\"> AWS prioritizes security. It offers robust security features and compliance certifications, making it easier for organizations to meet regulatory requirements and safeguard their data.<\/span><\/p>\n<p><strong>7. Reduced Maintenance<\/strong>: Maintaining and operating a data center requires a skilled and dedicated Infrastructure team, which may be difficult to recruit and retain.\u00a0 AWS handles much of the underlying infrastructure&#8217;s maintenance and updates, freeing IT staff to focus on higher-value tasks.<\/p>\n<p><b>8. Backups &amp; Disaster Recovery<\/b><span style=\"font-weight: 400;\">: AWS provides comprehensive backup and disaster recovery solutions. Services like <\/span>Amazon S3, Amazon Data Lifecycle Manager, and AWS Backup<span style=\"font-weight: 400;\"> ensure that data is safe and recoverable during a disaster. Often complex in data centers, with manual backup processes and limited storage.<\/span><\/p>\n<h2><b>What were we dealing with?<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">Before starting the actual migration, we need to understand the existing infrastructure &amp; application stack. So that things can be planned accordingly. This also helps us identify the right services we must choose in AWS.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Highlights of the Existing Data Center Setup:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Advertisement Application Volume (Peak Time): 35 billion requests per day<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">ETL Data Size:<\/span>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">2 Petabytes of historical data<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">A 40-node Hadoop cluster<\/span><\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Web\/CDN<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Haproxy<\/span>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Nginx &amp; Apache2<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Fastly<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">NSONE (DNS Management)<\/span><\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Database\/Storage<\/span>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">MySQL (Percona XtraDB)<\/span><\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Approximately 30 mission-critical applications written in different languages such as Java, Python, NodeJs, etc.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Aerospike Cluster<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Druid Cluster<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Messaging\/Queue: Kafka<\/span><\/li>\n<\/ul>\n<ul>\n<li style=\"list-style-type: none;\"><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">This big infrastructure reflects the challenges and complexities faced by organizations with substantial workloads.<\/span><\/p>\n<h2><b>Navigating the AWS Migration Journey: From Discovery to Setup<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">In this step, we started by collecting information about servers, applications, and data along with their inter-dependencies. Choose the right discovery tool, like AWS Migration Hub, to get more insights into the applications &amp; servers running in DC. A discovery tool helps you gather information about the inter-dependence of the workloads by collecting server utilization data like configuration, usage, and behavior on your on-premise data center. Plan your data and application migration based on these dependencies.<\/span><\/p>\n<p><b>Step A: Discovery:<\/b><span style=\"font-weight: 400;\"> The journey begins with a thorough discovery phase, where you collect essential information about your existing infrastructure. Key activities include:<\/span><\/p>\n<ol>\n<li style=\"font-weight: 400;\"><b>AWS Migration Hub<\/b><span style=\"font-weight: 400;\">: We utilized AWS Migration Hub for inventory scanning, ensuring we have a complete view of on-premises resources.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Ansible Scripts<\/b><span style=\"font-weight: 400;\">: Executed Ansible scripts to collect information about operating systems, running services, and open ports.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Application Portfolio Analysis<\/b><span style=\"font-weight: 400;\">: Identification of application portfolio and their dependencies. Understanding the communication between different applications was necessary for this transition.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Knowledge Transfer Calls<\/b><span style=\"font-weight: 400;\">: Engage in detailed knowledge transfer calls with the customer to ensure alignment and clarity about the migration plan.<\/span><\/li>\n<\/ol>\n<p><b>Step B: Migration Planning<\/b><\/p>\n<ol>\n<li style=\"font-weight: 400;\"><b>Migration Document:<\/b><span style=\"font-weight: 400;\"> Begin working on a comprehensive migration document that outlines your strategy and objectives.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Customer Signoff:<\/b><span style=\"font-weight: 400;\"> Collaborate closely with the customer and obtain their signoff on the migration plan.<\/span><\/li>\n<\/ol>\n<p><b>Step C: Resource Mapping and Estimation:<\/b><span style=\"font-weight: 400;\"> As we advanced in the migration plan, it was crucial to map existing resources to their AWS counterparts:<\/span><\/p>\n<ol>\n<li style=\"font-weight: 400;\"><b>Resource Mapping<\/b><span style=\"font-weight: 400;\">: Finalize the mapping of your existing infrastructure to AWS components, ensuring a smooth transition.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Storage Requirements<\/b><span style=\"font-weight: 400;\">: Calculate your storage needs, considering data volumes and growth projections.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>AWS Calculator<\/b><span style=\"font-weight: 400;\">: Utilize AWS&#8217;s calculator tools to estimate the costs associated with your AWS infrastructure.<\/span><\/li>\n<\/ol>\n<p><b>Step D: Creating Infrastructure on AWS<\/b><span style=\"font-weight: 400;\">:\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">1. In this phase, we utilized Terraform as our Infrastructure as Code (IaC) tool to launch the infrastructure on Amazon Web Services (AWS). Our objective was to replicate the existing Data Center (DC) environment on AWS, encompassing all applications, data components,\u00a0 Terraform server, Jenkins server, and various monitoring tools.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">2. The deployment process was synchronized with our DC releases. For each release on the DC, a corresponding release was executed on AWS. This parallel setup allowed us to maintain consistency and ensure a seamless transition between environments.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">3. As part of the migration strategy, we transferred data from different data components in the DC to their corresponding counterparts in AWS. This ensured that our applications could seamlessly consume data from AWS sources. Thorough testing was conducted to validate the integrity and performance of the migrated data.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">4. The pivotal moment in this migration was the DNS cutover, which marked the traffic transition from the DC to AWS. This cutover was executed gradually, employing the weighted routing feature provided by NS1, our domain hosting partner. Initially, we directed only 1% of the traffic to AWS. This cautious approach allowed us to monitor for any issues and provided the flexibility to roll back to the DC instantly if any anomalies were detected.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">5. Upon successful validation and confidence in the AWS environment, we incrementally increased the traffic allocation until reaching 100%. At this point, having confirmed the stability and reliability of our AWS infrastructure, we proceeded to shut down the data center entirely.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">6. This comprehensive migration strategy, encompassing infrastructure replication, data transfer, synchronized releases, and a gradual DNS cutover, ensured a smooth transition to AWS without disruptions to our services. The use of Terraform for infrastructure provisioning and NS1&#8217;s weighted routing for controlled traffic redirection played key roles in the success of this migration.<\/span><\/p>\n<h2><b>DC TO AWS Architecture Diagram\u00a0<\/b><\/h2>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-60367 size-large\" src=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/AWS_DC-1024x384.png\" alt=\"\" width=\"625\" height=\"234\" srcset=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/AWS_DC-1024x384.png 1024w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/AWS_DC-300x112.png 300w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/AWS_DC-768x288.png 768w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/AWS_DC-1536x575.png 1536w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/AWS_DC-624x234.png 624w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/AWS_DC.png 2048w\" sizes=\"(max-width: 625px) 100vw, 625px\" \/><\/p>\n<h2><b>Direct Connect<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">In the journey from data center to AWS migration, one important aspect is to maintain reliable connectivity between DC and the AWS cloud. AWS Direct Connect is the bridge that makes this seamless connection possible. Without this keeping sync of data between both DC &amp; AWS was next to impossible.<\/span><\/p>\n<h2><b>What is AWS Direct Connect?<\/b><b><br \/>\n<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">AWS Direct Connect is a network service that provides dedicated and private network connections from your on-premises DC to AWS. It bypasses the public internet, ensuring a secure, low-latency, and high-bandwidth connection. We needed Direct Connect for the secure transfer of data from DC to AWS. Also, there was a requirement to do a cutover without any downtime. We had to keep real-time sync between the DC &amp; AWS environments. A few Considerations before setting up Direct Connect:-<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><b>Select a Direct Connect Location<\/b><span style=\"font-weight: 400;\">: Choose a Direct Connect location based on your geographical location and AWS region to establish the connection.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Choose a Direct Connect Partner<\/b><span style=\"font-weight: 400;\">: AWS Direct Connect Delivery Partners help you establish network connectivity between AWS Direct Connect locations and their data center, office, or colocation environment via the following models: Dedicated Connections, Hosted Connections, and Hosted Virtual Interfaces. These partners offer a range of services, including port connections and equipment setup.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Determine Connection Capacity<\/b><span style=\"font-weight: 400;\">: Decide on the connection capacity, which can range from <\/span><b>1 Gbps<\/b><span style=\"font-weight: 400;\"> to <\/span><b>100 Gbps<\/b><span style=\"font-weight: 400;\">, depending on your requirements.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Testing and Validation<\/b><span style=\"font-weight: 400;\">: It&#8217;s important to test the connection&#8217;s performance, security, and reliability to ensure that it meets your requirements<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Setup:- <\/span><a href=\"https:\/\/docs.aws.amazon.com\/directconnect\/latest\/UserGuide\/getting_started.html\"><span style=\"font-weight: 400;\">https:\/\/docs.aws.amazon.com\/directconnect\/latest\/UserGuide\/getting_started.html<\/span><\/a><\/p>\n<h2><b>Major Activities<\/b><\/h2>\n<h3><b>Performance Testing<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">One critical phase that can&#8217;t be overlooked is performance testing. It&#8217;s the assurance that your web applications can handle the expected traffic with reliability and scalability. AWS&#8217;s autoscaling options are impressive, but performing performance testing based on real production traffic is necessary.<\/span><\/p>\n<p><b>Selecting the Right Performance Testing Tool<\/b><span style=\"font-weight: 400;\">: There are various performance testing tools available, each with its strengths. Tools like Performance Runner, Locust, Webperformance, and Apache JMeter offer versatile solutions. In our case, we opted for Apache JMeter, a powerful and open-source tool known for its flexibility and scalability.<\/span><\/p>\n<h3><b>Why Apache JMeter<\/b><span style=\"font-weight: 400;\">?<\/span><\/h3>\n<p><span style=\"font-weight: 400;\">JMeter is a great choice for performance testing due to its flexibility and adaptability. It can simulate heavy performances, analyze performance, and pinpoint issues with precision. Plus, it&#8217;s open-source, making it cost-effective and accessible.<\/span><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-60368 size-full\" src=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-1.png\" alt=\"\" width=\"600\" height=\"300\" srcset=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-1.png 600w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-1-300x150.png 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/p>\n<p><b>Preparing for Performance Testing<\/b><span style=\"font-weight: 400;\">: We took a data-driven approach to prepare for performance testing. Using data center numbers, we calculated the expected production traffic, which, in our case, is approximately 2.5 million requests per minute. However, we didn&#8217;t stop there. To ensure our infrastructure&#8217;s robustness, we subjected it to performance testing at 3.5 million requests per minute, allowing for a comfortable margin.<\/span><\/p>\n<p><b>Our performance Testing Setup<\/b><span style=\"font-weight: 400;\">: We set up a JMeter cluster that consisted of one master and ten slave nodes, each equipped with <\/span>c5.18xlarge capacity. This configuration allowed us to generate a staggering performance of 3.5 million requests per minute<span style=\"font-weight: 400;\">. Adjusting the throughput value within the JMeter test scripts was critical, as it directly influenced the number of requests sent.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Here&#8217;s a code snippet illustrating how to adjust the throughput value in JMeter test scripts to match the required performance :<\/span><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-60369 size-large\" src=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-2-1024x166.png\" alt=\"\" width=\"625\" height=\"101\" srcset=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-2-1024x166.png 1024w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-2-300x49.png 300w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-2-768x124.png 768w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-2-1536x249.png 1536w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-2-624x101.png 624w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-2.png 1538w\" sizes=\"(max-width: 625px) 100vw, 625px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">Our performance testing with Apache JMeter proved invaluable. It allowed us to simulate real-world traffic conditions, uncover bottlenecks, and fine-tune our AWS infrastructure for peak performance.<\/span><\/p>\n<h2><b>Application Migration &amp; DNS Cutover<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">Application migration and DNS cutover are important steps in the process of moving an application or service from one environment to another or migrating from an on-premises data center to the cloud. The DNS cutover is a critical step in the application migration process. It involves updating Domain Name System (DNS) records to direct traffic to the new environment hosted by the migrated application. This ensures that users and clients are properly routed to the new location. Doing that with zero downtime is challenging and involves lots of preparation before the cutover. Let&#8217;s dive deeper into each of these concepts:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><b>Data Migration<\/b><b>:<\/b><span style=\"font-weight: 400;\"> Move the application&#8217;s data to the new environment while ensuring data integrity, consistency, and minimal downtime. Various data migration tools and techniques can be employed.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Application Replication or Reconfiguration<\/b><b>:<\/b><span style=\"font-weight: 400;\"> Set up the application in the new environment, either by replicating its configurations or by reconfiguring it to adapt to the new infrastructure. This may involve adjusting settings, database connections, and other configurations.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Testing<\/b><b>:<\/b><span style=\"font-weight: 400;\"> Rigorously test the migrated application in the new environment to ensure it functions correctly. This includes functional testing, performance testing, and security testing.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Validation<\/b><b>:<\/b><span style=\"font-weight: 400;\"> Verify that all data, configurations, and dependencies have been successfully migrated and that the application is performing optimally in the new environment.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Cutover Planning<\/b><b>:<\/b><span style=\"font-weight: 400;\"> Plan for the DNS cutover, which is a crucial step to ensure that users are directed to the application in the new environment.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Monitoring and Optimization:<\/b><span style=\"font-weight: 400;\"> After the cutover, continuously monitor the application&#8217;s performance in the new environment and make any necessary optimizations to ensure it operates efficiently<\/span><span style=\"font-weight: 400;\">.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">With the above steps, you now now have two parallel setups running at the same time. One in the Data Center and one in AWS. But still, the traffic is being served from the data center. Here is when DNS cutover comes into the picture to so a smooth transition of the traffic to AWS.<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><b>TTL (Time to Live) Settings<\/b><b>:<\/b><span style=\"font-weight: 400;\"> Before performing the DNS cutover, adjust the TTL settings for your DNS records. Reducing the TTL allows for a shorter propagation time when DNS records are updated.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Testing<\/b><b>:<\/b> Thoroughly test the new environment to ensure that the migrated application functions correctly. Confirm that all configurations and data have been successfully transferred.<\/li>\n<\/ul>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-60370 size-full\" src=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/Untitled-2023-12-11-1907.png\" alt=\"\" width=\"787\" height=\"717\" srcset=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/Untitled-2023-12-11-1907.png 787w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/Untitled-2023-12-11-1907-300x273.png 300w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/Untitled-2023-12-11-1907-768x700.png 768w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/Untitled-2023-12-11-1907-624x568.png 624w\" sizes=\"(max-width: 787px) 100vw, 787px\" \/><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><b>DNS Record Updates<\/b><b>:<\/b><span style=\"font-weight: 400;\"> Update the DNS records, such as A records, CNAME records, or other relevant records, to point to the IP addresses or domain names of the new environment.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Propagation Time<\/b><b>:<\/b><span style=\"font-weight: 400;\"> Understand that DNS changes may take some time to propagate across the internet. The propagation duration depends on the TTL settings and can vary from minutes to hours.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Rollback Plan<\/b><b>:<\/b><span style=\"font-weight: 400;\"> Always have a rollback plan if issues arise during the cutover. This plan should enable you to switch back to the old environment while addressing any problems that may arise.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Monitoring<\/b><b>:<\/b><span style=\"font-weight: 400;\"> Continuously monitor DNS propagation and the application&#8217;s performance after the cutover to ensure everything functions as expected.<\/span><\/li>\n<\/ul>\n<h2><b>Kafka Migration<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">For most of the data, we used the<\/span> <b>LIFT &amp; SHIFT<\/b><span style=\"font-weight: 400;\">\u00a0 (<\/span><b>Rehos<\/b><span style=\"font-weight: 400;\">t<\/span><span style=\"font-weight: 400;\">) mechanism. However, Kafka was one of the things for which we had to keep active-active sync. In this portion of the blog, We would like to discuss that.<\/span><\/p>\n<h3><b>What is MirrorMaker?<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">MirrorMaker is a tool in Apache Kafka for replicating data between Kafka clusters. It&#8217;s useful for maintaining a replica of an existing Kafka cluster in another data center.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/p>\n<h3><b>How do we use MirrorMaker for Migration?<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">For a seamless migration, we used MirrorMaker to copy data from the on-premises Kafka to AWS Kafka. Here are the steps we followed:<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">Create a list of topics in the source cluster to copy.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Replicate those topics on the target cluster.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Set up a MirrorMaker cluster based on data load.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Optimize mirroring with different configuration parameters.<\/span><\/li>\n<\/ul>\n<h3><strong>During Kafka Migration<\/strong><\/h3>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-60371 size-full\" src=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-3.png\" alt=\"\" width=\"907\" height=\"427\" srcset=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-3.png 907w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-3-300x141.png 300w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-3-768x362.png 768w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-3-624x294.png 624w\" sizes=\"(max-width: 907px) 100vw, 907px\" \/><\/p>\n<h3><strong>After Migration<\/strong><\/h3>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-60372 size-full\" src=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-4.png\" alt=\"\" width=\"913\" height=\"506\" srcset=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-4.png 913w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-4-300x166.png 300w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-4-768x426.png 768w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/pasted-image-0-4-624x346.png 624w\" sizes=\"(max-width: 913px) 100vw, 913px\" \/><\/p>\n<h3><b>Monitoring Kafka migration<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">Use the consumer offset checker tool to ensure your mirror keeps up with the source cluster.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><span style=\"font-weight: 400;\">For more information on this, check this blog \u2192 <\/span><a href=\"https:\/\/www.tothenew.com\/blog\/mirror-maker-for-kafka-migration\/\"><span style=\"font-weight: 400;\">https:\/\/www.tothenew.com\/blog\/mirror-maker-for-kafka-migration\/<\/span><\/a><\/p>\n<h2><b>Benefits after Moving To AWS<\/b><\/h2>\n<ul>\n<li><b>Enhanced Visibility<\/b><span style=\"font-weight: 400;\">:- \u00a0 Better understanding of traffic patterns and peak load<\/span><\/li>\n<\/ul>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-60373 size-large\" src=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/Screenshot-from-2023-10-22-16-44-35-1024x340.png\" alt=\"\" width=\"625\" height=\"208\" srcset=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/Screenshot-from-2023-10-22-16-44-35-1024x340.png 1024w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/Screenshot-from-2023-10-22-16-44-35-300x100.png 300w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/Screenshot-from-2023-10-22-16-44-35-768x255.png 768w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/Screenshot-from-2023-10-22-16-44-35-624x207.png 624w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/Screenshot-from-2023-10-22-16-44-35.png 1202w\" sizes=\"(max-width: 625px) 100vw, 625px\" \/><\/p>\n<ul>\n<li><b>Scalability and Elasticity:<\/b><span style=\"font-weight: 400;\">&#8211;\u00a0 One of the most notable advantages of AWS is its ability to scale resources up or down based on demand. Your organization can easily adjust its computing capacity to handle traffic spikes, reducing the risk of over-provisioning and savings on infrastructure costs.<\/span><\/li>\n<\/ul>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-60374 size-large\" src=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/Screenshot-from-2023-10-22-16-52-05-1024x377.png\" alt=\"\" width=\"625\" height=\"230\" srcset=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/Screenshot-from-2023-10-22-16-52-05-1024x377.png 1024w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/Screenshot-from-2023-10-22-16-52-05-300x110.png 300w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/Screenshot-from-2023-10-22-16-52-05-768x282.png 768w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/Screenshot-from-2023-10-22-16-52-05-624x229.png 624w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/Screenshot-from-2023-10-22-16-52-05.png 1202w\" sizes=\"(max-width: 625px) 100vw, 625px\" \/><\/p>\n<ul>\n<li><b>Visibility For Cost Management<\/b><b>: <\/b>Visibility is not limited to applications and infrastructure but also extends to spending<span style=\"font-weight: 400;\">. AWS provides comprehensive cost management tools that allow us to monitor expenses granularly. We now have insights into per-environment costs and even per-resource costs.<\/span><\/li>\n<\/ul>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-60375 size-full\" src=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/Untitled.png\" alt=\"\" width=\"651\" height=\"463\" srcset=\"\/blog\/wp-ttn-blog\/uploads\/2024\/02\/Untitled.png 651w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/Untitled-300x213.png 300w, \/blog\/wp-ttn-blog\/uploads\/2024\/02\/Untitled-624x444.png 624w\" sizes=\"(max-width: 651px) 100vw, 651px\" \/><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><b>Efficient Secrets Management<\/b><span style=\"font-weight: 400;\">:- Secrets management is a critical aspect of security and compliance. SSM simplifies this task by centralizing secret storage and access. We can use <\/span>AWS Secrets Manager<span style=\"font-weight: 400;\"> to inject secrets and sensitive environment variables into applications.<\/span><span style=\"font-weight: 400;\">\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Scalability Of Components:<\/b><span style=\"font-weight: 400;\">&#8211;\u00a0 AWS&#8217;s scalability is one of its standout features. Regarding data components, such as EMR Cluster &amp; Autoscaling, the ability to scale is a game-changer. Adding nodes or increasing capacity can be a task that takes as little as a few minutes. This quick scalability ensures that your data infrastructure can effortlessly adapt to changing workloads and handle increased demands without interruptions.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Effective Monitoring with CloudWatch<\/b><span style=\"font-weight: 400;\">:- Monitoring is essential for maintaining the health and performance of applications and infrastructure. AWS Cloudwatch is one go-to solution for this.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/li>\n<\/ul>\n<h2><b>Learnings<\/b><\/h2>\n<ul>\n<li style=\"font-weight: 400;\"><b>Fail Fast When You Don&#8217;t Have Visibility<\/b><b>:-<\/b><span style=\"font-weight: 400;\"> One of the most crucial lessons learned is <\/span>the importance of identifying and addressing issues. When migrating to AWS, it&#8217;s essential to have visibility into your environment. This means real-time insights into resource utilization, application performance, and security. It&#8217;s difficult to detect issues early and take the right actions without visibility.<\/li>\n<li style=\"font-weight: 400;\"><b>Checklists Work<\/b><b>:-<\/b><span style=\"font-weight: 400;\"> Migrating to AWS is a comprehensive process involving numerous tasks, spanning from resource provisioning to security configurations. The key lesson learned is the importance of utilizing checklists. Brief checklists ensure that every critical step is noticed during the migration process. To streamline the procedure, we&#8217;ve created two distinct checklists\u2014one for production and another for non-production environments.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">\u00a0 \u00a0 The checklist for the production environment encompasses essential steps such as:<\/span><\/p>\n<ol>\n<li style=\"list-style-type: none;\">\n<ol>\n<li><span style=\"font-weight: 400;\">Prepare detailed steps involved in the cutover process to minimize downtime and ensure a smooth transition.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Identification of stakeholders involved in different stages of the migration, ensuring clear communication and collaboration.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Thorough documentation of each step to provide a detailed record for future reference or auditing.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">Inclusion of prerequisite steps such as ensuring that AWS Service quotas are sufficient for the migration. This might involve steps such as raising an AWS Support Case to warm up load balancers or increasing soft limits for the number of targets in a target group.<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<ul>\n<li style=\"font-weight: 400;\"><b>Identifying the Data Generators is Crucial<\/b><b>:- <\/b><span style=\"font-weight: 400;\">Understanding your data landscape is important for making better decisions.\u00a0<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">In this context, data generators refer to applications, tools, or software that process static data residing on storage platforms such as S3 or file systems. They play a crucial role in seeding data into other data sources like databases or queuing systems such as Kafka in our case. In scenarios involving these data generators, there is no need for direct migration of the data, as the data generators themselves autonomously generate it.<\/span><\/p>\n<h2><b>Conclusion<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">Partnering with a managed cloud services provider who can help you choose the right migration path is one of the surefire ways to overcome this cloud migration challenge.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">We at <\/span><b>TO THE NEW<\/b><span style=\"font-weight: 400;\"> provide simplified migration to the AWS cloud by following the above activities to deploy the solution for you. A cloud-managed services provider can also help you manage the solution and provide backup and disaster recovery for your business. Outsourcing AWS migration to <\/span><b>TO THE NEW\u2019s\u00a0 AWS Certified Architects &amp; DevOps Engineers<\/b><span style=\"font-weight: 400;\"> can help you save time and resources and increase business efficiency.<\/span><\/p>\n<div class=\"ap-custom-wrapper\"><\/div><!--ap-custom-wrapper-->","protected":false},"excerpt":{"rendered":"<p>Introduction For many organizations, the journey to the cloud begins with an important decision: migrating from their traditional on-premises data centers to Amazon Web Services (AWS). This transition demands courage, dedication, and extensive planning, as it involves moving their already operational infrastructure to the cloud without downtime. One such great journey is the story of [&hellip;]<\/p>\n","protected":false},"author":1721,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":231},"categories":[2348],"tags":[5689,248,5688,1892],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/60376"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/1721"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=60376"}],"version-history":[{"count":4,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/60376\/revisions"}],"predecessor-version":[{"id":77505,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/60376\/revisions\/77505"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=60376"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=60376"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=60376"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}