{"id":61712,"date":"2024-05-14T02:57:39","date_gmt":"2024-05-13T21:27:39","guid":{"rendered":"https:\/\/www.tothenew.com\/blog\/?p=61712"},"modified":"2024-07-08T10:30:01","modified_gmt":"2024-07-08T05:00:01","slug":"driving-efficiency-and-cost-reduction-kafka-migration-to-aws-msk-for-a-leading-advertising-firm","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/driving-efficiency-and-cost-reduction-kafka-migration-to-aws-msk-for-a-leading-advertising-firm\/","title":{"rendered":"Driving Efficiency and Cost Reduction: Kafka Migration to AWS MSK for a Leading Advertising Firm"},"content":{"rendered":"

Introduction<\/b><\/h1>\n

In the world of data management, companies seek to streamline operations and enhance scalability. One key journey involves migrating self-managed Apache Kafka clusters from AWS EC2 to Amazon MSK. We executed such a migration for a client with zero downtime, offering insights and strategies in this blog.<\/p>\n

Motivations Behind Migration<\/b><\/h2>\n
    \n
  1. Scalability Limitations<\/b>: <\/b>Scaling self-managed Kafka clusters for increasing data volumes and processing demands was challenging, requiring manual intervention for deploying additional EC2 instances, configuring files, and rebalancing partitions.<\/li>\n<\/ol>\n
      \n
    1. Operational Overhead: <\/b>Self-managed Kafka system requires significant human efforts and knowledge to deploy, configure, and maintain effectively. Security patches, monitoring, and backup plans added to the administrative burden, diverting minds from real business requirements.<\/span><\/li>\n<\/ol>\n
        \n
      1. Efficiency Issues: <\/b>Underutilized resource utilization & over-provisioned capacity, <\/span>resulted in unnecessary costs and operational inefficiencies.<\/span>\u00a0<\/span><\/li>\n<\/ol>\n
          \n
        1. Upgrading Issues: <\/b>Upgrading Kafka versions in self-managed environments was difficult and time-consuming, requiring careful planning, testing, and coordination to minimize blast radius and ensure compatibility with existing applications and infrastructure.<\/span><\/li>\n<\/ol>\n
            \n
          1. Security and Compliance Reasons: <\/b>Of course not upgrading the Kafka cluster resulted in many security & compliance issues. The Kafka version was old and had many security vulnerabilities.<\/span><\/li>\n<\/ol>\n

            Existing Setup and Cost Analysis<\/b><\/h2>\n
              \n
            1. Kafka Cluster Configurations<\/b>: The setup included 15<\/strong> Kafka nodes with m5.2xlarge<\/strong> instances, totaling around 200 TB<\/strong> of disk space. Apache Kafka Version: 0.8.2.<\/span><\/li>\n
            2. Zookeeper Node Configurations<\/b>: 3 <\/span><\/span>Kafka Zookeeper nodes, using c5.xlarge<\/strong> instances with 100 GB disks, managed the Kafka cluster.<\/li>\n
            3. Replication Factor in Kafka:<\/b> The replication factor refers to the number of copies maintained for each Kafka topic partition across the cluster. In our setup, each message in Kafka was triplicated across 3 brokers, maintaining RF of 3.<\/span><\/li>\n
            4. Inter-AZ Data Transfer Costs:<\/b>\u00a0 Architecture was spanned across 3 AZs in the US-West-2 region on AWS for high availability. However, AWS imposes inter-AZ data transfer costs for communication between Kafka brokers located in different availability zones. The architecture, designed to ensure HA, inadvertently incurred substantial inter-AZ data transfer costs, contributing significantly to monthly AWS expenses.<\/span><\/li>\n<\/ol>\n

              Problem Statement<\/b><\/h2>\n

              Adding all these factors & infrastructure was costing us monthly expenditures, totaling approximately 40,000 US Dollars. The primary cost driver was from inter-AZ data transfer charges, reflecting the substantial volume of data exchanged between Kafka brokers across availability zones. To address this issue, we explored migrating to Amazon MSK for benefits like high availability, quick scaling, and no data transfer costs between brokers across different AZs.<\/p>\n

              Open Questions<\/b><\/h2>\n