Introduction Apache Kafka has become the backbone of modern event-driven architectures, powering everything from micro services communication to real-time analytics and stream processing. As Kafka deployments grow, developers and operators frequently need answers to questions such as: Are messages reaching Kafka successfully? Which partition received a message? Are consumers keeping up with producers? How much […]
Introduction Real-time systems are now central to modern businesses. Payments, order updates, customer activity, telemetry, notifications, and analytics all depend on events moving quickly and reliably between services. The challenge is not just speed. The challenge is preserving correctness and resilience when systems are distributed, traffic is variable, and failures are inevitable. Apache Kafka […]
Introduction In DevOps, upgrades are rarely exciting. They don’t ship new features (most of the time). They don’t impress clients. They don’t always get leadership applause. And yet, over the years at To The New, one thing has become very clear to us: DevOps teams that do upgrades regularly move faster, stay safer, and break […]
Introduction If you’ve ever worked with Kafka, you know the problem: data grows fast. Every click, impression, or event adds up, and before you know it, your Kafka broker’s disks are full. Disk is not very cheap on AWS, and storing everything on expensive broker storage is costly, and scaling up to handle growth feels […]
Introduction Apache Kafka is the default substrate for event-driven systems. Amazon MSK takes the operational floor out from under most of it (provisioning, patching, storage), but partitions, consumer groups, and replication are still yours to operate. This is a working reference: routines you’ll actually run, what a rolling upgrade really does, the concepts worth internalizing, […]
Imagine you’re managing a busy e-commerce website. Every time a customer places an order, it triggers several events: an email confirmation, a shipping update, a payment confirmation, and much more. From updating the inventory to sending a confirmation email and processing the payment, everything needs to happen instantly and in sync. But how do you […]
Introduction Have you ever wondered about real-time data streaming? If not, let’s embark on a journey to explore the Kafka system and explore its strengths and weaknesses. Before jumping into learning about it, first, we should talk about Big Data, as we are surrounded by an enormous amount of data and the volume is huge; […]
Kafka is a distributed streaming platform designed for real-time data pipelines, stream processing, and data integration. AWS lambda, on the other hand, is a serverless compute service that executes your code in response to events, managing the underlying compute resources for you. In organizations where Kafka plays a central role in streaming and data integration, […]
Introduction As companies grow and their data streaming needs change, it’s important to optimize resources to stay efficient and control costs. AWS has rolled out a new feature for Amazon Managed Streaming for Apache Kafka (Amazon MSK) that enables the removal of brokers from MSK clusters. This capability enables you to adjust the size of […]