{"id":78092,"date":"2026-03-15T09:22:46","date_gmt":"2026-03-15T03:52:46","guid":{"rendered":"https:\/\/www.tothenew.com\/blog\/?p=78092"},"modified":"2026-03-16T09:53:43","modified_gmt":"2026-03-16T04:23:43","slug":"from-logstash-to-fluent-bit-how-we-streamlined-logging-for-an-ad-tech-client","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/from-logstash-to-fluent-bit-how-we-streamlined-logging-for-an-ad-tech-client\/","title":{"rendered":"From Logstash to Fluent Bit: How We Streamlined Logging for an Ad Tech Client"},"content":{"rendered":"<h2><span style=\"text-decoration: underline;\"><strong>Introduction<\/strong><\/span><\/h2>\n<p>In ad-tech, logs are not \u201c<strong>nice to have.<\/strong>\u201d They are the product\u2019s heartbeat. Every<strong> impression, every click, every bid request<\/strong> \u2014 everything generates logs. Multiply that by millions of requests per minute, and you\u2019re suddenly dealing with millions of events and TB\u2019s of logs per day. That\u2019s exactly where one of our platforms was. And that\u2019s where Logstash started hurting us.<\/p>\n<div id=\"attachment_78091\" style=\"width: 635px\" class=\"wp-caption alignleft\"><img aria-describedby=\"caption-attachment-78091\" decoding=\"async\" loading=\"lazy\" class=\"wp-image-78091 size-large\" src=\"https:\/\/www.tothenew.com\/blog\/wp-ttn-blog\/uploads\/2026\/03\/m-1024x314.png\" alt=\"log forwarding\" width=\"625\" height=\"192\" srcset=\"\/blog\/wp-ttn-blog\/uploads\/2026\/03\/m-1024x314.png 1024w, \/blog\/wp-ttn-blog\/uploads\/2026\/03\/m-300x92.png 300w, \/blog\/wp-ttn-blog\/uploads\/2026\/03\/m-768x236.png 768w, \/blog\/wp-ttn-blog\/uploads\/2026\/03\/m-624x191.png 624w, \/blog\/wp-ttn-blog\/uploads\/2026\/03\/m.png 1108w\" sizes=\"(max-width: 625px) 100vw, 625px\" \/><p id=\"caption-attachment-78091\" class=\"wp-caption-text\">log forwarding<\/p><\/div>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>In this blog, we will see why we migrated from logstash to Fluentbit as our standard log forwarder. Let\u2019s get started.<\/p>\n<h2><span style=\"text-decoration: underline;\"><strong>Problem Statement: When Logstash Became the Bottleneck<\/strong><\/span><\/h2>\n<p>Logstash had been running for a while, and it worked \u2014 until traffic scaled. Under peak load, we saw:<\/p>\n<ul>\n<li>Memory usage is climbing aggressively<\/li>\n<li>CPU spikes during traffic bursts<\/li>\n<li>Slower log ingestion<\/li>\n<li>Delayed troubleshooting<\/li>\n<li>High Resource Utilization resulting in high cloud cost<\/li>\n<\/ul>\n<p>The bigger issue wasn\u2019t just resource usage. It was operational friction. When logging becomes heavy, debugging becomes slow. And in ad tech, slow debugging directly impacts revenue. We needed something lighter, and that\u2019s where Fluentbit comes into the picture.<\/p>\n<h2><span style=\"text-decoration: underline;\"><strong>Why We Chose Fluent Bit<\/strong><\/span><\/h2>\n<p>We weren\u2019t looking for fancy features. We wanted:<\/p>\n<ul>\n<li>Lower resource usage<\/li>\n<li>Stable performance under load<\/li>\n<li>Structured logs across services<\/li>\n<li>Predictable cost and a cheap log forwarder<\/li>\n<\/ul>\n<p>Fluent Bit checked those boxes. It\u2019s lightweight by design, built for container environments, and doesn\u2019t need the<strong> JVM overhead<\/strong> that Logstash carries. That alone made it worth testing in our ECS setup.<\/p>\n<h2><span style=\"text-decoration: underline;\"><strong>How We Deployed It<\/strong><\/span><\/h2>\n<p>We run our services on ECS Fargate, so the cleanest approach was to deploy Fluent Bit as a sidecar container. Each application task got its own logging sidecar. That meant:<\/p>\n<ul>\n<li>No dependency on host-level log collectors<\/li>\n<li>Same setup across dev, QA, staging, and prod<\/li>\n<li>Logs collected directly from the container<\/li>\n<li>This simplified everything. Logging became part of the task definition. If a service scaled, its logging scaled with it.<\/li>\n<li>No special infrastructure. No separate logging nodes.<\/li>\n<\/ul>\n<h2><span style=\"text-decoration: underline;\"><strong>Standardizing the Logs (This Helped More Than We Expected)<\/strong><\/span><\/h2>\n<p>One problem we had before migration was inconsistency. Different services are logged differently. Searching across services meant guessing field names or parsing raw messages manually. So during migration, we created custom parsing rules in Fluent Bit and enforced structured fields:<\/p>\n<ul>\n<li>classname<\/li>\n<li>loglevel<\/li>\n<li>thread<\/li>\n<li>message<\/li>\n<li>timestamp<\/li>\n<\/ul>\n<p>It sounds basic, but this changed how fast we could troubleshoot. Dashboards became clean. Alerts became precise. Engineers stopped digging through raw log blobs. This alone improved operational speed more than the tool switch itself.<\/p>\n<h2><span style=\"text-decoration: underline;\"><strong>The Unexpected CPU Win<\/strong><\/span><\/h2>\n<p>During testing, we noticed something interesting. Fluent Bit was still consuming noticeable CPU under high traffic. After digging into it, we experimented with:<\/p>\n<pre>Inotify_Watcher false<\/pre>\n<p>This switches Fluent Bit from event-based file watching to polling.<\/p>\n<p>After that change:<\/p>\n<ul>\n<li>CPU usage dropped significantly<\/li>\n<li>Stability improved during traffic spikes<\/li>\n<li>No log loss<\/li>\n<li>No performance impact on the applications<\/li>\n<\/ul>\n<p>It was one of those small tweaks that delivered disproportionate value. No major architecture change. Just one config line.<\/p>\n<h2><span style=\"text-decoration: underline;\"><strong>The Cost Impact: FinOps Win<\/strong><\/span><\/h2>\n<p>Logstash wasn\u2019t just heavy \u2014 it was expensive. <strong>High memory and CPU<\/strong> meant larger task sizes in production. That adds up quickly in ECS, especially in an ad-tech workload where traffic fluctuates constantly.<\/p>\n<p>After migrating:<\/p>\n<ul>\n<li>Smaller containers<\/li>\n<li>More predictable scaling<\/li>\n<li>Lower overall infrastructure cost<\/li>\n<\/ul>\n<p>We didn\u2019t sacrifice reliability. We just removed waste.<\/p>\n<h2><span style=\"text-decoration: underline;\"><strong>What Changed After Migration<\/strong><\/span><\/h2>\n<p>Once Fluent Bit was fully rolled out:<\/p>\n<ul>\n<li>Production memory and CPU stabilized<\/li>\n<li>Logs became consistent across all services<\/li>\n<li>Troubleshooting got faster<\/li>\n<li>Alerts became cleaner<\/li>\n<li>The infrastructure cost dropped<\/li>\n<\/ul>\n<p>But the biggest win wasn\u2019t technical. The team stopped worrying about the logging pipeline. That mental overhead disappearing was huge.<\/p>\n<h2><span style=\"text-decoration: underline;\"><strong>What We Learned<\/strong><\/span><\/h2>\n<p>A few practical lessons from this migration:<\/p>\n<p><strong>1. Logging tools should not compete with your application for resources.<\/strong><br \/>\n&#8211;&gt; If your log collector needs serious compute, something is off.<\/p>\n<p><strong>2. Structured logging matters more than the tool itself.<\/strong><br \/>\n&#8211;&gt; Clean fields improve debugging more than swapping technologies.<\/p>\n<p><strong>3. Small configuration changes can outperform big architectural shifts.<\/strong><br \/>\n&#8211;&gt; The Inotify_Watcher tweak alone made a measurable difference.<\/p>\n<p><strong>4. Sidecars work well in ECS if done consistently.<\/strong><br \/>\n&#8211;&gt; It kept environments identical and reduced drift.<\/p>\n<h2><span style=\"text-decoration: underline;\"><strong>Final Thoughts<\/strong><\/span><\/h2>\n<p>This wasn\u2019t a <strong>\u201cmodernization\u201d<\/strong> story. It was a cleanup story.<\/p>\n<ul>\n<li>We removed unnecessary overhead.<\/li>\n<li>We standardized logs.<\/li>\n<li>We reduced the cost.<\/li>\n<li>We simplified operations. That\u2019s it.<\/li>\n<\/ul>\n<p>For teams running high-volume container workloads, Fluent Bit isn\u2019t just lighter than Logstash \u2014 it\u2019s more aligned with how modern infrastructure actually runs. And sometimes, the best upgrade is the one that makes your system boring again. Reach out to us at <a href=\"https:\/\/www.tothenew.com\/\"><strong>TO THE NEW<\/strong><\/a> for simplifying your logging!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction In ad-tech, logs are not \u201cnice to have.\u201d They are the product\u2019s heartbeat. Every impression, every click, every bid request \u2014 everything generates logs. Multiply that by millions of requests per minute, and you\u2019re suddenly dealing with millions of events and TB\u2019s of logs per day. That\u2019s exactly where one of our platforms was. [&hellip;]<\/p>\n","protected":false},"author":1601,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":24},"categories":[2348],"tags":[7232,248,5547,6723,4494,5210,1892,5947,3688,6131,7541,6457,1587,288,3390,3979,1499,7501,7760,7323],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/78092"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/1601"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=78092"}],"version-history":[{"count":2,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/78092\/revisions"}],"predecessor-version":[{"id":78505,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/78092\/revisions\/78505"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=78092"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=78092"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=78092"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}