From Logstash to Fluent Bit: How We Streamlined Logging for an Ad Tech Client

Introduction

In ad-tech, logs are not “nice to have.” They are the product’s heartbeat. Every impression, every click, every bid request — everything generates logs. Multiply that by millions of requests per minute, and you’re suddenly dealing with millions of events and TB’s of logs per day. That’s exactly where one of our platforms was. And that’s where Logstash started hurting us.

log forwarding

In this blog, we will see why we migrated from logstash to Fluentbit as our standard log forwarder. Let’s get started.

Problem Statement: When Logstash Became the Bottleneck

Logstash had been running for a while, and it worked — until traffic scaled. Under peak load, we saw:

• Memory usage is climbing aggressively
• CPU spikes during traffic bursts
• Slower log ingestion
• Delayed troubleshooting
• High Resource Utilization resulting in high cloud cost

The bigger issue wasn’t just resource usage. It was operational friction. When logging becomes heavy, debugging becomes slow. And in ad tech, slow debugging directly impacts revenue. We needed something lighter, and that’s where Fluentbit comes into the picture.

Why We Chose Fluent Bit

We weren’t looking for fancy features. We wanted:

• Lower resource usage
• Stable performance under load
• Structured logs across services
• Predictable cost and a cheap log forwarder

Fluent Bit checked those boxes. It’s lightweight by design, built for container environments, and doesn’t need the JVM overhead that Logstash carries. That alone made it worth testing in our ECS setup.

How We Deployed It

We run our services on ECS Fargate, so the cleanest approach was to deploy Fluent Bit as a sidecar container. Each application task got its own logging sidecar. That meant:

• No dependency on host-level log collectors
• Same setup across dev, QA, staging, and prod
• Logs collected directly from the container
• This simplified everything. Logging became part of the task definition. If a service scaled, its logging scaled with it.
• No special infrastructure. No separate logging nodes.

Standardizing the Logs (This Helped More Than We Expected)

One problem we had before migration was inconsistency. Different services are logged differently. Searching across services meant guessing field names or parsing raw messages manually. So during migration, we created custom parsing rules in Fluent Bit and enforced structured fields:

• classname
• loglevel
• thread
• message
• timestamp

It sounds basic, but this changed how fast we could troubleshoot. Dashboards became clean. Alerts became precise. Engineers stopped digging through raw log blobs. This alone improved operational speed more than the tool switch itself.

The Unexpected CPU Win

During testing, we noticed something interesting. Fluent Bit was still consuming noticeable CPU under high traffic. After digging into it, we experimented with:

Inotify_Watcher false

This switches Fluent Bit from event-based file watching to polling.

After that change:

• CPU usage dropped significantly
• Stability improved during traffic spikes
• No log loss
• No performance impact on the applications

It was one of those small tweaks that delivered disproportionate value. No major architecture change. Just one config line.

The Cost Impact: FinOps Win

Logstash wasn’t just heavy — it was expensive. High memory and CPU meant larger task sizes in production. That adds up quickly in ECS, especially in an ad-tech workload where traffic fluctuates constantly.

After migrating:

• Smaller containers
• More predictable scaling
• Lower overall infrastructure cost

We didn’t sacrifice reliability. We just removed waste.

What Changed After Migration

Once Fluent Bit was fully rolled out:

• Production memory and CPU stabilized
• Logs became consistent across all services
• Troubleshooting got faster
• Alerts became cleaner
• The infrastructure cost dropped

But the biggest win wasn’t technical. The team stopped worrying about the logging pipeline. That mental overhead disappearing was huge.

What We Learned

A few practical lessons from this migration:

1. Logging tools should not compete with your application for resources.
–> If your log collector needs serious compute, something is off.

2. Structured logging matters more than the tool itself.
–> Clean fields improve debugging more than swapping technologies.

3. Small configuration changes can outperform big architectural shifts.
–> The Inotify_Watcher tweak alone made a measurable difference.

4. Sidecars work well in ECS if done consistently.
–> It kept environments identical and reduced drift.

Final Thoughts

This wasn’t a “modernization” story. It was a cleanup story.

• We removed unnecessary overhead.
• We standardized logs.
• We reduced the cost.
• We simplified operations. That’s it.

For teams running high-volume container workloads, Fluent Bit isn’t just lighter than Logstash — it’s more aligned with how modern infrastructure actually runs. And sometimes, the best upgrade is the one that makes your system boring again. Reach out to us at TO THE NEW for simplifying your logging!