Centralized Compliance Logging on AWS with Fluentd: EC2 to S3 for SIEM Integration

Introduction

Centralized log visibility across various systems is necessary and modern organizations must adhere to stringent compliance requirements. For audit and SIEM (Security Information and Event Management) purposes a dependable log forwarding pipeline is necessary whether it is for monitoring authentication, events, system activity or security logs

A robust open-source data collection tool called Fluentd aggregates log data from various sources and sends it to several locations. To forward important logs into Amazon S3 it can be installed on Linux and Windows EC2 instances on AWS. The information can then be ingested by SIEM platforms such as IBM QRadar Splunk or Elastic SIEM for security auditing and real-time threat detection

In this blog you will find a detailed guide to installing and configuring Fluentd on Linux and Windows EC2 instances., covering:

• Installation and configuration procedures Manually.
• Automated deployment using AWS Systems Manager (Linux) and PowerShell scripting (Windows).
• Best practices for buffering, retention, and IAM permissions.

By the end, you’ll have a scalable solution where:

• Fluentd runs as a service on both Linux and Windows EC2.
• System, authentication, application, and security logs are continuously streamed to Amazon S3.
• Logs are easily integrated with SIEM platforms for compliance and centralized security monitoring.

Purpose

The main goal of this arrangement is to make it easy to integrate SIEM for security and compliance monitoring.When you use Fluentd on EC2 instances (Linux and Windows), it collects logs of system events, authentication attempts, and security activity and sends them to Amazon S3.

From there, SIEM solutions like IBM QRadar, Splunk, and Elastic SIEM can take in these logs for:

• Following the rules (PCI-DSS, SOC 2, HIPAA, ISO 27001)
• Centralized security monitoring for several EC2 setups.
• Detecting threats and responding to incidents in real time
• Audit readiness with organised log storage on S3

This ensures organizations maintain visibility, compliance, and security posture across hybrid environments.

Manual Setup

1.  Save the Script
   

Paste the script into a newly created file called install_fluentd.sh:

Script : install_fluentd.sh

  2. Run the Script
chmod +x install_fluentd.sh
sudo ./install_fluentd.sh

Automatic Configuration (AWS Systems Manager).
Fluentd can be automatically deployed across EC2 instances using AWS Systems Manager (SSM) State Manager eliminating the need for manual SSH access.

Prerequisites

• Make sure EC2 is assigned an IAM role with:
  AmazonSSMManagedInstanceCore
  AmazonS3FullAccess (limitations on S3 write access)
• Make sure the SSM Agent is installed and operational (it is the default in Ubuntu and Amazon Linux AMIs).

3. Form an Association for State Managers

1. To access State Manager navigate to AWS Systems Manager Console
2. To create an association click Create association.
3. Select AWS-RunShellScript under Document.
4. In the Command parameters section paste the scripts content.
5. To choose your EC2 instances select Specify instance manually.
6. For schedule, select:
   Run association immediately (for one-time execution), or
   Run on schedule (for recurring setups).
7. Click Create Association.

Troubleshooting

• Metadata Access: Verify that EC2 has the metadata service (IMDS) enabled.
• Examine the /var/log/fluentd/fluentd.log
• Running sudo systemctl status fluentd.service will fix service issues.
• Problems with S3 Access: Make sure the IAM role has the right write permissions to S3 buckets.

Setup for Fluentd on Windows

1. Install Fluentd Package
Step 1: Download Fluentd LTS as the first step
The MSI installer can be downloaded from: Fluentd Windows MSI Installer

Step 2: Run Fluentd as a Windows Service

• GUI: Start fluentdwinsvc from Control Panel → Services
• Command Prompt:
  net start fluentdwinsvc
• PowerShell:
  Start-Service fluentdwinsvc

2. Fluentd Configuration (Windows Event Logs → S3)
Create/edit fluentd.conf:

<source>

@type windows_eventlog

channels ["System", "Application", "Security"]  # Collect System, Application, and Security logs

read_interval 2  # Fetch logs every 2 seconds

tag windows.eventlog  # Tag for filtering logs

<storage>

@type local

persistent true

path C:/fluentd/storage/system_winlog.json

remove_on_idle true

remove_interval 3600  # Remove logs older than 2 hours (7200 seconds)

</storage>

</source>

<match windows.eventlog>

@type s3

s3_bucket qradar-seim-prod               # Replace with your S3 bucket name

s3_region ap-south-1              # Replace with your S3 region (e.g., us-east-1)

path windows/i-08a4e0b773c0ca1ff/%Y/%m/%d/  # Use dynamic values from the record to organize logs by instance

time_slice_format %Y%m%d%H                  # Log chunk format: year, month, day, hour

time_slice_wait 2m                         # Wait before finalizing a log chunk

localtime true

buffer_path C:/opt/fluent/buffer/s3         # Buffer directory for Fluentd

<format>

@type json                                # Store logs in JSON format

</format>

<buffer>

@type file                                # File-based buffer

path C:/opt/fluent/buffer/s3

flush_mode interval                       # Flush logs at intervals

flush_interval 1m                         # Flush every 1 minute

retry_max_times 10                        # Retry up to 10 times on failure

retry_wait 30                             # Wait 30 seconds between retries

chunk_limit_records 1000                  # Adjust based on your needs (optional)

flush_thread_count 2                      # Adjust for performance (optional)

timekey 3600                              # Logs will expire after 2 hours (2 * 60 * 60)

</buffer>

</match>

3. Time Slice & Retention
time_slice_format %Y%m%d%H: Groups logs hourly.
time_slice_wait 10m: Wait 10 minutes to ensure late logs are included.
Retention: Logs auto-cleaned if idle for 24 hours.

4. List Installed Plugins
fluent-gem list

5. Automation Script for Fluentd Installation on Windows

1. Script for Windows Server Installation
   Script : windows_fluentd.ps1
2. Create a State Manager Association

• Go to AWS Systems Manager Console → State Manager.
• Click Create association.
• Under Document, choose AWS-RunPowerShellScript.
• Paste the script content from above in the Command parameters section.
• Choose Specify instance manually and select your EC2 instances.
• For schedule, select:
  Run association immediately (for one-time execution), or
  Run on schedule (for recurring setups).
• Click Create Association.

This script:

• Evaluates the version of Windows Server.
• Through MSI installs Fluentd.
• EC2 Instance ID is retrieved.
• Fluentd is configured to send Event Logs to S3.
• Fluentd is launched as a Windows service.

Conclusion

Organizations create a unified and scalable log forwarding pipeline that guarantees all crucial security and compliance data is recorded by implementing Fluentd across Linux and Windows EC2 instances. System application and security event logs from Windows servers and systemd authentication and application logs from Linux systems are continuously streamed into Amazon S3 by Fluentd which establishes a central location for organized log storage. With the help of this architecture, logs can be correlated, examined, and responded to in real time on top SIEM platforms like IBM QRadar, Splunk, and Elastic SIEM. The strategy not only backs frameworks for regulatory compliance (e. g. 3. enhances threat detection incident response and overall security posture (PCI-DSS HIPAA SOC 2 ISO 27001. A dependable and compliant basis for centralized security monitoring across multi-OS EC2 architectures is offered by this solution, which can be set up manually or automatically using AWS Systems Manager (Linux) and PowerShell (Windows). In the end, Fluentd makes sure that your cloud environment stays secure, audit-ready, and visibility-driven regardless of the operating system or workload.

This architecture enables seamless integration with leading SIEM platforms such as IBM QRadar, Splunk, and Elastic SIEM, where logs can be correlated, analyzed, and acted upon in real time. The approach not only supports regulatory compliance frameworks (e.g., PCI-DSS, HIPAA, SOC 2, ISO 27001) but also enhances threat detection, incident response, and overall security.

A dependable and compliant basis for centralized security monitoring across multi-OS EC2 architectures is offered by this solution, which can be set up manually or automatically using AWS Systems Manager (Linux) and PowerShell (Windows). In the end, Fluentd makes sure that your cloud environment stays secure, audit-ready and visibility-driven regardless of the operating system or workload.