{"id":74028,"date":"2025-08-21T13:55:15","date_gmt":"2025-08-21T08:25:15","guid":{"rendered":"https:\/\/www.tothenew.com\/blog\/?p=74028"},"modified":"2025-09-03T19:52:15","modified_gmt":"2025-09-03T14:22:15","slug":"centralized-compliance-logging-on-aws-fluentd-ec2-to-s3-for-siem-integration","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/centralized-compliance-logging-on-aws-fluentd-ec2-to-s3-for-siem-integration\/","title":{"rendered":"Centralized Compliance Logging on AWS with Fluentd: EC2 to S3 for SIEM Integration"},"content":{"rendered":"<h2>Introduction<\/h2>\n<p>Centralized log visibility across various systems is necessary and modern organizations must adhere to stringent compliance requirements. For audit and SIEM (Security Information and Event Management) purposes a dependable log forwarding pipeline is necessary whether it is for monitoring authentication, events, system activity or security logs<\/p>\n<p>A robust open-source data collection tool called Fluentd aggregates log data from various sources and sends it to several locations. To forward important logs into Amazon S3 it can be installed on Linux and Windows EC2 instances on AWS. The information can then be ingested by SIEM platforms such as IBM QRadar Splunk or Elastic SIEM for security auditing and real-time threat detection<\/p>\n<p>In this blog you will find a detailed guide to installing and configuring Fluentd on Linux and Windows EC2 instances., covering:<\/p>\n<ul>\n<li>Installation and configuration procedures Manually.<\/li>\n<li>Automated deployment using AWS Systems Manager (Linux) and PowerShell scripting (Windows).<\/li>\n<li>Best practices for buffering, retention, and IAM permissions.<\/li>\n<\/ul>\n<p>By the end, you\u2019ll have a scalable solution where:<\/p>\n<ul>\n<li>Fluentd runs as a service on both Linux and Windows EC2.<\/li>\n<li>System, authentication, application, and security logs are continuously streamed to Amazon S3.<\/li>\n<li>Logs are easily integrated with SIEM platforms for compliance and centralized security monitoring.<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<h2>Purpose<\/h2>\n<p>The main goal of this arrangement is to make it easy to integrate SIEM for security and compliance monitoring.When you use Fluentd on EC2 instances (Linux and Windows), it collects logs of system events, authentication attempts, and security activity and sends them to Amazon S3.<\/p>\n<p>From there, SIEM solutions like IBM QRadar, Splunk, and Elastic SIEM can take in these logs for:<\/p>\n<ul>\n<li>Following the rules (PCI-DSS, SOC 2, HIPAA, ISO 27001)<\/li>\n<li>Centralized security monitoring for several EC2 setups.<\/li>\n<li>Detecting threats and responding to incidents in real time<\/li>\n<li>Audit readiness with organised log storage on S3<\/li>\n<\/ul>\n<p>This ensures organizations maintain visibility, compliance, and security posture across hybrid environments.<\/p>\n<p><strong>Manual Setup<br \/>\n<\/strong><\/p>\n<ol>\n<li><strong>\u00a0Save the Script<br \/>\n<\/strong><\/li>\n<\/ol>\n<p>Paste the script into a newly created file called <span style=\"color: #99cc00;\">install_fluentd.sh<\/span>:<\/p>\n<p>Script : <a href=\"https:\/\/raw.githubusercontent.com\/prateek21390\/logging\/refs\/heads\/main\/fluentd-linux.sh\" target=\"_blank\" rel=\"noopener\">install_fluentd.sh<\/a><\/p>\n<p><strong>\u00a0 2. Run the Script<\/strong><br \/>\nchmod +x install_fluentd.sh<br \/>\nsudo .\/install_fluentd.sh<\/p>\n<p><strong>Automatic Configuration (AWS Systems Manager).<\/strong><br \/>\nFluentd can be automatically deployed across EC2 instances using AWS Systems Manager (SSM) State Manager eliminating the need for manual SSH access<strong>.<\/strong><\/p>\n<p><strong> Prerequisites<\/strong><\/p>\n<ul>\n<li>Make sure EC2 is assigned an IAM role with:<br \/>\nAmazonSSMManagedInstanceCore<br \/>\nAmazonS3FullAccess (limitations on S3 write access)<\/li>\n<li>Make sure the SSM Agent is installed and operational (it is the default in Ubuntu and Amazon Linux AMIs).<\/li>\n<\/ul>\n<p><strong>3. Form an Association for State Managers<\/strong><\/p>\n<ol>\n<li>To access State Manager navigate to AWS Systems Manager Console<\/li>\n<li>To create an association click Create association.<\/li>\n<li>Select AWS-RunShellScript under Document.<\/li>\n<li>In the Command parameters section paste the scripts content.<\/li>\n<li>To choose your EC2 instances select Specify instance manually.<\/li>\n<li>For schedule, select:<br \/>\nRun association immediately (for one-time execution), or<br \/>\nRun on schedule (for recurring setups).<\/li>\n<li>Click Create Association.<\/li>\n<\/ol>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"wp-image-74047 size-full\" src=\"https:\/\/www.tothenew.com\/blog\/wp-ttn-blog\/uploads\/2025\/08\/st-linux.png\" alt=\"st-linux\" width=\"637\" height=\"987\" srcset=\"\/blog\/wp-ttn-blog\/uploads\/2025\/08\/st-linux.png 637w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/st-linux-194x300.png 194w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/st-linux-624x967.png 624w\" sizes=\"(max-width: 637px) 100vw, 637px\" \/><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"wp-image-74027 size-large\" src=\"https:\/\/www.tothenew.com\/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-18-14-48-45-2-1024x637.png\" alt=\"s3\" width=\"625\" height=\"389\" srcset=\"\/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-18-14-48-45-2-1024x637.png 1024w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-18-14-48-45-2-300x187.png 300w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-18-14-48-45-2-768x478.png 768w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-18-14-48-45-2-624x388.png 624w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-18-14-48-45-2.png 1378w\" sizes=\"(max-width: 625px) 100vw, 625px\" \/><\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"wp-image-74024 size-large\" src=\"https:\/\/www.tothenew.com\/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-18-14-50-50-1-1024x674.png\" alt=\"image\" width=\"625\" height=\"411\" srcset=\"\/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-18-14-50-50-1-1024x674.png 1024w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-18-14-50-50-1-300x197.png 300w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-18-14-50-50-1-768x505.png 768w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-18-14-50-50-1-624x411.png 624w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-18-14-50-50-1.png 1392w\" sizes=\"(max-width: 625px) 100vw, 625px\" \/><\/p>\n<p><strong>Troubleshooting<\/strong><\/p>\n<ul>\n<li>Metadata Access: Verify that EC2 has the metadata service (IMDS) enabled.<\/li>\n<li>Examine the \/var\/log\/fluentd\/fluentd.log<\/li>\n<li>Running sudo systemctl status fluentd.service will fix service issues.<\/li>\n<li>Problems with S3 Access: Make sure the IAM role has the right write permissions to S3 buckets.<\/li>\n<\/ul>\n<h2>Setup for Fluentd on Windows<\/h2>\n<p><strong>1. Install Fluentd Package<\/strong><br \/>\n<strong>Step 1:\u00a0Download Fluentd LTS as the first step<\/strong><br \/>\nThe MSI installer can be downloaded from: <a href=\"https:\/\/docs.fluentd.org\/installation\/install-by-msi\">Fluentd Windows MSI Installer<\/a><\/p>\n<p><strong>Step 2:\u00a0Run Fluentd as a Windows Service<\/strong><\/p>\n<ul>\n<li>GUI: Start fluentdwinsvc from Control Panel \u2192 Services<\/li>\n<li>Command Prompt:<br \/>\nnet start fluentdwinsvc<\/li>\n<li>PowerShell:<br \/>\nStart-Service fluentdwinsvc<\/li>\n<\/ul>\n<p><strong>2. Fluentd Configuration (Windows Event Logs \u2192 S3)<\/strong><br \/>\nCreate\/edit <span style=\"color: #339966;\">fluentd.conf<\/span>:<\/p>\n<pre>&lt;source&gt;\r\n\r\n@type windows_eventlog\r\n\r\nchannels [\"System\", \"Application\", \"Security\"]\u00a0 # Collect System, Application, and Security logs\r\n\r\nread_interval 2\u00a0 # Fetch logs every 2 seconds\r\n\r\ntag windows.eventlog\u00a0 # Tag for filtering logs\r\n\r\n&lt;storage&gt;\r\n\r\n@type local\r\n\r\npersistent true\r\n\r\npath C:\/fluentd\/storage\/system_winlog.json\r\n\r\nremove_on_idle true\r\n\r\nremove_interval 3600\u00a0 # Remove logs older than 2 hours (7200 seconds)\r\n\r\n&lt;\/storage&gt;\r\n\r\n&lt;\/source&gt;\r\n\r\n&lt;match windows.eventlog&gt;\r\n\r\n@type s3\r\n\r\ns3_bucket qradar-seim-prod \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # Replace with your S3 bucket name\r\n\r\ns3_region ap-south-1\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # Replace with your S3 region (e.g., us-east-1)\r\n\r\npath windows\/i-08a4e0b773c0ca1ff\/%Y\/%m\/%d\/\u00a0 # Use dynamic values from the record to organize logs by instance\r\n\r\ntime_slice_format %Y%m%d%H\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # Log chunk format: year, month, day, hour\r\n\r\ntime_slice_wait 2m \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # Wait before finalizing a log chunk\r\n\r\nlocaltime true\r\n\r\nbuffer_path C:\/opt\/fluent\/buffer\/s3 \u00a0 \u00a0 \u00a0 \u00a0 # Buffer directory for Fluentd\r\n\r\n&lt;format&gt;\r\n\r\n@type json\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # Store logs in JSON format\r\n\r\n&lt;\/format&gt;\r\n\r\n&lt;buffer&gt;\r\n\r\n@type file\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # File-based buffer\r\n\r\npath C:\/opt\/fluent\/buffer\/s3\r\n\r\nflush_mode interval \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # Flush logs at intervals\r\n\r\nflush_interval 1m \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # Flush every 1 minute\r\n\r\nretry_max_times 10\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # Retry up to 10 times on failure\r\n\r\nretry_wait 30 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # Wait 30 seconds between retries\r\n\r\nchunk_limit_records 1000\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # Adjust based on your needs (optional)\r\n\r\nflush_thread_count 2\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # Adjust for performance (optional)\r\n\r\ntimekey 3600\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # Logs will expire after 2 hours (2 * 60 * 60)\r\n\r\n&lt;\/buffer&gt;\r\n\r\n&lt;\/match&gt;<\/pre>\n<p>&nbsp;<\/p>\n<p><strong>3. Time Slice &amp; Retention<\/strong><br \/>\ntime_slice_format %Y%m%d%H: Groups logs hourly.<br \/>\ntime_slice_wait 10m: Wait 10 minutes to ensure late logs are included.<br \/>\nRetention: Logs auto-cleaned if idle for 24 hours.<\/p>\n<p><strong>4. List Installed Plugins<\/strong><br \/>\nfluent-gem list<\/p>\n<p><strong>5. Automation Script for Fluentd Installation on Windows<\/strong><\/p>\n<ol>\n<li><strong>Script for Windows Server Installation<\/strong><br \/>\nScript : <a href=\"https:\/\/raw.githubusercontent.com\/prateek21390\/logging\/refs\/heads\/main\/fluend-s3-windows.ps1\" target=\"_blank\" rel=\"noopener\">windows_fluentd.ps1<\/a><\/li>\n<li><strong>Create a State Manager Association<\/strong><\/li>\n<\/ol>\n<ul>\n<li>Go to AWS Systems Manager Console \u2192 State Manager.<\/li>\n<li>Click Create association.<\/li>\n<li>Under Document, choose AWS-RunPowerShellScript.<\/li>\n<li>Paste the script content from above in the Command parameters section.<\/li>\n<li>Choose Specify instance manually and select your EC2 instances.<\/li>\n<li>For schedule, select:<br \/>\nRun association immediately (for one-time execution), or<br \/>\nRun on schedule (for recurring setups).<\/li>\n<li>Click Create Association.<\/li>\n<\/ul>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"wp-image-74044 size-large\" src=\"https:\/\/www.tothenew.com\/blog\/wp-ttn-blog\/uploads\/2025\/08\/ssm-state-windows-1024x540.png\" alt=\"st\" width=\"625\" height=\"330\" srcset=\"\/blog\/wp-ttn-blog\/uploads\/2025\/08\/ssm-state-windows-1024x540.png 1024w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/ssm-state-windows-300x158.png 300w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/ssm-state-windows-768x405.png 768w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/ssm-state-windows-624x329.png 624w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/ssm-state-windows.png 1517w\" sizes=\"(max-width: 625px) 100vw, 625px\" \/><\/p>\n<p>&nbsp;<\/p>\n<p><strong>This script:<\/strong><\/p>\n<ul>\n<li>Evaluates the version of Windows Server.<\/li>\n<li>Through MSI installs Fluentd.<\/li>\n<li>EC2 Instance ID is retrieved.<\/li>\n<li>Fluentd is configured to send Event Logs to S3.<\/li>\n<li>Fluentd is launched as a Windows service.<\/li>\n<\/ul>\n<h2>Conclusion<\/h2>\n<p>Organizations create a unified and scalable log forwarding pipeline that guarantees all crucial security and compliance data is recorded by implementing Fluentd across Linux and Windows EC2 instances. System application and security event logs from Windows servers and systemd authentication and application logs from Linux systems are continuously streamed into Amazon S3 by Fluentd which establishes a central location for organized log storage. With the help of this architecture, logs can be correlated, examined, and responded to in real time on top SIEM platforms like IBM QRadar, Splunk, and Elastic SIEM. The strategy not only backs frameworks for regulatory compliance (e. g. 3. enhances threat detection incident response and overall security posture (PCI-DSS HIPAA SOC 2 ISO 27001. A dependable and compliant basis for centralized security monitoring across multi-OS EC2 architectures is offered by this solution, which can be set up manually or automatically using AWS Systems Manager (Linux) and PowerShell (Windows). In the end, Fluentd makes sure that your cloud environment stays secure, audit-ready, and visibility-driven regardless of the operating system or workload.<\/p>\n<p>This architecture enables seamless integration with leading SIEM platforms such as IBM QRadar, Splunk, and Elastic SIEM, where logs can be correlated, analyzed, and acted upon in real time. The approach not only supports regulatory compliance frameworks (e.g., PCI-DSS, HIPAA, SOC 2, ISO 27001) but also enhances threat detection, incident response, and overall security.<\/p>\n<p>A dependable and compliant basis for centralized security monitoring across multi-OS EC2 architectures is offered by this solution, which can be set up manually or automatically using AWS Systems Manager (Linux) and PowerShell (Windows). In the end, Fluentd makes sure that your cloud environment stays secure, audit-ready and visibility-driven regardless of the operating system or workload.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction Centralized log visibility across various systems is necessary and modern organizations must adhere to stringent compliance requirements. For audit and SIEM (Security Information and Event Management) purposes a dependable log forwarding pipeline is necessary whether it is for monitoring authentication, events, system activity or security logs A robust open-source data collection tool called Fluentd [&hellip;]<\/p>\n","protected":false},"author":2010,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":41},"categories":[2348],"tags":[3802,7554],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/74028"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/2010"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=74028"}],"version-history":[{"count":23,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/74028\/revisions"}],"predecessor-version":[{"id":75247,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/74028\/revisions\/75247"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=74028"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=74028"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=74028"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}