{"id":74599,"date":"2025-09-05T23:49:34","date_gmt":"2025-09-05T18:19:34","guid":{"rendered":"https:\/\/www.tothenew.com\/blog\/?p=74599"},"modified":"2025-09-17T11:45:54","modified_gmt":"2025-09-17T06:15:54","slug":"unlock-seamless-multi-account-logging-stream-cloudwatch-logs-to-central-s3-with-kinesis-firehose","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/unlock-seamless-multi-account-logging-stream-cloudwatch-logs-to-central-s3-with-kinesis-firehose\/","title":{"rendered":"Unlock Seamless Multi-Account Logging: Stream CloudWatch logs to Central S3 with Kinesis Firehose"},"content":{"rendered":"<div class=\"container\">\n<h2>Introduction<\/h2>\n<p>Look, if you&#8217;re running stuff across multiple AWS accounts \u2013 dev, staging, prod, maybe even separate accounts because your security team said so \u2013 you already know this pain. Something breaks, alarms start screaming, and suddenly you&#8217;re bouncing between six different accounts trying to figure out what the hell happened.<\/p>\n<div class=\"highlight-box\">\n<p><strong>What I&#8217;m going to show you:<\/strong><\/p>\n<ul>\n<li><strong>The Real Problem:<\/strong> Why scattered logs cost us money and sleep<\/li>\n<li><strong>What We Built:<\/strong> A streaming setup that cut our log storage costs by 70%<\/li>\n<li><strong>The Nitty-Gritty Details:<\/strong> Every resource, every permission, every gotcha<\/li>\n<li><strong>Step-by-Step Setup:<\/strong> Everything&#8217;s in the GitHub repo<\/li>\n<\/ul>\n<\/div>\n<div class=\"container\">\n<h2>What We Actually Built<\/h2>\n<p>Here&#8217;s what we did: we built a pipeline that sucks up CloudWatch logs from every account and dumps them into one massive S3 bucket. No more account hopping, no more missing pieces of the puzzle.<\/p>\n<p>The secret sauce? Kinesis Data Firehose. This thing batches up logs, compresses the hell out of them, and moves everything across account boundaries while we sleep. It&#8217;s like having a dedicated intern whose only job is moving logs around, except this intern never calls in sick.<\/p>\n<div class=\"highlight-box\">\n<p><strong>What this thing actually does:<\/strong><\/p>\n<ul>\n<li>Grabs logs from any CloudWatch group automatically<\/li>\n<li>Squashes them down (we&#8217;re seeing 70% size reduction)<\/li>\n<li>Moves everything to our main account without breaking security<\/li>\n<li>Sorts everything by time so you can find stuff later<\/li>\n<li>Works with your existing apps (zero code changes required)<\/li>\n<\/ul>\n<\/div>\n<h3>The Technical Bits<\/h3>\n<p>Your apps keep dumping logs into CloudWatch like they always have. A subscription filter sits there quietly watching every log that comes in, then forwards it to a Firehose stream. Firehose is pretty clever \u2013 it waits until it collects 5MB of logs OR 5 minutes pass (whichever happens first), then squashes everything with GZIP and ships it off to S3.<\/p>\n<div class=\"warning-box\"><strong>Architecture diagram:<\/strong> Check out the complete flow from your apps through CloudWatch, subscription filters, Firehose, and finally to centralized S3 storage.<\/div>\n<div style=\"text-align: center; margin: 20px 0;\">\n<div id=\"attachment_74647\" style=\"width: 904px\" class=\"wp-caption aligncenter\"><img aria-describedby=\"caption-attachment-74647\" decoding=\"async\" loading=\"lazy\" class=\" wp-image-74647\" src=\"https:\/\/www.tothenew.com\/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-2025-08-31-at-2.24.59\u202fPM-1024x365.png\" alt=\"CloudWatch Logs to S3 Architecture Diagram\" width=\"894\" height=\"319\" srcset=\"\/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-2025-08-31-at-2.24.59\u202fPM-1024x365.png 1024w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-2025-08-31-at-2.24.59\u202fPM-300x107.png 300w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-2025-08-31-at-2.24.59\u202fPM-768x274.png 768w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-2025-08-31-at-2.24.59\u202fPM-1536x548.png 1536w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-2025-08-31-at-2.24.59\u202fPM-2048x731.png 2048w, \/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-2025-08-31-at-2.24.59\u202fPM-624x223.png 624w\" sizes=\"(max-width: 894px) 100vw, 894px\" \/><p id=\"caption-attachment-74647\" class=\"wp-caption-text\">Architecture Diagram: Complete flow from CloudWatch to centralized S3 storage<\/p><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"container\">\n<div class=\"container\">\n<p>Yeah, there&#8217;s a 5-15 minute delay, but honestly? That&#8217;s perfect for log analysis. You get near real-time data without paying through the nose for constant small transfers.<\/p>\n<h2>Setting Up The Infrastructure<\/h2>\n<p>This is where it gets interesting. You need stuff in two places: the accounts where logs come from (we call them member accounts) and the account where everything gets stored (the central account).<\/p>\n<h3>What Goes in Each Member Account<\/h3>\n<div class=\"tech-grid\">\n<div class=\"tech-card\">\n<h4>\ud83d\udd10 IAM Roles (The Security Layer)<\/h4>\n<p>Two roles handle everything:<\/p>\n<ul>\n<li><strong>CloudWatch Logs Role:<\/strong> Can only send data to Firehose, that&#8217;s it<\/li>\n<li><strong>Firehose Role:<\/strong> Can write to the central S3 bucket plus some basic monitoring stuff<\/li>\n<\/ul>\n<p>We&#8217;re obsessive about permissions. Each role gets exactly what it needs to do its job, nothing extra.<\/p>\n<\/div>\n<div class=\"tech-card\">\n<h4>\ud83d\ude80 Kinesis Data Firehose Stream<\/h4>\n<p>This is where the magic happens. We set it up with:<\/p>\n<ul>\n<li>5MB or 300-second buffering (whichever hits first)<\/li>\n<li>GZIP compression turned on<\/li>\n<li>Error handling (failed stuff goes to a separate folder)<\/li>\n<li>CloudWatch monitoring so we know when things break<\/li>\n<\/ul>\n<\/div>\n<div class=\"tech-card\">\n<h4>\ud83d\udcca CloudWatch Components<\/h4>\n<p>The pieces that make it all work together:<\/p>\n<ul>\n<li><strong>Subscription filters:<\/strong>Gets attached to your log groups<\/li>\n<li><strong>Logs destination:<\/strong> Routes the filtered data to Firehose<\/li>\n<li><strong>Destination policy:<\/strong> Controls which AWS accounts\/roles can create subscription filters to this destination (crucial for cross-account access)<\/li>\n<\/ul>\n<\/div>\n<\/div>\n<h3>What Goes in the Central Account<\/h3>\n<p>The central account keeps things simple \u2013 just one S3 bucket with a really strict policy.<\/p>\n<div class=\"success-box\">\n<h4>\ud83e\udea3 The Central S3 Bucket<\/h4>\n<p>Our bucket policy is locked down tight:<\/p>\n<ul>\n<li>Only specific Firehose roles from member accounts can write<\/li>\n<li>All connections must use HTTPS (no exceptions)<\/li>\n<li>Versioning is turned on for data protection<\/li>\n<li>Logs get organized automatically by year\/month\/day\/hour<\/li>\n<\/ul>\n<p>No wildcards, no broad permissions \u2013 every single ARN is listed explicitly.<\/p>\n<\/div>\n<h4>How We Organize Everything<\/h4>\n<p>The time-based folder structure isn&#8217;t just pretty \u2013 it makes queries fast and cheap:<\/p>\n<pre><code>s3:\/\/our-central-logs\/\r\n    \u251c\u2500\u2500 cloudwatch-logs\/\r\n    \u2502   \u251c\u2500\u2500 year=2025\/month=01\/day=15\/hour=14\/\r\n    \u2502   \u2502   \u251c\u2500\u2500 batch-001.gz\r\n    \u2502   \u2502   \u251c\u2500\u2500 batch-002.gz\r\n    \u2502   \u2502   \u2514\u2500\u2500 batch-003.gz\r\n    \u2502   \u2514\u2500\u2500 year=2025\/month=01\/day=15\/hour=15\/\r\n    \u2502       \u2514\u2500\u2500 ...\r\n    \u2514\u2500\u2500 cloudwatch-logs-errors\/\r\n        \u2514\u2500\u2500 delivery-failures\/<\/code><\/pre>\n<p>When you need to search logs from a specific time period, AWS Athena\/Splunk only looks at the relevant folders. This saves both time and money \u2013 a lot of money if you&#8217;re dealing with terabytes of logs.<\/p>\n<h2>Security Stuff (The Important Part)<\/h2>\n<p>Cross-account anything makes security teams break out in cold sweats, so we had to nail this part.<\/p>\n<h3>Our Security Approach<\/h3>\n<p>Every role has the bare minimum permissions needed. The CloudWatch role can&#8217;t touch S3 directly, and the Firehose role can&#8217;t peek at other log groups. The central bucket policy explicitly lists which member account roles are allowed \u2013 new accounts don&#8217;t get automatic access.<\/p>\n<p>All data moves over HTTPS. We actually have a bucket policy condition that flat-out rejects any request that isn&#8217;t encrypted.<\/p>\n<div class=\"warning-box\"><strong>Heads up:<\/strong> When you add new member accounts, you need to update the central bucket policy.<\/div>\n<h3>The Trade-offs<\/h3>\n<p>Nothing&#8217;s perfect. Here&#8217;s what you should know going in:<\/p>\n<ul>\n<li>There&#8217;s a 5-15 minute delay from log creation to S3 availability<\/li>\n<li>Each Firehose stream handles about 5,000 records per second (you can run multiple streams if needed)<\/li>\n<li>This works within a single AWS region<\/li>\n<li>During really high volume periods, you might see some brief delays<\/li>\n<\/ul>\n<h2>Getting Started<\/h2>\n<p>We&#8217;ve documented everything in the GitHub repository. Seriously, everything:<\/p>\n<ul>\n<li>All the AWS CLI command you need to run to complete the setup<\/li>\n<li>All the IAM policies (just update your account IDs)<\/li>\n<li>Test scripts to verify everything&#8217;s working<\/li>\n<\/ul>\n<div class=\"success-box\">\n<p><strong>Before you start, make sure you have:<\/strong><\/p>\n<ul>\n<li>AWS CLI access to both member and central accounts<\/li>\n<li>Permission to create IAM roles and policies<\/li>\n<li>Basic understanding of CloudWatch and S3<\/li>\n<\/ul>\n<p>If you can create an S3 bucket and attach an IAM policy, you can build this.<\/p>\n<\/div>\n<div class=\"cta-section\">\n<h3>\ud83d\udd17 Implementation Repository<\/h3>\n<p><strong>GitHub Link:<\/strong> <a href=\"https:\/\/github.com\/RootUserGit\/cloudwatch-logs-exporter-cross-account-s3\" target=\"_blank\" rel=\"noopener\">cloudwatch-logs-exporter-cross-account-s3<\/a><\/p>\n<p>The whole setup takes about 15-20 minutes if you follow our guide step-by-step.<\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Introduction Look, if you&#8217;re running stuff across multiple AWS accounts \u2013 dev, staging, prod, maybe even separate accounts because your security team said so \u2013 you already know this pain. Something breaks, alarms start screaming, and suddenly you&#8217;re bouncing between six different accounts trying to figure out what the hell happened. What I&#8217;m going to [&hellip;]<\/p>\n","protected":false},"author":1811,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":85},"categories":[2348],"tags":[7904,7907,7905,7908,7906,7909],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/74599"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/1811"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=74599"}],"version-history":[{"count":9,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/74599\/revisions"}],"predecessor-version":[{"id":76506,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/74599\/revisions\/76506"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=74599"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=74599"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=74599"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}