{"id":72649,"date":"2025-06-13T11:35:40","date_gmt":"2025-06-13T06:05:40","guid":{"rendered":"https:\/\/www.tothenew.com\/blog\/?p=72649"},"modified":"2025-09-03T19:52:02","modified_gmt":"2025-09-03T14:22:02","slug":"automating-server-log-mangement-moving-logs-to-s3-bucket","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/automating-server-log-mangement-moving-logs-to-s3-bucket\/","title":{"rendered":"Automating Server log mangement: Offloading logs to S3 bucket."},"content":{"rendered":"

Introduction<\/strong><\/span><\/h1>\n

Server monitoring is one of the crucial task of the overall Infrastructure monitoring and observability whether it is a production, staging or development environment and on the key factor of server monitoring is the disk that has been attached to that server as it contains most of the crucial logs that may play a key role while debugging or while facing any server related issue but these logs may cause the performance issue and the storage issue if these logs are not managed properly and without some automated way for log rotation these logs can consume high disk usage and my cause in loosing of some critical logs.<\/span><\/p>\n

So, In this blog we will be looking how we can rotate these logs and moving these logs over the S3 bucket that can further be rotated using lifecycle rules on S3 bucket.<\/p>\n

Problem Statement<\/strong><\/p>\n

Most of us have gone through the problem where servers crash unexpectedly or they become unresponsive because of the high disk utilization which is caused by unmanaged logs. And these logs increase with the time<\/p>\n

So we need a automated way to rotate these logs so that we can prevent high disk utilization which result in<\/span><\/p>\n

    \n
  • High Latency<\/span><\/li>\n
  • Failed Deployments<\/span><\/li>\n
  • Server Unresponsiveness<\/span><\/li>\n<\/ul>\n

    \"Image\"<\/p>\n

    Prerequisites<\/strong><\/span><\/p>\n

      \n
    • \n
        \n
      • AWS Linux server<\/span><\/li>\n
      • S3 bucket where the logs will be moved<\/span><\/li>\n
      • Basic knowledge of AWS CLI, Linux command line and Bash Scripting.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
        \"Log

        Log Management<\/p><\/div>\n

        Step 1 – Create a S3 bucket.<\/strong><\/span><\/h1>\n

        First, we need to create a S3 bucket, where we will be moving the logs from our EC2 server. Also, we will be applying the lifecycle policy on it for log rotation.<\/span><\/p>\n

        \"Create

        Create Bucket<\/span><\/p><\/div>\n

        \"Bucket\"<\/span><\/p>\n

        \"Bucket2\"<\/span><\/p>\n


        \n<\/strong>Create a lifecycle policy for rotating the logs, like below –<\/span><\/p>\n

         <\/p>\n

        \"Bucket3\"<\/span><\/p>\n

        \"lifecycle<\/span><\/p>\n

        Step 2 – IAM Role and Policy creation<\/strong><\/span><\/h1>\n

        So, For moving the logs from the server to S3 bucket we will be needing a role that will be used by server to move the logs-<\/span><\/p>\n

        Example of AWS IAM Policy<\/strong><\/span><\/h2>\n

        \"Example\"<\/p>\n

        So, after creating a role. Add the above policy to it and add the role over the server.<\/span><\/p>\n

        Step 4 – Install AWS CLI on server.
        \n<\/span><\/h1>\n

        Use below command to install AWS CLI over the server-<\/span><\/p>\n

        curl \"https:\/\/awscli.amazonaws.com\/awscli-exe-linux-x86_64.zip\" -o \"awscliv2.zip\"\r\nunzip awscliv2.zip\r\nsudo .\/aws\/install<\/span><\/pre>\n
        For other OS, you can\u00a0refer\u00a0this official\u00a0guide\u00a0of AWS\u00a0 – https:\/\/docs.aws.amazon.com\/cli\/latest\/userguide\/getting-started-install.html<\/span><\/p>\n
        Step 5 – Log transfer script<\/span><\/h1>\n
        Below is sample script that we would be using to move logs from server to S3-<\/span><\/p>\n
        \"ag\"<\/p>\n
         <\/p>\n
        We need to give executable permission to the above sample script, use below command for it-<\/p>\n
        sudo chmod +x logs-to-s3.sh<\/span><\/pre>\n
        Step 6 – Setup Cron Job<\/span><\/h1>\n
        use below commands to setup cron job-<\/span><\/p>\n
        sudo crontab -e\r\n\r\n0 1 * * * \/opt\/logs-to-s3.sh<\/span><\/pre>\n
        Conclusion<\/span><\/h1>\n
        Finally, we have setup an automated way to rotate the logs and move the logs to s3 bucket with the lifecycle rule implemented. You can also create or modify the sample script give above in the blog as per the requirements. The cronjob will make sure that the script runs after a fixed interval of time and the lifecycle rules that has been implemented over the bucket will make sure that the objects(Logs) retire after a fixed interval of time.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
        Introduction Server monitoring is one of the crucial task of the overall Infrastructure monitoring and observability whether it is a production, staging or development environment and on the key factor of server monitoring is the disk that has been attached to that server as it contains most of the crucial logs that may play a […]<\/p>\n","protected":false},"author":1873,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":31},"categories":[5877],"tags":[248,1782,670,7554],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/72649"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/1873"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=72649"}],"version-history":[{"count":18,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/72649\/revisions"}],"predecessor-version":[{"id":75244,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/72649\/revisions\/75244"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=72649"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=72649"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=72649"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}