Introduction

Server monitoring is one of the crucial task of the overall Infrastructure monitoring and observability whether it is a production, staging or development environment and on the key factor of server monitoring is the disk that has been attached to that server as it contains most of the crucial logs that may play a key role while debugging or while facing any server related issue but these logs may cause the performance issue and the storage issue if these logs are not managed properly and without some automated way for log rotation these logs can consume high disk usage and my cause in loosing of some critical logs.

So, In this blog we will be looking how we can rotate these logs and moving these logs over the S3 bucket that can further be rotated using lifecycle rules on S3 bucket.

Problem Statement

Most of us have gone through the problem where servers crash unexpectedly or they become unresponsive because of the high disk utilization which is caused by unmanaged logs. And these logs increase with the time

So we need a automated way to rotate these logs so that we can prevent high disk utilization which result in

• High Latency
• Failed Deployments
• Server Unresponsiveness

Prerequisites

• • AWS Linux server
  • S3 bucket where the logs will be moved
  • Basic knowledge of AWS CLI, Linux command line and Bash Scripting.

Log Management

Step 1 – Create a S3 bucket.

First, we need to create a S3 bucket, where we will be moving the logs from our EC2 server. Also, we will be applying the lifecycle policy on it for log rotation.

Create Bucket

Create a lifecycle policy for rotating the logs, like below –

Step 2 – IAM Role and Policy creation

So, For moving the logs from the server to S3 bucket we will be needing a role that will be used by server to move the logs-

Example of AWS IAM Policy

So, after creating a role. Add the above policy to it and add the role over the server.

Step 4 – Install AWS CLI on server.

Use below command to install AWS CLI over the server-

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

For other OS, you can refer this official guide of AWS  – https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

Step 5 – Log transfer script

Below is sample script that we would be using to move logs from server to S3-

We need to give executable permission to the above sample script, use below command for it-

sudo chmod +x logs-to-s3.sh

Step 6 – Setup Cron Job

use below commands to setup cron job-

sudo crontab -e

0 1 * * * /opt/logs-to-s3.sh

Conclusion

Finally, we have setup an automated way to rotate the logs and move the logs to s3 bucket with the lifecycle rule implemented. You can also create or modify the sample script give above in the blog as per the requirements. The cronjob will make sure that the script runs after a fixed interval of time and the lifecycle rules that has been implemented over the bucket will make sure that the objects(Logs) retire after a fixed interval of time.