Amazon ElasticSearch for Centralized Log Management

26 / Oct / 2015 by Aakash Garg 1 comments

This blog is about using Amazon Elasticsearch with Amazon Cloudwatch. I had a use case where I have to make a setup for centralised logs, which can be done by using Amazon ElasticSearch and CloudWatch services.

Pre-requisites:

New Amazon Elasticsearch Instance can be setup from here.

A new EC2 Instance with an attached role having Cloudwatch full access policy should be launched to install Cloudwatch daemon. Steps to install Cloudwatch deamon are as listed below:

[js]

wget <a href="https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py">https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py</a>
sudo python ./awslogs-agent-setup.py –region us-east-1

[/js]

 

While executing above steps, a few details has to be provided. These are listed below and also shown in the next screenshot:

  1. AWS region name
  2. AWS output format
  3. Path of log file to upload [/var/log/syslog]
  4. Destination Log Group name [/var/log/syslog]
  5. Log Stream name
  6. Log Event timestamp format
  7. Initial position of upload

blog 1

Now, the log streams can be seen on AWS Cloudwatch Console as:

blog2

Since, Elasticsearch instance is already up and running in the account, so, now we are going to stream the above syslogs to Amazon Elasticsearch. We need to provide the Elasticsearch cluster name as shown in below screenshot and click next:

blog3

Now, Filters can be applied which can be referred from here, and debugging can be done while writing the pattern using “Test Pattern”:

blog4

blog5

Now, when we click start streaming button, we get success window as shown below. This process takes some time.

blog6

Now, we simply need to click on “Kibana 3” or “Kibana4” (as shown in above screenshot) to get desired logs on Kibana UI. This UI is illustrated in below screenshot:
blog9

Kibana4

blog8

Kibana3
FOUND THIS USEFUL? SHARE IT

comments (1 “Amazon ElasticSearch for Centralized Log Management”)

Leave a Reply

Your email address will not be published. Required fields are marked *