{"id":28221,"date":"2015-10-26T14:38:23","date_gmt":"2015-10-26T09:08:23","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=28221"},"modified":"2015-10-27T13:42:57","modified_gmt":"2015-10-27T08:12:57","slug":"amazon-elasticsearch-for-centralized-log-management","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/amazon-elasticsearch-for-centralized-log-management\/","title":{"rendered":"Amazon ElasticSearch for Centralized Log Management"},"content":{"rendered":"

This blog is about using Amazon Elasticsearch with Amazon Cloudwatch. I had a use case where I have to make a setup for centralised logs, which can be done by using Amazon ElasticSearch and CloudWatch services.<\/p>\n

Pre-requisites:<\/h3>\n

New Amazon Elasticsearch Instance can be setup from here<\/a>.<\/p>\n

A new EC2 Instance with an attached role having Cloudwatch full access policy should be launched to install Cloudwatch daemon. Steps to install Cloudwatch deamon are as listed below:<\/p>\n

[js]<\/p>\n

wget <a href="https:\/\/s3.amazonaws.com\/aws-cloudwatch\/downloads\/latest\/awslogs-agent-setup.py">https:\/\/s3.amazonaws.com\/aws-cloudwatch\/downloads\/latest\/awslogs-agent-setup.py<\/a>
\nsudo python .\/awslogs-agent-setup.py –region us-east-1<\/p>\n

[\/js]<\/p>\n

 <\/p>\n

While executing above steps, a few details has to be provided. These are listed below and also shown in the next screenshot:<\/p>\n

    \n
  1. AWS region name<\/li>\n
  2. AWS output format<\/li>\n
  3. Path of log file to upload [\/var\/log\/syslog]<\/li>\n
  4. Destination Log Group name [\/var\/log\/syslog]<\/li>\n
  5. Log Stream name<\/li>\n
  6. Log Event timestamp format<\/li>\n
  7. Initial position of upload<\/li>\n<\/ol>\n

    \"blog<\/p>\n

    Now, the log streams can be seen on AWS Cloudwatch Console as:<\/p>\n

    \"blog2\"<\/p>\n

    Since, Elasticsearch instance is already up and running in the account, so, now we are going to stream the above syslogs\u00a0to Amazon Elasticsearch. We need to provide the Elasticsearch cluster name as shown in below screenshot and click next:<\/p>\n

    \"blog3\"<\/p>\n

    Now, Filters can be applied which can be referred from here<\/a>, and debugging can be done while writing the pattern using “Test Pattern”:<\/p>\n

    \"blog4\"<\/p>\n

    \"blog5\"<\/p>\n

    Now, when we click start streaming button, we get success window as shown below. This process takes some time.<\/p>\n

    \"blog6\"<\/p>\n

    Now, we simply need to click on “Kibana 3” or “Kibana4” (as shown in above screenshot) to get desired logs on Kibana UI. This UI is illustrated\u00a0in below screenshot:
    \n\"blog9\"<\/p>\n

    Kibana4<\/h5>\n

    \"blog8\"<\/p>\n

    Kibana3<\/h5>\n","protected":false},"excerpt":{"rendered":"

    This blog is about using Amazon Elasticsearch with Amazon Cloudwatch. I had a use case where I have to make a setup for centralised logs, which can be done by using Amazon ElasticSearch and CloudWatch services. Pre-requisites: New Amazon Elasticsearch Instance can be setup from here. A new EC2 Instance with an attached role having […]<\/p>\n","protected":false},"author":261,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":3},"categories":[1174,2348],"tags":[2662,2661],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/28221"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/261"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=28221"}],"version-history":[{"count":0,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/28221\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=28221"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=28221"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=28221"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}