{"id":19845,"date":"2015-05-24T01:00:04","date_gmt":"2015-05-23T19:30:04","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=19845"},"modified":"2016-12-19T15:07:34","modified_gmt":"2016-12-19T09:37:34","slug":"autoscaling-environment-log-collection-using-logentries","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/autoscaling-environment-log-collection-using-logentries\/","title":{"rendered":"Autoscaling Environment Logs Collection using LogEntries"},"content":{"rendered":"

LogEntries<\/span><\/h3>\n

Managing log data across large autoscaling environments can be a time-consuming and expensive job. Logentries has designed a scalable service that dynamically supports autoscaling environment as log volumes expand and change dynamically. By centralizing all logs across distributed instances into one secure location, we can efficiently monitor and track log events in real time, without complex configurations i.e.
\n<\/strong><\/p>\n

autoscaling environment logs collection using LogEntries.<\/span><\/h3>\n

Use Case<\/span><\/h3>\n

We have instances brought up as cloned instances in our production environment. In our case, we can use the Agent and <\/span>it’s configuration file as this would enable us to continue logging to Logentries and continue logging to the same logs without creating any orphaned logs.<\/span><\/p>\n

Config File<\/span><\/h3>\n

The agent stores configuration in ~\/.le\/config for ordinary users and in \/etc\/le\/config for root (daemon). It is created with init or reinit commands and can be created or modified manually.<\/p>\n

There are two ways to follow logs:<\/span><\/h3>\n

1)Follow log files through server-side configuration
\n2)Follow log files through your configuration file<\/span><\/p>\n

We would be using the second way & the configure file will be like this:<\/p>\n

[js][Main]
\nuser-key = ACCOUNT-KEY
\npull-server-side-config = False<\/p>\n

[Syslog]
\npath = \/var\/log\/syslog
\ndestination = Production\/Syslog<\/p>\n

[Auth]
\npath = \/var\/log\/auth.log
\ndestination = Production\/Auth<\/p>\n

[Boot]
\npath = \/var\/log\/boot.log
\ndestination = Production\/Boot[\/js]<\/p>\n

Simply paste this into the file \/etc\/le\/config.<\/p>\n

This configuration tells the Agent to follow Syslog, auth.log and boot.log and to send their contents to a Log set called Production, each log will be sent to a log inside that log set. If the destination exists then we reuse the token that exists for that Log in Logentries, if it does not exist then the Agent will create a new Log set called Production and a new log.<\/span><\/p>\n

If we\u00a0use the config as described below and reuse it across your environments then it will not create a new log<\/span> set. If we want to have three Log<\/span> Sets in our Account such as Production, Staging, Development then you would need three separate configuration files that would be used on each of those environments.<\/span><\/p>\n

*For example if we had three Production servers we could use the same Production configuration file (like below) in all three of those servers. The Agent will tag each log\u00a0<\/span>message with the hostname so you will be able to identify where the logs originated from as well.<\/span><\/p>\n

Installing LogEntries<\/span><\/h3>\n

On an Ubuntu instance our installation commands would be the following:<\/p>\n

[js]#!\/bin\/bash
\nsudo echo ‘deb http:\/\/rep.logentries.com\/ trusty main’ &gt; \/etc\/apt\/sources.list.d\/logentries.list
\nsudo gpg –keyserver pgp.mit.edu –recv-keys C43C79AD &amp;&amp; gpg -a –export C43C79AD | apt-key add –
\nsudo apt-get update
\nsudo apt-get install logentries -y
\nsudo mkdir \/etc\/le
\ncd \/etc\/le\/
\nsudo wget
\nsudo apt-get install logentries-daemon[\/js]<\/p>\n

* We are fetching the config file from s3 bucket “logentries”<\/p>\n

We will get the logs in our LogEntries account as follows:
\n\"LE\"<\/a><\/p>\n

Autoscaling the setup<\/span><\/h3>\n

We will make AMI of the above set up the server and we can use it to form a launch configuration for an autoscaling environment. Now the servers coming up in the autoscaling environment will be using the same config file and environment and thus will be sending their logs to the desired destination in our log entries account.<\/p>\n

This is how the LogEntries screen will look like, where we can further do the following:<\/p>\n

– Tagging the logs
\n– SettinG Alerts on Logs including inactivity alert
\n– Graphical representation and analysis of logs
\n– Performing search on logs using Regular Expressions<\/p>\n","protected":false},"excerpt":{"rendered":"

LogEntries Managing log data across large autoscaling environments can be a time-consuming and expensive job. Logentries has designed a scalable service that dynamically supports autoscaling environment as log volumes expand and change dynamically. By centralizing all logs across distributed instances into one secure location, we can efficiently monitor and track log events in real time, […]<\/p>\n","protected":false},"author":166,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":2},"categories":[1174],"tags":[1217,1784,1787,561,1783,1782,1785,1786],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/19845"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/166"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=19845"}],"version-history":[{"count":0,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/19845\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=19845"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=19845"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=19845"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}