{"id":41269,"date":"2016-10-05T11:22:26","date_gmt":"2016-10-05T05:52:26","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=41269"},"modified":"2016-10-05T11:22:26","modified_gmt":"2016-10-05T05:52:26","slug":"collecting-tomcat-logs-using-fluentd-and-elasticsearch","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/collecting-tomcat-logs-using-fluentd-and-elasticsearch\/","title":{"rendered":"Collecting Tomcat logs using Fluentd and Elasticsearch"},"content":{"rendered":"<p>In our previous <a title=\"Fluentd\" href=\"http:\/\/www.tothenew.com\/blog\/fluentd-the-log-collector\/\">blog<\/a>, we have covered the basics of fluentd, the lifecycle of fluentd\u00a0events and the\u00a0primary\u00a0directives involved. In this blog, we&#8217;ll configure fluentd\u00a0to dump tomcat logs to Elasticsearch. We&#8217;ll also\u00a0talk about filter directive\/plugin and how to configure it to add hostname field in the event stream.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"\" src=\"\/blog\/wp-ttn-blog\/uploads\/2016\/07\/fluentd.png\" alt=\"fluentd\" width=\"184\" height=\"190\" \/><\/p>\n<p>Fluentd is a log collector that works on Unified Logging Layer. It collects logs from various sources and uploads them to datastores. Fluentd reads the log file and forwards data as an event stream to either some datastore or fluentd aggregator that in turn send logs to datastore. In our use-case, we&#8217;ll forward logs directly to our datastore i.e. Elasticsearch. Elasticsearch is a search server that stores data in schema-free JSON documents.<\/p>\n<p><strong>So, let&#8217;s get started.<\/strong><\/p>\n<p>First, we need to install td-agent on the application server. Td-agent is the stable distribution of fluentd provided by Treasure data. To install td-agent on Ubuntu run the following command:<\/p>\n<p>[js]<\/p>\n<p>curl -L https:\/\/toolbelt.treasuredata.com\/sh\/install-ubuntu-trusty-td-agent2.sh | sh<\/p>\n<p>[\/js]<\/p>\n<p>Next, we need to install Elasticsearch plugin for td-agent that provides td-agent the ability to forward events to Elasticsearch.<\/p>\n<p>Run the below given command to install Elasticsearch plugin in td-agent<\/p>\n<p>[js]<\/p>\n<p>\/usr\/sbin\/td-agent-gem install fluent-plugin-elasticsearch<\/p>\n<p>[\/js]<\/p>\n<p>Now that everything is installed, it&#8217;s time to jump into the td-agent configuration to forward logs to Elasticsearch.<\/p>\n<p>Configuration file of td-agent is located at \/etc\/td-agent\/td-agent.conf. In our configuration, we&#8217;ll define three blocks\/directives\/plugins:<\/p>\n<ul>\n<li><strong>source<\/strong>: It defines the input source of events.<\/li>\n<\/ul>\n<p>[js]<br \/>\n&lt;source&gt;<br \/>\ntype tail<br \/>\nformat multiline<br \/>\nformat_firstline \/[0-9]{2}-[A-Za-z]{3}-[0-9]{4}\/<br \/>\nformat1 \/^(?&lt;datetime&gt;[0-9]{2}-[A-Za-z]{3}-[0-9]{4} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}) (?&lt;Log-Level&gt;[A-Z]*) (?&lt;message&gt;.*)$\/<br \/>\npath \/opt\/apache-tomcat-8.0.33\/logs\/catalina.out<br \/>\ntag tomcat.logs<br \/>\n&lt;\/source&gt;<br \/>\n[\/js]<\/p>\n<p style=\"padding-left: 30px\">Parameters defined inside source directive:<\/p>\n<p style=\"padding-left: 60px\"><strong>type<\/strong>: This\u00a0defines the type of input plugin used. Tail plugin reads events by tailing the text file.<br \/>\n<strong>format<\/strong>: It explains\u00a0how to read events from the text file. In our configuration, it says multi-line which means single event can contain multiple lines. Multi-line is useful in collecting stacktrace as a single event in tomcat log.<br \/>\n<strong>format_firstline<\/strong>: It defines the first line of an event using the regular expression. In our configuration, the regular expression corresponds to the date which means a log event starts with the date.<br \/>\n<strong>format1<\/strong>: It defines the fields and full format of the single log event using the regular expression. In our configuration, we have created three fields: datetime, Log-Level, and message.<br \/>\n<strong>path<\/strong>: This defines the path of the log file<br \/>\n<strong>tag<\/strong>: This is used to provide a unique name to log events. Tags are used in routing log events to datastore.<\/p>\n<ul>\n<li><strong>filter<\/strong>: filter directive enables us to modify event stream.<\/li>\n<\/ul>\n<p>[js]<br \/>\n&lt;filter tomcat.logs&gt;<br \/>\ntype record_transformer<br \/>\n&lt;record&gt;<br \/>\nhostname ${hostname}<br \/>\n&lt;\/record&gt;<br \/>\n&lt;\/filter&gt;<br \/>\n[\/js]<\/p>\n<p style=\"padding-left: 30px\">Parameters defined\u00a0in filter directive:<\/p>\n<p style=\"padding-left: 60px\"><strong>type<\/strong>: This specifies\u00a0the type of filter plugin. type record_tranformer is used to manipulate the incoming event stream.<br \/>\n<strong>record<\/strong>: The record block refers to the whole record event.<br \/>\n<strong>hostname<\/strong>: It defines the name of the field to be added to the record.<\/p>\n<ul>\n<li><strong>match<\/strong>: match directives finds the event stream with matching tags and processes them. In our configuration, we have used match directive to forward logs to the Elasticsearch.<\/li>\n<\/ul>\n<p>[js]<br \/>\n&lt;match tomcat.logs&gt;<br \/>\ntype elasticsearch<br \/>\nhost\u00a0demo.elasticsearch.tothenew.com<br \/>\nport 9200<br \/>\nlogstash_format true<br \/>\nlogstash_prefix tomcat.logs<br \/>\nflush_interval 1s<br \/>\n&lt;\/match&gt;<br \/>\n[\/js]<\/p>\n<p style=\"padding-left: 30px\">Parameters defined in match directives:<\/p>\n<p style=\"padding-left: 60px\"><strong>type<\/strong>: This represents\u00a0the type of output plugin. In our case, it&#8217;s Elasticsearch.<br \/>\n<strong>host<\/strong>: It specifies\u00a0the IP or domain name of the Elasticsearch.<br \/>\n<strong>port<\/strong>: It specifies\u00a0port of the Elasticsearch server.<br \/>\n<strong>logstash_format<\/strong>: Use\u00a0logstash format for storing events on Elasticsearch.<br \/>\n<strong>logstash_prefix<\/strong>: It defines prefix of the index name on which events will be stored.<br \/>\n<strong>flush_interval<\/strong>: \u00a0This sets\u00a0the time after which the data will be flushed to the Elasticsearch.<\/p>\n<p>Now, start td-agent using the following command:<\/p>\n<p>[js]<\/p>\n<p>service td-agent start<\/p>\n<p>[\/js]<\/p>\n<p>So, this is how you configure centralized logging using Fluentd and Elasticsearch.<\/p>\n<p>Hope this blog was useful.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In our previous blog, we have covered the basics of fluentd, the lifecycle of fluentd\u00a0events and the\u00a0primary\u00a0directives involved. In this blog, we&#8217;ll configure fluentd\u00a0to dump tomcat logs to Elasticsearch. We&#8217;ll also\u00a0talk about filter directive\/plugin and how to configure it to add hostname field in the event stream. Fluentd is a log collector that works on [&hellip;]<\/p>\n","protected":false},"author":918,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":117},"categories":[1],"tags":[1784,1524,3802,4111,4109,4108,3805,288,4106,4110,4107],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/41269"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/918"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=41269"}],"version-history":[{"count":0,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/41269\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=41269"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=41269"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=41269"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}