{"id":37898,"date":"2016-07-28T10:41:00","date_gmt":"2016-07-28T05:11:00","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=37898"},"modified":"2016-07-30T06:01:13","modified_gmt":"2016-07-30T00:31:13","slug":"fluentd-the-log-collector","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/fluentd-the-log-collector\/","title":{"rendered":"Fluentd – The Log Collector"},"content":{"rendered":"

<\/h1>\n

Whenever we talk about Log Analysis which is to create some sense out of the computer generated records, we always need some tools which can first collect these logs from different devices, operating systems or any applications. These tools are generally termed as Log Collectors.<\/p>\n

There are two popular log collectors – <\/span>Logstash<\/span><\/a> & Fluentd. Logstash is written in JRuby and is maintained by elastic.co. However, Fluentd is written in CRuby and is maintained by <\/span>Treasure Data Inc<\/span>.<\/span><\/p>\n

\"fluentd\"<\/p>\n

Fluentd works on Unified Logging Layer means it tries to structure logs as JSON as much as possible. The idea is to\u00a0provide an interface which can be used by almost any producer or any consumer of the\u00a0logs. This helps in all phases of log processing like Collection, Filter, and Output\/Display.<\/p>\n

Like Logstash, it also provides 300+ plugins out of which only a\u00a0few are provided by official Fluentd repo and a majority of them are maintained by individuals.<\/span><\/p>\n

It can easily be replaced with Logstash as a log collector subject to some pros and cons in existing ELK stack making it EFK stack. Unlike Logstash which works with Aggregation of logs from different sources, Fluentd works on Routing on the basis of TAGs assigned on each input.<\/span><\/p>\n

 <\/p>\n

Lifecycle of Fluentd Event:<\/strong><\/h2>\n

\"Fluentd_lifecycle\"<\/p>\n

 <\/p>\n

Basic Directives Involved<\/strong><\/h2>\n

source<\/strong>: <\/strong>\u00a0This mandatory directive is required to define the type parameter which specifies which input plugin to use:<\/span><\/p>\n

\"source\"<\/p>\n

match: <\/b>\u201cdecide what to do !\u201d \u00a0Each match directive must include a match pattern and a type parameter. Events with a tag matching the pattern will be picked and sent to the output destination. The destination could be Elasticsearch, S3, Mongo or to Treasure Data directly:<\/span><\/p>\n

\"match\"<\/p>\n

filter: \u00a0<\/b>It has same the syntax as \u201cmatch\u201d but \u201cfilter\u201d could be used for further processing before pushing it to output. Using filters, event flow is like below:<\/span><\/p>\n

Input -> filter 1 -> … -> filter N -> Output<\/span><\/h2>\n

\u00a0\"filter\"<\/strong><\/strong><\/p>\n

In my case, I wanted to forward all Nginx access log to Elasticsearch, I used below configuration using tag \u2018nginx.access\u2019:<\/p>\n

\"input_tag\"<\/p>\n

\"output_match\"<\/p>\n

There are multiple pre-defined <\/span>recipes<\/span><\/a> available on Fluentd which you can utilize. Like Logstash, Fluentd also makes use of Regex patterns for the logs whose format is not known or is not already available with Fluentd. Those patterns can be verified on <\/span>Fluentular<\/span><\/a>. <\/span><\/p>\n

Unlike Logstash, which can only be configured as Active-Standby, Fluentd can be configured as Active-Active (Load Balancing mode), Active-Standby mode, Weighted Load Balancing modes.<\/p>\n

\"Fluentd_active-active\"<\/p>\n

Since Fluentd was invented by Treasure Data Inc, TD also provides Fluentd in <\/span>td-agent<\/span><\/a> form which is a\u00a0more stable distribution of Fluentd. It supports multiple Installation medium and comes with preconfigured recommended settings.<\/span><\/p>\n

So when a Simple, Flexible, Reliable Unified Logging tool is required, you can directly choose Fluentd.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

Whenever we talk about Log Analysis which is to create some sense out of the computer generated records, we always need some tools which can first collect these logs from different devices, operating systems or any applications. These tools are generally termed as Log Collectors. There are two popular log collectors – Logstash & Fluentd. […]<\/p>\n","protected":false},"author":181,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":9},"categories":[2348,1],"tags":[3802,3807,3808,3809,1973,3806,3803,3805,3804],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/37898"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/181"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=37898"}],"version-history":[{"count":0,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/37898\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=37898"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=37898"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=37898"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}