{"id":34841,"date":"2016-05-25T22:44:18","date_gmt":"2016-05-25T17:14:18","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=34841"},"modified":"2024-01-02T17:46:25","modified_gmt":"2024-01-02T12:16:25","slug":"tweaking-logstashs-s3-plugin-to-create-folders-in-yyyymmdd-format-on-aws-s3","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/tweaking-logstashs-s3-plugin-to-create-folders-in-yyyymmdd-format-on-aws-s3\/","title":{"rendered":"Tweaking Logstash’s S3 plugin to create folders in YYYY\/MM\/DD format on AWS S3"},"content":{"rendered":"

Logstash is a service that accepts logs from a variety of systems, processes it and allows us to index it in Elasticsearch<\/a>\u00a0etc which can be visualised using Kibana.<\/p>\n

Our DevOps\u00a0engineers\u00a0have been using<\/a> Logstash S3 plugin which simply puts all data in a S3 bucket<\/a> location. Since we have configured files to be created in every hour on S3, the number of files in the S3 location touched thousand in just one and a half month. We decided to store the files in YYYY\/MM\/DD folder structure. For example, files created on date 2016-06-16 would go inside s3:\/\/location\/2016\/06\/16. These YYYY, MM, and DD folder would get created dynamically.<\/p>\n

\"logstash-logo.png<\/p>\n

Scenario<\/strong>: Store data received by Logstash on S3 in folder structure YYYY\/MM\/DD such the each day\u2019s data would go into its respective folder.<\/p>\n

Logstash S3 plugin does not provide the functionality to pass a\u00a0variable. To achieve our objective, we decided to tweak the ruby code of S3 plugin. You can do that by following the below steps:<\/p>\n

    \n
  1. Go to your Logstash\u2019s home directory.<\/span><\/span><\/li>\n
  2. Open vendor\/bundle\/jruby\/1.9\/gems\/logstash-output-s3-1.0.2\/lib\/logstash\/outputs\/s3.rb<\/strong> file.<\/span><\/span><\/li>\n
  3. Search for string \u201cremote_filename<\/strong>\u201d. We need to tweak the below line:<\/span>
    \n<\/span>remote_filename = \"#{@prefix}#{File.basename(file)}\"<\/code><\/span><\/span><\/li>\n
  4. Create time object say \u201ct\u201d above this line and store the desired format i.e YYYY\/MM\/DD in a variable \u00a0say \u201cdate_s3\u201d as below:<\/span>
    \n<\/span><\/p>\n
    t = Time.new \r\ndate_s3 = t.strftime(\"%Y\/%m\/%d\/\")<\/pre>\n<\/li>\n
  5. Append this variable in the line specified in point 3 as below:<\/span>
    \n<\/span>remote_filename = \u201c#{@prefix}#{date_s3}#{File.basename(file)}\u201d<\/code><\/span><\/span><\/li>\n
  6. The modified code would look like this:<\/span>
    \n<\/span><\/p>\n
    t = Time.new\r\ndate_s3 = t.strftime(\u201c%Y\/%m\/%d\/\u201d)\r\nremote_filename = \u201c#{@prefix}#{date_s3}#{File.basename(file)}\u201d<\/pre>\n<\/li>\n
  7. Save the file and check configuration and restart Logstash.<\/span><\/span><\/li>\n
  8. Logstash\u2019s S3 plugin configuration in Logstash\u2019s output would be:<\/span>\n
    s3{\r\n    access_key_id => \u201cxxxxxxxxxxxxxxxxxxxxxxxxx\u201d\r\nsecret_access_key => \u201cXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\u201d\r\nendpoint_region => \u201cap-southeast-1\u2033\r\nbucket => \u201cs3-bucket-name\u201d\r\nprefix => \u201cdirectory-inside-s3\/\u201d\r\ntime_file => 60\r\ncanned_acl => \u201cprivate\u201d\r\n}<\/pre>\n
    Note<\/strong>: Do not forget to add \u201c\/\u201d in the prefix parameter.<\/p>\n
    This configuration would create date-based directory structure inside \u201cs3:\/\/s3-bucket-name\/directory-inside-s3\/<\/strong>“.<\/span><\/p>\n
    Yes, it is done now. You can now browse the files easily on S3. Hope this blog was useful to you. I will be coming up with more such interesting use cases.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"
    Logstash is a service that accepts logs from a variety of systems, processes it and allows us to index it in Elasticsearch\u00a0etc which can be visualised using Kibana. Our DevOps\u00a0engineers\u00a0have been using Logstash S3 plugin which simply puts all data in a S3 bucket location. Since we have configured files to be created in every […]<\/p>\n","protected":false},"author":154,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":31},"categories":[1174,2348,1],"tags":[248,3391,1892,3392,3389,3390,980,670],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/34841"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/154"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=34841"}],"version-history":[{"count":1,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/34841\/revisions"}],"predecessor-version":[{"id":59859,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/34841\/revisions\/59859"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=34841"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=34841"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=34841"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}