{"id":35041,"date":"2016-05-30T23:27:15","date_gmt":"2016-05-30T17:57:15","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=35041"},"modified":"2016-06-10T10:40:35","modified_gmt":"2016-06-10T05:10:35","slug":"running-curator-in-docker-container-to-remove-old-elasticsearch-indexes","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/running-curator-in-docker-container-to-remove-old-elasticsearch-indexes\/","title":{"rendered":"Running Curator in Docker container to remove old Elasticsearch indexes"},"content":{"rendered":"

We have been using ELK as a centralized logs management system. ELK stands for Elasticsearch, Logstash<\/a>, and Kibana. Each of the three services is running inside its own docker container in one docker-network (say elk-net) on a single server. A separate block storage device is mapped from the\u00a0host into Elasticsearch container (named es) as a data directory.<\/p>\n

\"1-q6CHtsWI54-BkYGGAGBTZQ\"<\/p>\n

Since ELK creates one index each day into Elasticsearch, we wanted to keep only previous 15 days\u2019 indexes on the system and remove\u00a0the older ones after taking their backup on AWS S3 service. Elasticsearch allows us to take backup on s3 so we configured a cronjob which daily takes incremental backups on the indexes.<\/p>\n

Curator is a good tool to remove the older indexes based on the dates. The problem was that the curator needs IP address of the Elasticsearch container and it is not a good practice to use container\u2019s IP as this may change later if we play around with the container. So, we came up with an idea of running a curator inside a container in the same docker-network in which Elasticsearch container is running.<\/p>\n

Scenario:<\/strong> Remove indexes of Elasticsearch older than fifteen days using a curator where Elasticsearch is running inside a docker container without using IP address of Elasticsearch container.
\nThe following steps can be followed to implement this scenario:<\/p>\n

    \n
  1. Create a docker image which contains the curator binary. We have created a public docker images navjotsingh\/curator using ubuntu:14.04 image.<\/li>\n
  2. Create a container using this image as below:\n

    [js]docker run –rm –name curator –net \u201celk-net\u201d –entrypoint curator navjotsingh\/curator –host es delete indices –older-than 15 –time-unit days –timestring ‘%Y.%m.%d'[\/js]<\/p>\n

    We have run a container name \u201ccurator\u201d using docker image \u201cnavjotsingh\/curator\u201d in docker network \u201celk-net\u201d with –rm option. This option removes the container only the container exits.
    \nWe have used curator command as entry point and asked it to delete indices of format \u201c%Y.%m.%d\u201d which are older than 15 days .<\/li>\n

  3. Schedule this container to run every night. So every night this container will get created and will destroy itself after removing older indexes.<\/li>\n<\/ol>\n

    This is one typical use-case of docker where we are using<\/a> it as a curator binary and is based on docker philosophy of “one process per container”.<\/p>\n","protected":false},"excerpt":{"rendered":"

    We have been using ELK as a centralized logs management system. ELK stands for Elasticsearch, Logstash, and Kibana. Each of the three services is running inside its own docker container in one docker-network (say elk-net) on a single server. A separate block storage device is mapped from the\u00a0host into Elasticsearch container (named es) as a […]<\/p>\n","protected":false},"author":154,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":21},"categories":[2348,1],"tags":[3409,3410,1892,1883,2758,3407,3389,3408],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/35041"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/154"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=35041"}],"version-history":[{"count":0,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/35041\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=35041"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=35041"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=35041"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}