![\"logo_elasticsearch\"](\"\/blog\/wp-ttn-blog\/uploads\/2016\/11\/logo_elasticsearch.png\"){kind=logo}
<\/p>\n<\/p>\n
One of the most challenging tasks in any microservices ecosystem is the centralized log management, and there are many open source and paid solutions available in the market. In our ecosystem, we are using ELK stack as it provides scalability and the multitenant-capable full-text search engine that easily integrates with Logstash and Kibana for centralized logging and visualizations.<\/p>\n
In the initial days, we did the setup of elasticsearch<\/a> cluster with 2 nodes and both of them were configured as master eligible data nodes to mitigate node failure. Both the nodes were using the default configuration for shards and replica i.e. shard: 5 and replica: 1. This step was perfect for handling small data set i.e. 30-40 GB logs per day, but in our case the log size increased considerably from 30-40GB per day to 200 GB per day in a matter of 2-3 months. As a result, we started facing operational issues such as low disk space, high load average, and performance issues\u00a0with the elasticsearch cluster. With this experience we realized that we were unable to leverage elasticsearch scalability to the optimum. Through this blog, I will be sharing how we can leverage spot instances in the elasticsearch cluster with default settings. It is highly recommended to tweak elasticsearch settings based on your use-case and that can be done only after thorough testing.<\/p>\n Prerequisite:<\/strong>\u00a0To implement highly scalable elasticsearch cluster for logging, you should have the basic understanding of how\u00a0ELK stack works.<\/p>\n In order to scale elasticsearch efficiently, it is recommended to have a separate master, data, and client nodes. The main purpose of all the nodes is mentioned below:<\/p>\n For more details click here<\/a><\/p>\n After going through the blog “Designing for Scale” on elastic.co. In our case – We had to refactor\u00a0Elasticsearch cluster setup as shown below:<\/p>\n In this setup, we should use private DNS endpoints, for master and client nodes. Using this setup helps us in scaling in\/out data nodes without any changes in the logstash configuration file, as we are using client private DNS endpoints in the config.\u00a0<\/span>Using all the above instance types in on-demand pricing model (master nodes: t2.medium, client nodes: m3.medium and data nodes: m3.large, m4.large, c3.xlarge and c4.xlarge) will incur good monthly AWS bill, in our case, therefore, we started playing with spot instances. The only concern before starting this activity was to ensure we have one copy of data available on the on-demand instance regularly. To achieve this, we had used cluster.routing.allocation.awareness.attributes, that helped in shard routing and allocation on different rack ids.<\/p>\n If Elasticsearch is <\/span>aware<\/span><\/i> of the physical configuration of your hardware, it can ensure that the primary shard and its replica shards are spread across different physical servers, racks, or zones, to minimize the risk of losing all shard copies at the same time.<\/span><\/p>\n \u00a0You can find the\u00a0master, data, and client node configurations below, we had used ep utility (<\/span>envplate utility<\/span><\/a>) to replace environment variables in the configuration file at the instance startup.<\/span><\/p>\n 2. Setup Master, Data, and Client nodes:<\/strong><\/span><\/p>\n Note: Running production workloads on spot instances is not recommended, as they can terminate any time AWS market price rises above your bid price.<\/strong><\/p>\n\n
![\"Scalable_Elasticsearch](\"\/blog\/wp-ttn-blog\/uploads\/2016\/11\/Scalable_Elasticsearch-5.png\")
<\/p>\n\n
\n
\n
\n
\n
\n
\n
\n
\n
\n