{"id":43191,"date":"2016-12-15T15:50:38","date_gmt":"2016-12-15T10:20:38","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=43191"},"modified":"2016-12-15T15:50:38","modified_gmt":"2016-12-15T10:20:38","slug":"how-to-reduce-the-number-of-druid-segments","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/how-to-reduce-the-number-of-druid-segments\/","title":{"rendered":"How to Reduce the Number of Druid Segments?"},"content":{"rendered":"

We have been using Druid<\/a> in our project for a while and had shared our experiences during GIDS<\/a>.\u00a0It has given us great results\u00a0as it powers our real-time dashboards, reports on\u00a0running Ad campaigns and provides real-time data to make quick decisions. But like any other databases there will be times when it needs to be optimized. This is similar to how we may optimize various things in MySQL, Oracle or any other database.<\/p>\n

The Problem<\/h3>\n

Druid stores all of its data\u00a0sharded by time, data sources and optionally by other dimensions. In Druid terminology this sharded data is called segments<\/strong>. These segments\u00a0are\u00a0served by Druid’s\u00a0historical node. For serving all of these segments\u00a0the historical node needs to open\u00a0file handlers to all\u00a0the segments. The problem occurs as the\u00a0amount of data increases then the number of memory mapped files\u00a0will also keep on increasing and there is a limit on the\u00a0memory mapped files on Linux.<\/p>\n

We had reached a point where we had more than 140k+ segments which was pushing towards this limit although we had distributed the data across historical nodes in the cluster.\u00a0So we had to somehow reduce the number of segments on each node or the overall of segments in our Druid. Less number of segments has other benefits also as there would be less references in zookeeper and the coordinator node of Druid will have to balance less\u00a0number of segments.<\/p>\n

Initial Solutions Considered<\/h3>\n