![\"Flume](\"\/blog\/wp-ttn-blog\/uploads\/2015\/02\/Screenshot-from-2015-02-27-1519441.png\")
<\/a><\/p>\nSources could be anything like an Avro Client being used by Log4JAppender as well.<\/p>\n
It is worth noting, the above given configuration will produce separate files of 10 records each by default, taking timestamp ( an interceptor <\/strong><\/em>) as the point of reference of last update of the source.<\/p>\nThe file so created be saved as flume.conf, and can be run as :<\/p>\n
flume-ng agent -f flume.conf -n source_agent<\/strong><\/em><\/p>\nPerformance Measures, Issues and Comments:<\/strong><\/em><\/p>\n\n- Memory leaks in log4jappender<\/strong> when ingesting just 10000 records.\n
\n- Solved by making thread sleep after every 500 records, thus decreasing the load on the channel<\/li>\n<\/ul>\n<\/li>\n
- \u00a0GC Memory Leak (Flume level)\n
\n- Solved by keeping transactionCapacity of channel low and capacity of channel high enough<\/li>\n<\/ul>\n<\/li>\n
- Avro’s Optimized data serialization is not expolited when using log4jappender\n
\n- Solved by directly pushing the file onto avro-client (inbuilt in flume-ng)<\/li>\n<\/ul>\n<\/li>\n
- Due to memory leaks, could not write more than 12000 records from log4jappender\n
\n- Solved in points 1 & 2<\/li>\n<\/ul>\n<\/li>\n
- Did performance analysis of data ingestion directly from file as well as avro client of 14000 records when only 10 records rolled up per file in HDFS\n
\n- Using Avro Client\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0 –> Total time – 00:02:48<\/strong><\/li>\n
- Directly from File System –> Total time – 00:02:44<\/strong><\/li>\n<\/ul>\n<\/li>\n
- When using avro client, realtime update to the file was not taken into account\n
\n- Unsolved problem<\/li>\n<\/ul>\n<\/li>\n
- When ingesting directly from file, realtime updates were automatically registered, on the basis of timestamp for last modification\n
\n- Used the following configuration
\nsource_agent.sources.test_source.interceptors = itime
\n# http:\/\/flume.apache.org\/FlumeUserGuide.html#timestamp-interceptor
\nsource_agent.sources.test_source.interceptors.itime.type = timestamp<\/li>\n<\/ul>\n<\/li>\n - Flume could only create files to write at max 10 records in a single file by default. This decreased the ingestion rate, thus increasing the ingestion time\n
\n- Solved by changing the rollCount property of the HDFSSink to the desired number of events\/records per file; this increased the ingestion rate, thus decreasing the ingestion time.<\/li>\n
- Also made rollSize and rollInterval as 0, so that they are not used.<\/li>\n<\/ul>\n<\/li>\n
- While controlling the channel capacity, encountered Memory Leaks\n
\n- Solved by changing the rollCount(HDFSSink) and transactionCapacity(channel) so that the channel is cleared for more data<\/li>\n<\/ul>\n<\/li>\n
- Did performance analysis of data ingestion directly from file after applying solutions from 8 & 9 of 20000 records\n
\n- Total time – 00:00:04 (<\/strong>Previously around 00:02:30)<\/strong><\/li>\n<\/ul>\n<\/li>\n
- How to trigger notification on HDFS update from flume\n
\n- Can be potentially solved by Oozie Coordinator<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n
<\/p>\n
This covers the very basics of setting up Flume and mitigating some of the common issues which one encounters while using it.<\/p>\n","protected":false},"excerpt":{"rendered":"
Apache Flume is an open source project aimed at providing a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large volume of data. It is a complex task when moving data in large volume. We try to minimize the latency in transfer; this is achieved by specifically tweaking the configuration of Flume. […]<\/p>\n","protected":false},"author":158,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":8},"categories":[1395],"tags":[],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/17637"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/158"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=17637"}],"version-history":[{"count":1,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/17637\/revisions"}],"predecessor-version":[{"id":59899,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/17637\/revisions\/59899"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=17637"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=17637"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=17637"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"Flume](\"\/blog\/wp-ttn-blog\/uploads\/2015\/02\/Screenshot-from-2015-02-27-151944.png\")
<\/a><\/li>\n![\"\"](\"\/blog\/wp-ttn-blog\/uploads\/2024\/01\/DevGuide_image00.png\")
\n