In one of my project assignments I needed to insert large number of records into the database. I had to read the objects from an external source. Once I read all of the objects into a List, I iterated the list to save each one of them individually. In the beginning the process carried on fine but as the time passed the execution slowed down considerably. It almost took one second to insert one object into the database. Imagine the time it would have taken to insert 50000 records with this pace. Besides, many times it threw OutOfMemoryException. The code I had written did something like this :<\/p>\n
(0..60000).each{\r\n Person person = new Person(.....)\r\n person.save()\r\n }<\/pre>\n One of the solutions that I found was to use transactions and save the objects in batches, each transaction saving a batch of objects. It worked well and reduced the execution time considerably. What I did was like this :<\/p>\n
def startTime = System.nanoTime()\r\n List batch =[]\r\n (0..50000).each{\r\n Person person= new Person(....)\r\n batch.add(person)\r\n println \"Created:::::\"+it\r\n if(batch.size()>1000){\r\n Person.withTransaction{\r\n for(Person p in batch){\r\n p.save()\r\n }\r\n }\r\n }\r\n batch.clear()\r\n session = sessionFactory.getCurrentSession()\r\n session.clear()\r\n }\r\n def endTime = System.nanoTime()\r\n def diff = (startTime-endTime)\/1000000000\r\n println \"TIME TAKEN IS :::\"+diff<\/pre>\n In the previous case the time take to save 50,000 records was around 500 seconds<\/em>. But, here time taken to save the same number of records came out to be just 80 seconds<\/em>.<\/p>\n But there is one flaw in the method. If the objects are bulky, even this method would not work. Each action in a Grails Controller is executed within a Hibernate Session. The session is started right before the action starts and is closed once it returns. Thus Hibernate caches all the newly inserted Person instances in the session-level cache. As the number of objects grows, the session becomes bulkier, which slows down whole process .That also explains the reason for the memory issue, OutOfMemoryException<\/em>, because all the objects are being cached to the Hibernate session.The solution to this problem is to clear the session regularly so as to keep it light throughout the process. All that needs to be done is to get hold of the current session and clear it after each batch has been written to the database. To do this just inject SessionFactory object into your controller, get the current session object and then clear this current session.<\/p>\n\r\n def startTime = System.nanoTime()\r\n List batch =[]\r\n (0..50000).each{\r\n Person person= new Person(....)\r\n batch.add(person)\r\n println \"Created:::::\"+it\r\n if(batch.size()>1000){\r\n Person.withTransaction{\r\n for(Person p in batch){\r\n p.save()\r\n }\r\n }\r\n batch.clear()\r\n }\r\n session = sessionFactory.getCurrentSession()\r\n session.clear() \r\n }\r\n def endTime = System.nanoTime() \r\n def diff = (startTime-endTime)\/1000000000\r\n println \"TIME TAKEN IS :::\"+diff\r\n<\/pre>\nThank you,
\nImran Mir,
\nimran@intelligrape.com<\/p>\n","protected":false},"excerpt":{"rendered":"
In one of my project assignments I needed to insert large number of records into the database. I had to read the objects from an external source. Once I read all of the objects into a List, I iterated the list to save each one of them individually. In the beginning the process carried on […]<\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":56,"footnotes":""},"categories":[7],"tags":[129,124,128,4840,127],"class_list":["post-236","post","type-post","status-publish","format-standard","hentry","category-grails","tag-batch-insertion","tag-batch-processing","tag-clear-hibernate-session","tag-grails","tag-slow-insertion"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t