Export very large data from a MongoDB collection using script

04 / Jul / 2012 by Salil 3 comments

In this post, we will learn how to use javascript as MongoDB shell script to export (fields of) a collection.

 

If you just want to execute MongoDB commands then look at the another post there..

 

OK, so let’s focus back to the objective of this post.
Generally, to export data from a MongoDB collection, we use the following command –
[shell]
mongoexport -vvvv –host 127.0.0.1 –db my-db –username=’usr’ –password=’pwd’ –collection profiles –csv –out /home/data/usernames.csv -f ‘username’
[/shell]

Definitely I prefer the above mentioned way. But let’s consider that you have millions or billions of records in a MongoDB collection. And you want to export all these.
Probably, this will just hang. Well, I am not sure what happens on your computer but on my machine it was just stuck for hours and then I terminated the process. And came up with scripting solution – where I created multiple files (chunks) of the exported data.

 

[javascript]
var size = 1000000;
var maxCount = 1;
for (x=0;x<maxCount;x=x+1)
{
var recToSkip = x*size;
db.profiles.find().skip(recToSkip).limit(size).forEach( function(record){
var username = record.username;
print(record.username);
})
}

[/javascript]

And ofcourse to run the script (if above javascript is in file named – ‘usernames.js’)
[shell]
mongo 127.0.0.1/my-db –username=’username’ –password=’pwd’ export-usernames.js > output.txt
[/shell]

The above shell command will render all the output (from export-usernames.js) to the output.txt file. And mind it, that this is single file where you get all exported data. But if you want to chunkify this – just write your wrapper script or run multiple times. It’s upto you.

 

I hope this might help someone. Or please feel free to add your comments.

 

Cheers!
Salil Kalia
Salil [at] IntelliGrape [dot] com
Twitter LinkedIn

FOUND THIS USEFUL? SHARE IT

comments (3)

  1. Ismail

    Seems Good Approach it Would be even much more better if we can able to use Node Script to export rather then JS script ?
    Any possibilities ?

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *