Export very large data from a MongoDB collection using script

In this post, we will learn how to use javascript as MongoDB shell script to export (fields of) a collection.

If you just want to execute MongoDB commands then look at the another post there..

OK, so let’s focus back to the objective of this post.
Generally, to export data from a MongoDB collection, we use the following command –
[shell]
mongoexport -vvvv –host 127.0.0.1 –db my-db –username=’usr’ –password=’pwd’ –collection profiles –csv –out /home/data/usernames.csv -f ‘username’
[/shell]

Definitely I prefer the above mentioned way. But let’s consider that you have millions or billions of records in a MongoDB collection. And you want to export all these.
Probably, this will just hang. Well, I am not sure what happens on your computer but on my machine it was just stuck for hours and then I terminated the process. And came up with scripting solution – where I created multiple files (chunks) of the exported data.

[javascript]
var size = 1000000;
var maxCount = 1;
for (x=0;x<maxCount;x=x+1)
{
var recToSkip = x*size;
db.profiles.find().skip(recToSkip).limit(size).forEach( function(record){
var username = record.username;
print(record.username);
})
}

[/javascript]

And ofcourse to run the script (if above javascript is in file named – ‘usernames.js’)
[shell]
mongo 127.0.0.1/my-db –username=’username’ –password=’pwd’ export-usernames.js > output.txt
[/shell]

The above shell command will render all the output (from export-usernames.js) to the output.txt file. And mind it, that this is single file where you get all exported data. But if you want to chunkify this – just write your wrapper script or run multiple times. It’s upto you.

I hope this might help someone. Or please feel free to add your comments.

Cheers!
Salil Kalia
Salil [at] IntelliGrape [dot] com
Twitter LinkedIn