GoFuckYourself.com - Adult Webmaster Forum - View Single Post - Tech Is Anyone Using The Full Data Dump From HubTrafiic?..

EddyTheDog · 01-08-2019, 01:01 PM

OK - So this is the flow I used:

Download/Unzip CSV dump -> Use FileSplitter to split CSV into 500mb files -> Used NotePad++ to find and replace pipes with tabs (makes DB import easier) -> Used Navicat to import into MongoDB...

Now I just have to learn how to use the data lol.....

Notes: This was all done on a local Windows machine - Navicat will import them into an external MongoDB instance if needed - It's a lot of data and you need to be patient - Either that or start a large Google Cloud instance and do it on there, I wish I had done that lol...

01-08-2019, 01:01 PM
EddyTheDog Just Doing My Own Thing Industry Role: Join Date: Jan 2011 Location: London, Spain, New Zealand, GFY - Not Croydon... Posts: 25,339	OK - So this is the flow I used: Download/Unzip CSV dump -> Use FileSplitter to split CSV into 500mb files -> Used NotePad++ to find and replace pipes with tabs (makes DB import easier) -> Used Navicat to import into MongoDB... Now I just have to learn how to use the data lol..... Notes: This was all done on a local Windows machine - Navicat will import them into an external MongoDB instance if needed - It's a lot of data and you need to be patient - Either that or start a large Google Cloud instance and do it on there, I wish I had done that lol...