GoFuckYourself.com - Adult Webmaster Forum - View Single Post - How to crawl/scrape millions of URLs within couple of hours ?

mortenb · 02-09-2019, 06:32 AM

I'd look into using stormcrawler.net or Apache Nutch. Stormcrawler seems to be faster ATM, but YMMV. Should be relatively easy to set up. You will still need some beefy hardware and a good connection though.

02-09-2019, 06:32 AM
mortenb Confirmed User Join Date: Jul 2004 Location: Denmark ICQ: 7880009 Posts: 2,203	I'd look into using stormcrawler.net or Apache Nutch. Stormcrawler seems to be faster ATM, but YMMV. Should be relatively easy to set up. You will still need some beefy hardware and a good connection though.