GoFuckYourself.com - Adult Webmaster Forum - View Single Post - How to crawl/scrape millions of URLs within couple of hours ?

NoWhErE · 02-09-2019, 08:33 AM

Personally I would future proof what you're doing and build a python script with rotating proxies to handle all the scraping.

Do it all through a VPN and take all the load off of your server.

02-09-2019, 08:33 AM
NoWhErE Too lazy to set a custom title Industry Role: Join Date: Sep 2005 Location: Canada Posts: 10,480	Personally I would future proof what you're doing and build a python script with rotating proxies to handle all the scraping. Do it all through a VPN and take all the load off of your server. __________________ skype: lordofthecameltoe