GoFuckYourself.com - Adult Webmaster Forum - View Single Post - How to crawl/scrape millions of URLs within couple of hours ?

rowan · 02-11-2019, 05:15 AM

Fetching 138 pages per second (*), every second, from the same site, is kind of a cunty thing to do.

(*) A million pages in 2 hours is 138 per second

02-11-2019, 05:15 AM
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Fetching 138 pages per second (), every second, from the same site, is kind of a cunty thing to do. () A million pages in 2 hours is 138 per second