View Single Post
Old 02-11-2019, 02:44 PM  
rowan
Too lazy to set a custom title
 
Join Date: Mar 2002
Location: Australia
Posts: 17,393
Quote:
Originally Posted by Fenris Wolf View Post
In the link AdultKing posted, the author addresses the burden that crawlers can potentially impose on websites and designed his to be "polite".
That's a crawler designed to access multiple websites, but the OP wants to scrape one site only. There's no polite way to fetch a million pages from a single website.
rowan is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote