Quote:
Originally Posted by Fenris Wolf
In the link AdultKing posted, the author addresses the burden that crawlers can potentially impose on websites and designed his to be "polite".
|
That's a crawler designed to access multiple websites, but the OP wants to scrape one site only. There's no polite way to fetch a million pages from a single website.