Quote:
Originally Posted by rowan
Fetching 138 pages per second (*), every second, from the same site, is kind of a cunty thing to do.
(*) A million pages in 2 hours is 138 per second
|
In the link AdultKing posted, the author addresses the burden that crawlers can potentially impose on websites and designed his to be "polite".