GoFuckYourself.com - Adult Webmaster Forum - View Single Post

rowan · 08-09-2004, 12:56 PM

On one of my sites it's fetching a page every second or two... and this site has a *lot* of pages. Anyone else being 'battered' by this crawler? I would have thought their algorithm would still be gentle even on sites with many pages - if you need to fetch 5,000 pages from a site you don't have to do it all within 2 hours...

Sample IP: 65.214.36.183

AskJeeves, Inc. UU-65-214-36 (NET-65-214-36-0-1)
65.214.36.0 - 65.214.39.255

08-09-2004, 12:56 PM
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	teoma crawler hitting sites hard On one of my sites it's fetching a page every second or two... and this site has a lot of pages. Anyone else being 'battered' by this crawler? I would have thought their algorithm would still be gentle even on sites with many pages - if you need to fetch 5,000 pages from a site you don't have to do it all within 2 hours... Sample IP: 65.214.36.183 AskJeeves, Inc. UU-65-214-36 (NET-65-214-36-0-1) 65.214.36.0 - 65.214.39.255