|
teoma crawler hitting sites hard
On one of my sites it's fetching a page every second or two... and this site has a *lot* of pages. Anyone else being 'battered' by this crawler? I would have thought their algorithm would still be gentle even on sites with many pages - if you need to fetch 5,000 pages from a site you don't have to do it all within 2 hours...
Sample IP: 65.214.36.183
AskJeeves, Inc. UU-65-214-36 (NET-65-214-36-0-1)
65.214.36.0 - 65.214.39.255
|