teoma crawler hitting sites hard
 

On one of my sites it's fetching a page every second or two... and this site has a *lot* of pages. Anyone else being 'battered' by this crawler? I would have thought their algorithm would still be gentle even on sites with many pages - if you need to fetch 5,000 pages from a site you don't have to do it all within 2 hours...


Sample IP: 65.214.36.183

AskJeeves, Inc. UU-65-214-36 (NET-65-214-36-0-1)
65.214.36.0 - 65.214.39.255

us5$ tail -n 50000 access.log | grep -c 65.214.3
38630

38,630 of the past 50,000 object fetches were generated by Teoma.