teoma crawler hitting sites hard

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • rowan
    Too lazy to set a custom title
    • Mar 2002
    • 17393
    #1

    teoma crawler hitting sites hard

    On one of my sites it's fetching a page every second or two... and this site has a *lot* of pages. Anyone else being 'battered' by this crawler? I would have thought their algorithm would still be gentle even on sites with many pages - if you need to fetch 5,000 pages from a site you don't have to do it all within 2 hours...


    Sample IP: 65.214.36.183

    AskJeeves, Inc. UU-65-214-36 (NET-65-214-36-0-1)
    65.214.36.0 - 65.214.39.255
    Tags: None
    • rowan
      Too lazy to set a custom title
      • Mar 2002
      • 17393
      #2
      us5$ tail -n 50000 access.log | grep -c 65.214.3
      38630

      38,630 of the past 50,000 object fetches were generated by Teoma.

        Comment

        Previous template Next
        Working...