|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
08-09-2004, 12:56 PM
|
#1 |
|
Too lazy to set a custom title
Join Date: Mar 2002
Location: Australia
Posts: 17,393
|
teoma crawler hitting sites hard
On one of my sites it's fetching a page every second or two... and this site has a *lot* of pages. Anyone else being 'battered' by this crawler? I would have thought their algorithm would still be gentle even on sites with many pages - if you need to fetch 5,000 pages from a site you don't have to do it all within 2 hours...
Sample IP: 65.214.36.183 AskJeeves, Inc. UU-65-214-36 (NET-65-214-36-0-1) 65.214.36.0 - 65.214.39.255 |
|
|
|
08-09-2004, 01:00 PM
|
#2 |
|
Too lazy to set a custom title
Join Date: Mar 2002
Location: Australia
Posts: 17,393
|
us5$ tail -n 50000 access.log | grep -c 65.214.3
38630 38,630 of the past 50,000 object fetches were generated by Teoma. |
|
|
|
