|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
06-19-2004, 01:16 AM
|
#1 |
|
Tube groupie.
Industry Role:
Join Date: Aug 2002
Location: LoScandalous, CA
Posts: 13,482
|
google spidering takes more bandwidth now
Anyone noticed the google spiders are gobbling a lot more bandwidth now? Daretwo got hold of me and informed me that the CPU usage my boxes was getting pegged out, but we couldn?t figure out from where. Checked the server log to find out it was those ?damn? google spiders doing it.
I just hoped the grabbed and ranked a ton of my pages well. Anyone else noticed this? |
|
|
|
06-19-2004, 01:18 AM
|
#2 |
|
[----------------------]
Join Date: Aug 2001
Posts: 14,486
|
yes
but quite some time now if im not mistaken |
|
|
|
|
06-19-2004, 01:19 AM
|
#3 |
|
Confirmed User
Industry Role:
Join Date: Mar 2003
Posts: 5,074
|
Ive yet to see any but I dont think google likes me. If they are taking that much I hope you get some nice SE traffic and a PR from it.
|
|
|
|
|
06-19-2004, 01:20 AM
|
#4 |
|
Confirmed User
Join Date: Jan 2004
Location: Boston, Massachusetts
Posts: 837
|
That does depend on how many pages google is spidering and you must have a lot. I know for a webhost defiantweb.com I see when the spiders check all the pages its about 3 times a week sometimes 5.
__________________
|
|
|
|
|
06-19-2004, 01:23 AM
|
#5 |
|
So Fucking Banned
Join Date: Jan 2003
Posts: 11,486
|
They're storing every bit of information on the web for the Information Awareness Office.
|
|
|
|
|
06-19-2004, 03:42 AM
|
#6 |
|
Confirmed User
Join Date: Aug 2002
Location: S. Florida
Posts: 750
|
Google is suppose to be server friendly. Are you sure they're google? Did you check the ips against the known google ips?
|
|
|
|
|
06-19-2004, 03:47 AM
|
#7 | |
|
Confirmed User
Join Date: Jun 2004
Location: Portland Fucking Oregon
Posts: 546
|
Quote:
__________________
ICQ - 196560594 mecopmfeel -at- yahoo -dot- com |
|
|
|
|
|
06-19-2004, 03:47 AM
|
#8 |
|
ex-TeenGodFather
Industry Role:
Join Date: Nov 2001
Location: Suomi Finland Perkele
Posts: 20,306
|
I believe google hates me personally.
__________________
..and I'm off. |
|
|
|
|
06-19-2004, 04:10 AM
|
#9 |
|
Confirmed User
Join Date: Mar 2004
Location: email address
Posts: 1,986
|
I haven't noticed, anyway I think google made some changes recently about..
Samuel |
|
|
|
|
06-19-2004, 04:30 AM
|
#10 |
|
Confirmed User
Join Date: May 2004
Location: 4 8 15 16 23 42
Posts: 4,444
|
There was a huge increase in google activity lately, could be related to the going public. I don't know what's going on, but it grabs pages like there is no tomorrow. I recognized that suddenly links google never liked before are now crawled, forums and shops without mod rewrite, links with affiliate codes and stuff like this.
|
|
|
|
|
06-19-2004, 07:01 AM
|
#11 | |
|
Confirmed User
Join Date: Feb 2003
Location: Bostonnnn
Posts: 8,985
|
Quote:
 |
|
|
|
|
|
06-19-2004, 07:04 AM
|
#12 | |
|
Confirmed User
Join Date: Apr 2001
Posts: 8,245
|
Quote:
 |
|
|
|
|
|
06-19-2004, 09:44 AM
|
#13 |
|
Bon temps!
Join Date: Feb 2003
Location: down yonder
Posts: 14,194
|
You can block Google with robots.txt if you think they are taking too much of your bandwidth
__________________
. |
|
|
|
