google spidering takes more bandwidth now

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • xxxjay
    Tube groupie.
    • Aug 2002
    • 13482
    #1

    google spidering takes more bandwidth now

    Anyone noticed the google spiders are gobbling a lot more bandwidth now? Daretwo got hold of me and informed me that the CPU usage my boxes was getting pegged out, but we couldn?t figure out from where. Checked the server log to find out it was those ?damn? google spiders doing it.

    I just hoped the grabbed and ranked a ton of my pages well. Anyone else noticed this?
    http://donttellmehowtoruinmylife.com/ - http://www.jmdigitalmarketing.com/my...s-and-reviews/ - http://www.wouldyouhitit.org - http://shinyobjectreviews.com/
    Tags: None
    • Trax
      [----------------------]
      • Aug 2001
      • 14486
      #2
      yes
      but quite some time now if im not mistaken

        Comment

        • fr8
          Confirmed User
          • Mar 2003
          • 5074
          #3
          Ive yet to see any but I dont think google likes me. If they are taking that much I hope you get some nice SE traffic and a PR from it.
          █ joesmut (a) gmail Dot com
          █ Full Stack Developer

            Comment

            • Jarrod
              Confirmed User
              • Jan 2004
              • 837
              #4
              That does depend on how many pages google is spidering and you must have a lot. I know for a webhost defiantweb.com I see when the spiders check all the pages its about 3 times a week sometimes 5.

                Comment

                • Rich
                  So Fucking Banned
                  • Jan 2003
                  • 11486
                  #5
                  They're storing every bit of information on the web for the Information Awareness Office.

                    Comment

                    • VideoJ
                      Confirmed User
                      • Aug 2002
                      • 750
                      #6
                      Google is suppose to be server friendly. Are you sure they're google? Did you check the ips against the known google ips?
                      Somebody stole my damn signture...

                        Comment

                        • ScottMN
                          Confirmed User
                          • Jun 2004
                          • 546
                          #7
                          Originally posted by Rich
                          They're storing every bit of information on the web for the Information Awareness Office.
                          is this true?
                          ICQ - 196560594
                          mecopmfeel -at- yahoo -dot- com

                            Comment

                            • Jakke PNG
                              ex-TeenGodFather
                              • Nov 2001
                              • 20306
                              #8
                              I believe google hates me personally.
                              ..and I'm off.

                                Comment

                                • samuel
                                  Confirmed User
                                  • Mar 2004
                                  • 1986
                                  #9
                                  I haven't noticed, anyway I think google made some changes recently about..

                                  Samuel

                                    Comment

                                    • StarkReality
                                      Confirmed User
                                      • May 2004
                                      • 4444
                                      #10
                                      There was a huge increase in google activity lately, could be related to the going public. I don't know what's going on, but it grabs pages like there is no tomorrow. I recognized that suddenly links google never liked before are now crawled, forums and shops without mod rewrite, links with affiliate codes and stuff like this.

                                        Comment

                                        • AdultNex
                                          Confirmed User
                                          • Feb 2003
                                          • 8985
                                          #11
                                          Originally posted by ScottMN
                                          is this true?
                                          Wow.

                                            Comment

                                            • ldinternet
                                              Confirmed User
                                              • Apr 2001
                                              • 8245
                                              #12
                                              Originally posted by TeenGodFather
                                              I believe google hates me personally.
                                              hahaha

                                                Comment

                                                • mardigras
                                                  Bon temps!
                                                  • Feb 2003
                                                  • 14194
                                                  #13
                                                  You can block Google with robots.txt if you think they are taking too much of your bandwidth
                                                  .

                                                    Comment

                                                    Previous template Next
                                                    Working...