Google's been crawling my new site since 10am Dec 21st, none stop.

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • StuartD
    Sofa King Band
    • Jul 2002
    • 29903
    #1

    Google's been crawling my new site since 10am Dec 21st, none stop.

    Non stop, constant crawling, a new page view every few seconds, 24 hours a day... and it's still not indexed.

    Anybody know what's going on? I've never had google bot do this before.
    This is me on facebook
    This is me on twitter
    Tags: None
    • TampaToker
      Confirmed User
      • May 2006
      • 5828
      #2
      had same thing happened to me then boom went from 45 pages indexed to 1640 pages. give it a few weeks to catch up
      Icq 247-742-205

        Comment

        • JD
          Too lazy to set a custom title
          • Sep 2003
          • 22651
          #3
          i think GBot is doing a reindex/deepcrawl after the recent fuck up

            Comment

            • starpimps
              Confirmed User
              • Sep 2006
              • 6954
              #4
              ive been crawled alot lately also
              Teen Porn Models / Solo Girls

                Comment

                • Jace
                  FBOP Class Of 2013
                  • Jan 2004
                  • 35562
                  #5
                  give it 2-4 weeks to catch up, the index and crawl are two seperate thing

                  it will crawl, then the indexer will sort and BAM you will have listings

                  I have one site that gets hit by googlebot about 450 times a day, but the new pages don't make it into the main index for a week or more sometimes

                    Comment

                    • Scroto
                      Confirmed User
                      • Nov 2005
                      • 2804
                      #6
                      been crawling two of mine like a mofo the last two days too

                        Comment

                        • StuartD
                          Sofa King Band
                          • Jul 2002
                          • 29903
                          #7
                          Originally posted by Jace
                          give it 2-4 weeks to catch up, the index and crawl are two seperate thing

                          it will crawl, then the indexer will sort and BAM you will have listings

                          I have one site that gets hit by googlebot about 450 times a day, but the new pages don't make it into the main index for a week or more sometimes
                          No, I know... I understand how it works, but I've never had it crawl so much for so long. It's kinda freaking me out.
                          This is me on facebook
                          This is me on twitter

                            Comment

                            • PurrrsianPussyKat
                              Confirmed User
                              • Jan 2004
                              • 2088
                              #8
                              Don't worry stuart, It's just having some trouble translating it from canadian to english.
                              Need a reliable, reasonably priced php programmer? Hit me up! 2934409

                                Comment

                                • Fizzgig
                                  Registered User
                                  • Feb 2004
                                  • 9649
                                  #9
                                  What brand is your spider food?
                                  ---'-,-{@ Sassy Grrrl @}-'-,---

                                    Comment

                                    • StuartD
                                      Sofa King Band
                                      • Jul 2002
                                      • 29903
                                      #10
                                      It's STILL crawling!!

                                      It's pulling an average of 12megs a day during the crawl... sometimes peaking at 22megs!!

                                      Dec 21st is when it started and here we are almost a whole month later and my site still isn't indexed in their searches yet receives a google bot hit every few seconds.
                                      This is me on facebook
                                      This is me on twitter

                                        Comment

                                        • Wagerboy
                                          Confirmed User
                                          • Jul 2006
                                          • 2290
                                          #11
                                          Im sorry to sound stupid, but how do you know its crawling so much?
                                          The Main Program That Makes Me Fucking Money!!

                                          Teens Getting Pounded

                                            Comment

                                            • Naughty-Pages
                                              Confirmed User
                                              • Oct 2006
                                              • 4533
                                              #12
                                              Originally posted by StuartD
                                              It's STILL crawling!!
                                              Does the site have a forum by chance??

                                              I've got 2 forums that google has been indexing non stop for like 2+ months LOL

                                              Between both forums I have several thousand pages of topics and a shitload of members profiles..

                                              I think part of the friggin problem is session id's... (I seriously need to install a modification to remove them, but i have been lazy).

                                              I am pretty certain the problem is that each time google indexes the site, it's a different session id, so google "thinks" that they are completely "new" pages and that all of the pages from the previous crawl no longer exist.

                                                Comment

                                                • Naughty-Pages
                                                  Confirmed User
                                                  • Oct 2006
                                                  • 4533
                                                  #13
                                                  just took a look at your sites...

                                                  ya, i think what i said before is what your problem is... (your sites use session ids...)

                                                    Comment

                                                    • StuartD
                                                      Sofa King Band
                                                      • Jul 2002
                                                      • 29903
                                                      #14
                                                      Originally posted by Wagerboy
                                                      Im sorry to sound stupid, but how do you know its crawling so much?
                                                      2 ways actually... I have my own stats script monitoring my site so I can see hits as they happen. And the other way is with Google's sitemaps. It actually shows the crawl rate, complete with line graphs.

                                                      Naughty-Pages, I have no forum on my site that's being crawled, and the site I'm talking about isn't in my sig. It's definitely not a session id since my site doesn't use sessions.
                                                      This is me on facebook
                                                      This is me on twitter

                                                        Comment

                                                        • SCORE Ralph
                                                          Confirmed User
                                                          • Mar 2003
                                                          • 2090
                                                          #15
                                                          Originally posted by SPeRMiNaToR
                                                          i think GBot is doing a reindex/deepcrawl after the recent fuck up

                                                          I dont think it was a fuck up, cause otherwise theyd have a backup. But I do agree thats prob reindexing/deepcrawling porn sites again for fresh results.
                                                          GetSCORECash.com | In the Biz Since 1991
                                                          Big Tits | Granny & MILFs | Amateurs | Big Booty | Foot Fetish | BBW | Teens
                                                          Hosted Embeds | MP4s | RSS Feeds | FHGs | Model Directory

                                                            Comment

                                                            • Wagerboy
                                                              Confirmed User
                                                              • Jul 2006
                                                              • 2290
                                                              #16
                                                              I dont see where I can get that info off google sitemaps...can you pooint me in the right direction?
                                                              The Main Program That Makes Me Fucking Money!!

                                                              Teens Getting Pounded

                                                                Comment

                                                                • Wagerboy
                                                                  Confirmed User
                                                                  • Jul 2006
                                                                  • 2290
                                                                  #17
                                                                  nevermind, found it by accident.
                                                                  The Main Program That Makes Me Fucking Money!!

                                                                  Teens Getting Pounded

                                                                    Comment

                                                                    • PR_Sebas
                                                                      Confirmed User
                                                                      • Dec 2006
                                                                      • 2825
                                                                      #18
                                                                      IMO googles ranks for porn sites lately has been awful. I've typed in a few select aduly keywords and all im getting are news sites and other useless non-adult stuff.

                                                                        Comment

                                                                        • teomaxxx
                                                                          Confirmed User
                                                                          • May 2003
                                                                          • 2737
                                                                          #19
                                                                          Originally posted by Naughty-Pages
                                                                          Does the site have a forum by chance??

                                                                          I've got 2 forums that google has been indexing non stop for like 2+ months LOL

                                                                          Between both forums I have several thousand pages of topics and a shitload of members profiles..

                                                                          I think part of the friggin problem is session id's... (I seriously need to install a modification to remove them, but i have been lazy).

                                                                          I am pretty certain the problem is that each time google indexes the site, it's a different session id, so google "thinks" that they are completely "new" pages and that all of the pages from the previous crawl no longer exist.
                                                                          does your forum use vbulletin?
                                                                          we have one forum and when doing command "site:my-forum.com" google shows at first that it found 39 500 pages , but in reality it has only 70 pages from that domain, mainly these that arent generated by vbulletin.
                                                                          I was just thinking where is the problem.

                                                                            Comment

                                                                            • bDok
                                                                              Confirmed User
                                                                              • Feb 2005
                                                                              • 1917
                                                                              #20
                                                                              Originally posted by Fizzgig
                                                                              What brand is your spider food?


                                                                              nice one.
                                                                              =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
                                                                              Warriors come out to plaAAaayyy!
                                                                              =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

                                                                                Comment

                                                                                • ssp
                                                                                  Confirmed User
                                                                                  • Jan 2005
                                                                                  • 7990
                                                                                  #21
                                                                                  You are in control of Googles craw rate. That is, you can slow it down if you like.

                                                                                    Comment

                                                                                    • HomeFry
                                                                                      Confirmed User
                                                                                      • Jun 2006
                                                                                      • 1062
                                                                                      #22
                                                                                      Google has been crawling my sites non-stop, for the past 8 years.
                                                                                      Need Mushrooms Designed? Gimme-Website

                                                                                        Comment

                                                                                        • Bliggo
                                                                                          Registered User
                                                                                          • Jun 2006
                                                                                          • 99
                                                                                          #23
                                                                                          Originally posted by ssp
                                                                                          You are in control of Googles craw rate. That is, you can slow it down if you like.
                                                                                          YARGH! Yup within the sitemap section of google's webmaster tools area.

                                                                                          I seo'd my hair yesterday and today it's a pr6

                                                                                            Comment

                                                                                            • StuartD
                                                                                              Sofa King Band
                                                                                              • Jul 2002
                                                                                              • 29903
                                                                                              #24
                                                                                              Originally posted by ssp
                                                                                              You are in control of Googles craw rate. That is, you can slow it down if you like.
                                                                                              I realize that, but that won't solve the problem nor answer my questions.
                                                                                              This is me on facebook
                                                                                              This is me on twitter

                                                                                                Comment

                                                                                                • Brad
                                                                                                  Confirmed User
                                                                                                  • Jan 2007
                                                                                                  • 2510
                                                                                                  #25
                                                                                                  Originally posted by PurrrsianPussyKat
                                                                                                  Don't worry stuart, It's just having some trouble translating it from canadian to english.
                                                                                                  lol. what's that all aboot? Time to toss on the Canadian Tuxedo and brave the cold. I need some new candles for my igloo eh
                                                                                                  Affiliate Manager/Media Buyer
                                                                                                  ICQ 376 978 529
                                                                                                  Skype: bradleyjwscott
                                                                                                  [email protected]

                                                                                                    Comment

                                                                                                    • pussyluver
                                                                                                      Clueless OleMan
                                                                                                      • Mar 2003
                                                                                                      • 11009
                                                                                                      #26
                                                                                                      There is growing belief that Google is just plain broke. As in doesn't work! All fucked up.... How many horror stories do you have to read on the SEO forums before you believe. How many companies have to be driven out of business because they loose their Google placement for no reason that can be discerned.

                                                                                                      Hidden links and text still affect SERP as does .edu and .gov spamming.

                                                                                                      The notion that a site should be ranked on popularity as it relates to "quality" back links is flawed IMO.

                                                                                                      Yes, I know Google provided a means for many companies to succeed without paying for the placement on key cash words. So in that statement lies the moral to the story. Don't rely on google for your traffic.

                                                                                                        Comment

                                                                                                        • StuartD
                                                                                                          Sofa King Band
                                                                                                          • Jul 2002
                                                                                                          • 29903
                                                                                                          #27
                                                                                                          Still going...

                                                                                                          This is me on facebook
                                                                                                          This is me on twitter

                                                                                                            Comment

                                                                                                            • Adultnet
                                                                                                              Confirmed User
                                                                                                              • Sep 2003
                                                                                                              • 8713
                                                                                                              #28
                                                                                                              Originally posted by TampaToker
                                                                                                              had same thing happened to me then boom went from 45 pages indexed to 1640 pages. give it a few weeks to catch up
                                                                                                              good result


                                                                                                              TrafficCashGold Paying Webmasters Since 1996!
                                                                                                              Awesome Conversions! Fast Weekly Payments! Over 125 Tours!

                                                                                                                Comment

                                                                                                                • pussyluver
                                                                                                                  Clueless OleMan
                                                                                                                  • Mar 2003
                                                                                                                  • 11009
                                                                                                                  #29
                                                                                                                  Originally posted by StuartD
                                                                                                                  2 ways actually... I have my own stats script monitoring my site so I can see hits as they happen. And the other way is with Google's sitemaps. It actually shows the crawl rate, complete with line graphs.

                                                                                                                  Naughty-Pages, I have no forum on my site that's being crawled, and the site I'm talking about isn't in my sig. It's definitely not a session id since my site doesn't use sessions.

                                                                                                                  I found that google does not necessarily update the info every day. The graph you're referring to can be the same for several days, then all of a sudden catch up. Also the seven day moving average report on query stats is the same way. Maybe you are having a different experience. Also see some strange reports. Like I had an average of the No. 1 position for the word Ebony. There is no way. Don't have an Ebony site nor is it a keyword pushed on my sites. Plus a check confirmed no hits on that keyword.

                                                                                                                    Comment

                                                                                                                    • StuartD
                                                                                                                      Sofa King Band
                                                                                                                      • Jul 2002
                                                                                                                      • 29903
                                                                                                                      #30
                                                                                                                      Originally posted by pussyluver
                                                                                                                      I found that google does not necessarily update the info every day. The graph you're referring to can be the same for several days, then all of a sudden catch up. Also the seven day moving average report on query stats is the same way. Maybe you are having a different experience. Also see some strange reports. Like I had an average of the No. 1 position for the word Ebony. There is no way. Don't have an Ebony site nor is it a keyword pushed on my sites. Plus a check confirmed no hits on that keyword.
                                                                                                                      No, I know the graph isn't updated in real time, but look at it... it's showing non stop crawling. It doesn't let up at all, not for a minute, much less any days or anything.

                                                                                                                      I don't care much for whether or not I get a #1 result... right now I'd just like to be listed. But it doesn't appear to be happening since the crawling never stops.
                                                                                                                      This is me on facebook
                                                                                                                      This is me on twitter

                                                                                                                        Comment

                                                                                                                        • Sambo
                                                                                                                          So Fucking Banned
                                                                                                                          • Mar 2004
                                                                                                                          • 1187
                                                                                                                          #31
                                                                                                                          I actually had a problem with Google doing this - it was pulling GIGS every day on a Blog Host I run. Sometimes 3 or 4 hundred thousand hits a day.

                                                                                                                          It turned out that I had managed to 'break' it with some fucked up code in my Robots.txt file.

                                                                                                                          Go to Google Webmaster tools and do a check on your robots.txt

                                                                                                                            Comment

                                                                                                                            • teomaxxx
                                                                                                                              Confirmed User
                                                                                                                              • May 2003
                                                                                                                              • 2737
                                                                                                                              #32
                                                                                                                              Originally posted by Naughty-Pages
                                                                                                                              Does the site have a forum by chance??

                                                                                                                              I've got 2 forums that google has been indexing non stop for like 2+ months LOL

                                                                                                                              Between both forums I have several thousand pages of topics and a shitload of members profiles..

                                                                                                                              I think part of the friggin problem is session id's... (I seriously need to install a modification to remove them, but i have been lazy).

                                                                                                                              I am pretty certain the problem is that each time google indexes the site, it's a different session id, so google "thinks" that they are completely "new" pages and that all of the pages from the previous crawl no longer exist.
                                                                                                                              Originally posted by teomaxxx
                                                                                                                              does your forum use vbulletin?
                                                                                                                              we have one forum and when doing command "site:my-forum.com" google shows at first that it found 39 500 pages , but in reality it has only 70 pages from that domain, mainly these that arent generated by vbulletin.
                                                                                                                              I was just thinking where is the problem.
                                                                                                                              bump for my question.

                                                                                                                                Comment

                                                                                                                                • mattz
                                                                                                                                  Confirmed User
                                                                                                                                  • Dec 2001
                                                                                                                                  • 7697
                                                                                                                                  #33
                                                                                                                                  if you arnt indexed after a month thats not good

                                                                                                                                    Comment

                                                                                                                                    • StuartD
                                                                                                                                      Sofa King Band
                                                                                                                                      • Jul 2002
                                                                                                                                      • 29903
                                                                                                                                      #34
                                                                                                                                      Originally posted by mattz
                                                                                                                                      if you arnt indexed after a month thats not good
                                                                                                                                      Tell me about it. I've made a lot of sites and they've all been indexed pretty quickly.
                                                                                                                                      This is definitely a first for me.
                                                                                                                                      This is me on facebook
                                                                                                                                      This is me on twitter

                                                                                                                                        Comment

                                                                                                                                        Previous template Next
                                                                                                                                        Working...