Google's been crawling my new site since 10am Dec 21st, none stop. - GoFuckYourself.com

StuartD · 12-29-2006, 02:22 PM

Non stop, constant crawling, a new page view every few seconds, 24 hours a day... and it's still not indexed.

Anybody know what's going on? I've never had google bot do this before.

TampaToker · 12-29-2006, 02:26 PM

had same thing happened to me then boom went from 45 pages indexed to 1640 pages. give it a few weeks to catch up

JD · 12-29-2006, 02:56 PM

i think GBot is doing a reindex/deepcrawl after the recent fuck up

starpimps · 12-29-2006, 07:25 PM

ive been crawled alot lately also

Jace · 12-29-2006, 07:28 PM

give it 2-4 weeks to catch up, the index and crawl are two seperate thing

it will crawl, then the indexer will sort and BAM you will have listings

I have one site that gets hit by googlebot about 450 times a day, but the new pages don't make it into the main index for a week or more sometimes

Scroto · 12-29-2006, 07:29 PM

been crawling two of mine like a mofo the last two days too

StuartD · 12-29-2006, 08:01 PM

Quote:

Originally Posted by Jace

give it 2-4 weeks to catch up, the index and crawl are two seperate thing

it will crawl, then the indexer will sort and BAM you will have listings

I have one site that gets hit by googlebot about 450 times a day, but the new pages don't make it into the main index for a week or more sometimes

No, I know... I understand how it works, but I've never had it crawl so much for so long. It's kinda freaking me out.

PurrrsianPussyKat · 12-29-2006, 09:07 PM

Don't worry stuart, It's just having some trouble translating it from canadian to english.

Fizzgig · 12-29-2006, 09:08 PM

What brand is your spider food?

StuartD · 01-17-2007, 07:19 AM

It's STILL crawling!!

It's pulling an average of 12megs a day during the crawl... sometimes peaking at 22megs!!

Dec 21st is when it started and here we are almost a whole month later and my site still isn't indexed in their searches yet receives a google bot hit every few seconds.

Wagerboy · 01-17-2007, 08:05 AM

Im sorry to sound stupid, but how do you know its crawling so much?

Naughty-Pages · 01-17-2007, 08:17 AM

Quote:

Originally Posted by StuartD

It's STILL crawling!!

Does the site have a forum by chance??

I've got 2 forums that google has been indexing non stop for like 2+ months LOL

Between both forums I have several thousand pages of topics and a shitload of members profiles..

I think part of the friggin problem is session id's... (I seriously need to install a modification to remove them, but i have been lazy).

I am pretty certain the problem is that each time google indexes the site, it's a different session id, so google "thinks" that they are completely "new" pages and that all of the pages from the previous crawl no longer exist.

Naughty-Pages · 01-17-2007, 08:20 AM

just took a look at your sites...

ya, i think what i said before is what your problem is... (your sites use session ids...)

StuartD · 01-17-2007, 08:25 AM

Quote:

Originally Posted by Wagerboy

Im sorry to sound stupid, but how do you know its crawling so much?

2 ways actually... I have my own stats script monitoring my site so I can see hits as they happen. And the other way is with Google's sitemaps. It actually shows the crawl rate, complete with line graphs.

Naughty-Pages, I have no forum on my site that's being crawled, and the site I'm talking about isn't in my sig. It's definitely not a session id since my site doesn't use sessions.

SCORE Ralph · 01-17-2007, 08:30 AM

Quote:

Originally Posted by SPeRMiNaToR

i think GBot is doing a reindex/deepcrawl after the recent fuck up

I dont think it was a fuck up, cause otherwise theyd have a backup. But I do agree thats prob reindexing/deepcrawling porn sites again for fresh results.

Wagerboy · 01-17-2007, 08:40 AM

I dont see where I can get that info off google sitemaps...can you pooint me in the right direction?

Wagerboy · 01-17-2007, 08:41 AM

nevermind, found it by accident.

PR_Sebas · 01-17-2007, 08:42 AM

IMO googles ranks for porn sites lately has been awful. I've typed in a few select aduly keywords and all im getting are news sites and other useless non-adult stuff.

teomaxxx · 01-18-2007, 07:11 AM

Quote:

Originally Posted by Naughty-Pages

Does the site have a forum by chance??

I've got 2 forums that google has been indexing non stop for like 2+ months LOL

Between both forums I have several thousand pages of topics and a shitload of members profiles..

I think part of the friggin problem is session id's... (I seriously need to install a modification to remove them, but i have been lazy).

I am pretty certain the problem is that each time google indexes the site, it's a different session id, so google "thinks" that they are completely "new" pages and that all of the pages from the previous crawl no longer exist.

does your forum use vbulletin?
we have one forum and when doing command "site:my-forum.com" google shows at first that it found 39 500 pages , but in reality it has only 70 pages from that domain, mainly these that arent generated by vbulletin.
I was just thinking where is the problem.

bDok · 01-18-2007, 10:25 AM

Quote:

Originally Posted by Fizzgig

What brand is your spider food?

nice one.

ssp · 01-18-2007, 10:58 AM

You are in control of Googles craw rate. That is, you can slow it down if you like.

HomeFry · 01-18-2007, 11:00 AM

Google has been crawling my sites non-stop, for the past 8 years.

Bliggo · 01-18-2007, 11:04 AM

Quote:

Originally Posted by ssp

You are in control of Googles craw rate. That is, you can slow it down if you like.

YARGH! Yup within the sitemap section of google's webmaster tools area.

StuartD · 01-18-2007, 11:11 AM

Quote:

Originally Posted by ssp

You are in control of Googles craw rate. That is, you can slow it down if you like.

I realize that, but that won't solve the problem nor answer my questions.

Brad · 01-18-2007, 11:19 AM

Quote:

Originally Posted by PurrrsianPussyKat

Don't worry stuart, It's just having some trouble translating it from canadian to english.

lol. what's that all aboot? Time to toss on the Canadian Tuxedo and brave the cold. I need some new candles for my igloo eh

pussyluver · 01-18-2007, 11:46 AM

There is growing belief that Google is just plain broke. As in doesn't work! All fucked up.... How many horror stories do you have to read on the SEO forums before you believe. How many companies have to be driven out of business because they loose their Google placement for no reason that can be discerned.

Hidden links and text still affect SERP as does .edu and .gov spamming.

The notion that a site should be ranked on popularity as it relates to "quality" back links is flawed IMO.

Yes, I know Google provided a means for many companies to succeed without paying for the placement on key cash words. So in that statement lies the moral to the story. Don't rely on google for your traffic.

StuartD · 01-20-2007, 12:13 PM

Still going...

Adultnet · 01-20-2007, 12:24 PM

Quote:

Originally Posted by TampaToker

had same thing happened to me then boom went from 45 pages indexed to 1640 pages. give it a few weeks to catch up

good result

pussyluver · 01-20-2007, 01:11 PM

Quote:

Originally Posted by StuartD

2 ways actually... I have my own stats script monitoring my site so I can see hits as they happen. And the other way is with Google's sitemaps. It actually shows the crawl rate, complete with line graphs.

Naughty-Pages, I have no forum on my site that's being crawled, and the site I'm talking about isn't in my sig. It's definitely not a session id since my site doesn't use sessions.

I found that google does not necessarily update the info every day. The graph you're referring to can be the same for several days, then all of a sudden catch up. Also the seven day moving average report on query stats is the same way. Maybe you are having a different experience. Also see some strange reports. Like I had an average of the No. 1 position for the word Ebony. There is no way. Don't have an Ebony site nor is it a keyword pushed on my sites. Plus a check confirmed no hits on that keyword.

StuartD · 01-20-2007, 05:40 PM

Quote:

Originally Posted by pussyluver

I found that google does not necessarily update the info every day. The graph you're referring to can be the same for several days, then all of a sudden catch up. Also the seven day moving average report on query stats is the same way. Maybe you are having a different experience. Also see some strange reports. Like I had an average of the No. 1 position for the word Ebony. There is no way. Don't have an Ebony site nor is it a keyword pushed on my sites. Plus a check confirmed no hits on that keyword.

No, I know the graph isn't updated in real time, but look at it... it's showing non stop crawling. It doesn't let up at all, not for a minute, much less any days or anything.

I don't care much for whether or not I get a #1 result... right now I'd just like to be listed. But it doesn't appear to be happening since the crawling never stops.

Sambo · 01-20-2007, 06:15 PM

I actually had a problem with Google doing this - it was pulling GIGS every day on a Blog Host I run. Sometimes 3 or 4 hundred thousand hits a day.

It turned out that I had managed to 'break' it with some fucked up code in my Robots.txt file.

Go to Google Webmaster tools and do a check on your robots.txt

teomaxxx · 01-26-2007, 11:45 AM

Quote:

Originally Posted by Naughty-Pages

Does the site have a forum by chance??

I've got 2 forums that google has been indexing non stop for like 2+ months LOL

Between both forums I have several thousand pages of topics and a shitload of members profiles..

I think part of the friggin problem is session id's... (I seriously need to install a modification to remove them, but i have been lazy).

I am pretty certain the problem is that each time google indexes the site, it's a different session id, so google "thinks" that they are completely "new" pages and that all of the pages from the previous crawl no longer exist.

Quote:

Originally Posted by teomaxxx

does your forum use vbulletin?
we have one forum and when doing command "site:my-forum.com" google shows at first that it found 39 500 pages , but in reality it has only 70 pages from that domain, mainly these that arent generated by vbulletin.
I was just thinking where is the problem.

bump for my question.

mattz · 01-26-2007, 11:48 AM

if you arnt indexed after a month thats not good

StuartD · 01-26-2007, 04:57 PM

Quote:

Originally Posted by mattz

if you arnt indexed after a month thats not good

Tell me about it. I've made a lot of sites and they've all been indexed pretty quickly.
This is definitely a first for me.

12-29-2006, 07:25 PM	#4
starpimps Confirmed User Join Date: Sep 2006 Location: internets Posts: 6,954	ive been crawled alot lately also __________________ Teen Porn Models / Solo Girls

12-29-2006, 09:07 PM	#8
PurrrsianPussyKat Confirmed User Join Date: Jan 2004 Location: Lost in the mountains Posts: 2,088	Don't worry stuart, It's just having some trouble translating it from canadian to english. __________________ Need a reliable, reasonably priced php programmer? Hit me up! 2934409

12-29-2006, 09:08 PM	#9
Fizzgig Registered User Join Date: Feb 2004 Location: In Your Dreams Posts: 9,649	What brand is your spider food? __________________ ---'-,-{@ Sassy Grrrl @}-'-,---

01-17-2007, 07:19 AM	#10
StuartD Sofa King Band Join Date: Jul 2002 Location: Outside the box Posts: 29,903	It's STILL crawling!! It's pulling an average of 12megs a day during the crawl... sometimes peaking at 22megs!! Dec 21st is when it started and here we are almost a whole month later and my site still isn't indexed in their searches yet receives a google bot hit every few seconds. __________________ This is me on facebook This is me on twitter

01-17-2007, 08:05 AM	#11
Wagerboy Confirmed User Join Date: Jul 2006 Posts: 2,290	Im sorry to sound stupid, but how do you know its crawling so much? __________________ The Main Program That Makes Me Fucking Money!! Teens Getting Pounded

12-29-2006, 02:22 PM	#1
StuartD Sofa King Band Join Date: Jul 2002 Location: Outside the box Posts: 29,903	Google's been crawling my new site since 10am Dec 21st, none stop. Non stop, constant crawling, a new page view every few seconds, 24 hours a day... and it's still not indexed. Anybody know what's going on? I've never had google bot do this before. __________________ This is me on facebook This is me on twitter

12-29-2006, 02:26 PM	#2
TampaToker Confirmed User Join Date: May 2006 Location: Tampa Posts: 5,827	had same thing happened to me then boom went from 45 pages indexed to 1640 pages. give it a few weeks to catch up

12-29-2006, 02:56 PM	#3
JD Too lazy to set a custom title Industry Role: Join Date: Sep 2003 Posts: 22,651	i think GBot is doing a reindex/deepcrawl after the recent fuck up

12-29-2006, 07:28 PM	#5
Jace FBOP Class Of 2013 Industry Role: Join Date: Jan 2004 Location: bumfuck, ky Posts: 35,562	give it 2-4 weeks to catch up, the index and crawl are two seperate thing it will crawl, then the indexer will sort and BAM you will have listings I have one site that gets hit by googlebot about 450 times a day, but the new pages don't make it into the main index for a week or more sometimes

12-29-2006, 07:29 PM	#6
Scroto Confirmed User Join Date: Nov 2005 Posts: 2,804	been crawling two of mine like a mofo the last two days too

01-17-2007, 08:20 AM	#13
Naughty-Pages Confirmed User Industry Role: Join Date: Oct 2006 Location: SWFL Posts: 4,533	just took a look at your sites... ya, i think what i said before is what your problem is... (your sites use session ids...)

01-17-2007, 08:40 AM	#16
Wagerboy Confirmed User Join Date: Jul 2006 Posts: 2,290	I dont see where I can get that info off google sitemaps...can you pooint me in the right direction? __________________ The Main Program That Makes Me Fucking Money!! Teens Getting Pounded

01-17-2007, 08:41 AM	#17
Wagerboy Confirmed User Join Date: Jul 2006 Posts: 2,290	nevermind, found it by accident. __________________ The Main Program That Makes Me Fucking Money!! Teens Getting Pounded

01-17-2007, 08:42 AM	#18
PR_Sebas Confirmed User Join Date: Dec 2006 Posts: 2,825	IMO googles ranks for porn sites lately has been awful. I've typed in a few select aduly keywords and all im getting are news sites and other useless non-adult stuff. __________________

01-18-2007, 10:58 AM	#21
ssp Confirmed User Join Date: Jan 2005 Location: United Kingdom Posts: 7,990	You are in control of Googles craw rate. That is, you can slow it down if you like.

01-18-2007, 11:00 AM	#22
HomeFry Confirmed User Join Date: Jun 2006 Posts: 1,062	Google has been crawling my sites non-stop, for the past 8 years.

01-18-2007, 11:46 AM	#26
pussyluver Clueless OleMan Join Date: Mar 2003 Location: ICQ - 169903487 Posts: 11,009	There is growing belief that Google is just plain broke. As in doesn't work! All fucked up.... How many horror stories do you have to read on the SEO forums before you believe. How many companies have to be driven out of business because they loose their Google placement for no reason that can be discerned. Hidden links and text still affect SERP as does .edu and .gov spamming. The notion that a site should be ranked on popularity as it relates to "quality" back links is flawed IMO. Yes, I know Google provided a means for many companies to succeed without paying for the placement on key cash words. So in that statement lies the moral to the story. Don't rely on google for your traffic.

01-20-2007, 12:13 PM	#27
StuartD Sofa King Band Join Date: Jul 2002 Location: Outside the box Posts: 29,903	Still going... __________________ This is me on facebook This is me on twitter

01-20-2007, 06:15 PM	#31
Sambo So Fucking Banned Join Date: Mar 2004 Posts: 1,187	I actually had a problem with Google doing this - it was pulling GIGS every day on a Blog Host I run. Sometimes 3 or 4 hundred thousand hits a day. It turned out that I had managed to 'break' it with some fucked up code in my Robots.txt file. Go to Google Webmaster tools and do a check on your robots.txt

01-26-2007, 11:48 AM	#33
mattz Confirmed User Industry Role: Join Date: Dec 2001 Location: AZ Posts: 7,697	if you arnt indexed after a month thats not good