Google's been crawling my new site since 10am Dec 21st, none stop.
 

Non stop, constant crawling, a new page view every few seconds, 24 hours a day... and it's still not indexed.

Anybody know what's going on? I've never had google bot do this before.

had same thing happened to me then boom went from 45 pages indexed to 1640 pages. give it a few weeks to catch up :thumbsup

i think GBot is doing a reindex/deepcrawl after the recent fuck up

ive been crawled alot lately also

give it 2-4 weeks to catch up, the index and crawl are two seperate thing

it will crawl, then the indexer will sort and BAM you will have listings

I have one site that gets hit by googlebot about 450 times a day, but the new pages don't make it into the main index for a week or more sometimes

been crawling two of mine like a mofo the last two days too :thumbsup

No, I know... I understand how it works, but I've never had it crawl so much for so long. It's kinda freaking me out.

Don't worry stuart, It's just having some trouble translating it from canadian to english. :winkwink:

What brand is your spider food?

It's STILL crawling!!

It's pulling an average of 12megs a day during the crawl... sometimes peaking at 22megs!!

Dec 21st is when it started and here we are almost a whole month later and my site still isn't indexed in their searches yet receives a google bot hit every few seconds.

Im sorry to sound stupid, but how do you know its crawling so much?

Does the site have a forum by chance??

I've got 2 forums that google has been indexing non stop for like 2+ months LOL

Between both forums I have several thousand pages of topics and a shitload of members profiles..

I think part of the friggin problem is session id's... (I seriously need to install a modification to remove them, but i have been lazy).

I am pretty certain the problem is that each time google indexes the site, it's a different session id, so google "thinks" that they are completely "new" pages and that all of the pages from the previous crawl no longer exist.

just took a look at your sites...

ya, i think what i said before is what your problem is... (your sites use session ids...)

2 ways actually... I have my own stats script monitoring my site so I can see hits as they happen. And the other way is with Google's sitemaps. It actually shows the crawl rate, complete with line graphs.

Naughty-Pages, I have no forum on my site that's being crawled, and the site I'm talking about isn't in my sig. It's definitely not a session id since my site doesn't use sessions.

I dont think it was a fuck up, cause otherwise theyd have a backup. But I do agree thats prob reindexing/deepcrawling porn sites again for fresh results.

I dont see where I can get that info off google sitemaps...can you pooint me in the right direction?

nevermind, found it by accident. :)

IMO googles ranks for porn sites lately has been awful. I've typed in a few select aduly keywords and all im getting are news sites and other useless non-adult stuff.

does your forum use vbulletin?
we have one forum and when doing command "site:my-forum.com" google shows at first that it found 39 500 pages , but in reality it has only 70 pages from that domain, mainly these that arent generated by vbulletin.
I was just thinking where is the problem.

:1orglaugh :1orglaugh :thumbsup

nice one.

You are in control of Googles craw rate. That is, you can slow it down if you like.

Google has been crawling my sites non-stop, for the past 8 years.

YARGH! Yup within the sitemap section of google's webmaster tools area.

I realize that, but that won't solve the problem nor answer my questions.

lol. what's that all aboot? Time to toss on the Canadian Tuxedo and brave the cold. I need some new candles for my igloo eh

There is growing belief that Google is just plain broke. As in doesn't work! All fucked up.... How many horror stories do you have to read on the SEO forums before you believe. How many companies have to be driven out of business because they loose their Google placement for no reason that can be discerned.

Hidden links and text still affect SERP as does .edu and .gov spamming.

The notion that a site should be ranked on popularity as it relates to "quality" back links is flawed IMO.

Yes, I know Google provided a means for many companies to succeed without paying for the placement on key cash words. So in that statement lies the moral to the story. Don't rely on google for your traffic.

Still going...

http://www.9xs.net/pics/google1.jpg

good result:)

I found that google does not necessarily update the info every day. The graph you're referring to can be the same for several days, then all of a sudden catch up. Also the seven day moving average report on query stats is the same way. Maybe you are having a different experience. Also see some strange reports. Like I had an average of the No. 1 position for the word Ebony. There is no way. Don't have an Ebony site nor is it a keyword pushed on my sites. Plus a check confirmed no hits on that keyword.

No, I know the graph isn't updated in real time, but look at it... it's showing non stop crawling. It doesn't let up at all, not for a minute, much less any days or anything.

I don't care much for whether or not I get a #1 result... right now I'd just like to be listed. But it doesn't appear to be happening since the crawling never stops.

I actually had a problem with Google doing this - it was pulling GIGS every day on a Blog Host I run. Sometimes 3 or 4 hundred thousand hits a day.

It turned out that I had managed to 'break' it with some fucked up code in my Robots.txt file.

Go to Google Webmaster tools and do a check on your robots.txt

bump for my question.

if you arnt indexed after a month thats not good

Tell me about it. I've made a lot of sites and they've all been indexed pretty quickly.
This is definitely a first for me.