Quote:
Originally Posted by pussyserver
I meant to say index first not spider first
it spiders everything by default ... but the algo decides on what gets indexed and in what order
|
actually I'm not sure about that "spidering everything".... googlebot's algorithm makes decisions on what to spider, especially with newer sites..... a site that gets huge fast will not be completely spidered right away, or completely over and over, it would take up too much bandwidth and I think googlebot puts a relevancy factor on sites to decide how much time to spend on them
I've noticed a major shift in googlebot's spidering activities seems to occur with some sites when they hit a certain stage.... eg. site with 10,000 pages that google doesn't have trust in yet might only get 20% spidered and 5% indexed; then all of the sudden some months down the road much more will get spidered and indexed
(just from what I have seen, other factors like penalties may have been involved I guess, but no way of knowing that from where I'm sitting.... try it if you want to test it, start a brand new site and add 10,000 pages to it right away and see if googlebot spiders the whole works... I doubt that it will, at least not right away, and for sure google won't index all 10,000 pages right away either)