SEO test
 

If you register a new domain put up a site with 5 pages , the main page and 4 pages linked from the main domain

the 4 pages are named

apple.html
pear.html
squash.html
tomato.html

what page will google spider first ?

The index?

lol i was just thinking i should elaborate or some smartass will reply " the index " :)

assuming the index is already listed has one backlink and none of the other pages have backlinks other than from the main page of the domain

:winkwink:

Glad to be at your smartass service. :upsidedow

Which page is the first link Google sees on the main page?

Robots.txt
WG

If listed in order.. apple.html ?

grape.html ;-)

Smokey, hit me up please.

all four simultaneously :2 cents:

What ever fruit Google Labs says is more popular at the time...

Okay i'll back that up.... Apple would be first to index...

http://www.nnteenmodels.net/gfy/apst-gfy.jpg

Use Robot.txt

Why leave things to chance? Tell the fucker where to go.

same order as listed lets say

no robots.txt

as far as i am aware you cant tell google what order to crawl your site

i will narrow it down .

5 choices.

#1 Google will spider the links in alphabetical order.
#2 Google will spider the links by length.
#3 Google will spider the links in the order it found them.
#4 Google will spider the links by importance of keyword
#5 Google will spider them randomly.

:thumbsup

whichever shows up first in the page source so #3

the answer is PEAR.html

reason = #2

Google will crawl the first link indicated in your index page.

IF this is the order you used in your index page then google will crawl apple.html first.

apple.html
pear.html
squash.html
tomato.html

in my experience google uses the url length . it starts with the shortest urls first

I just did a test to confirm this. I dropped 500 links to a dynamic page, each page crawled creates a hardfile containing the first person to view it. when i sort the hardfiles by date they also are sorted by length. i.e. google crawled the shortest urls first in order to longest

We can't be 100% sure which url it'll spider first. But I pretty much think it uses heuristic method in doing that.

Borrrrrrrrrrrrrring.

Is this just from self-testing or something more formal?

My last test I had about 800 pages in total, all pages were of the format:
http://www.domain.com/Category_Name/
http://www.domain.com/Category_Name/index1 - index5 (.shtml)
http://www.domain.com/Category_Name/8-Character-Code (.shtml)

I noticed the first 80 pages indexed were full from the last set, domain, category, 8-character-code.shtml. It was never the category or index pages that got indexed first.

WG

those are dirs, not all same leveled pages in smokeys example. So maybe the deepest dir pages get indexed first? And then their parents?

So the shortest, deepest leveled pages, get indexed first?

longer names get love last.

assuming google does spider each link... within a timely manner for each...

is there an advantage to having google spider certain pages first?

unfortunately no lol, or none i can think of offhand

Then quit wasting our god damned time smokey the fucking bear!!! :mad::mad:

You insignificant trivia feedin mother fucker!!!


Your deck looks like shit!!!!

:pimp

Funny, I was thinking the same thing but was going to be a bit more diplomatic about it :1orglaugh

So you are suggesting that the spider takes the time to look at all the links, then sort in order of length?

you are all wrong

Google will spider the link most relative to the content of the index page first...

that being said... there is no such thing as spider first sice the google bot is in fact millions of bots that all spider constantly

then 2 pages or 20 pages on a site can all be indexed at the same time

the page most relative to the index snd thus the backlink pointing to the index will recieve the most 'love"

How will it know which is most relevant without spidering all first?

I meant to say index first not spider first

it spiders everything by default ... but the algo decides on what gets indexed and in what order

the first page who will give way for Google to visit your page.

That wasn't the question.

http://www.gofuckyourself.com/showpo...88&postcount=1

yes i know that wasnt the question... but since the original question didnt make any sense .... I assumed the OP meant index... not spider

actually I'm not sure about that "spidering everything".... googlebot's algorithm makes decisions on what to spider, especially with newer sites..... a site that gets huge fast will not be completely spidered right away, or completely over and over, it would take up too much bandwidth and I think googlebot puts a relevancy factor on sites to decide how much time to spend on them

I've noticed a major shift in googlebot's spidering activities seems to occur with some sites when they hit a certain stage.... eg. site with 10,000 pages that google doesn't have trust in yet might only get 20% spidered and 5% indexed; then all of the sudden some months down the road much more will get spidered and indexed :2 cents:

(just from what I have seen, other factors like penalties may have been involved I guess, but no way of knowing that from where I'm sitting.... try it if you want to test it, start a brand new site and add 10,000 pages to it right away and see if googlebot spiders the whole works... I doubt that it will, at least not right away, and for sure google won't index all 10,000 pages right away either)

again you are talking about indexing

in order for a url to be process by the algorithm it has to be spidered

The googlebot spiders ( crawls ) everything

But not everything is added to the index

you see in order to really understand SEO you must first understand what google is

Google is a massive automated index... nothing more

just an index

But I should proably stop as I get passionate about SEO and wouldnt want to give anything away :)

I was talking about spidering and indexing

I think googlebot will not completely spider a new site that grows extremely fast, at least not right away

I've noticed that before but I wasn't paying close enough attention to say it with certainty and there are always other possible variables to consider

maybe things have changed now and googlebot will completely spider a brand new 10,000 page site immediately, just saying what I noticed in the past

:thumbsup

wtf ? obviously something gets hit first , google does not simultaniously spider multiple pages, yes they do use multiple bots but they dont all pound your site at the same time or it would crash your site. Your server works the same way , it doesnt handle all tasks at once, it processes them at the same time yes , but it still takes them in an order

exactly.

basically if it finds 200 new pages it records the urls , then schedules the order they will be spidered by length.

The answer to the question is


There is no money in SEO

the spider does whatever it wants, whenever it wants , however way it wants to do it that day/hour/min/sec :2 cents:

... enjoy

Anyone tested this?