How does Google spider all the pages
 

How does Google spider all the pages on the web site?

yes

link to them.

Hope you have some free time.

http://www.webmasterworld.com

They outsource the work...

Just like a lot of designers here on GFY

its magic

"It's kind of magic", Connor MacLeod from Highlander :thumbsup

The use hundreds of linux boxes.

(really)

They feed their spiders a lot of flies.
Makes spiders big and strong.
Spiders run really fast all over web.
Spiders bring back webpages and feed to mother Google.
Mother google becomes happy and gives flies as reward.

Repeat.
WG

manually

start off DMOZ for websites, crawl from there...

Charles You forgot yourself... papa spider... who fucks with the little spiders and gets them to rank certain webpages Higher than others.

copy-paste pidgeons

Spiders and pidgeons. What will they think of next? Don't tell anyone but I heard they're putting the finishing touches on some top secret Jackalope project.

Shhhhh! Don't tell people about that. I only sleep with Googlebot, I gotta make sure mother Google doesn't find out or she'll ban my ass from ever using her spiders again!

WG

I hear these guys http://www.google.com.au/ use kangaroo's to fetch their pages.

WG

Look at this site
http://www.gritechnologies.com/tools...ourself.comcom