how can google do all those quick searches? - GoFuckYourself.com

mrthumbs · 03-26-2004, 12:16 AM

dont give me the cache explanation because cache needs to be indexed to..

How does google search billions of pages within a fraction of a second.. not only that is amazing.. she does it for 10 thousands simultaneously.

Is there a secret or is it just machine power?

psyko514 · 03-26-2004, 12:17 AM

It's all about the Pigeon Rank technology.

http://www.google.com/technology/pigeonrank.html

StuartD · 03-26-2004, 12:17 AM

rooms and rooms full of servers... I figure they gotta have a ton of machines all working together to perform all those searches and indexing and everything else.

psyko514 · 03-26-2004, 12:19 AM

<h3>The technology behind Google's great results</h3>
As a Google user, you're familiar with the speed and accuracy of a Google search. How exactly does Google manage to find the right results for every query as quickly as it does? The heart of Google's search technology is PigeonRank?, a system for ranking web pages developed by Google founders Larry Page and Sergey Brin at Stanford University.

Building upon the breakthrough work of B. F. Skinner, Page and Brin reasoned that low cost pigeon clusters (PCs) could be used to compute the relative value of web pages faster than human editors or machine-based algorithms. And while Google has dozens of engineers working to improve every aspect of our service on a daily basis, PigeonRank continues to provide the basis for all of our web search tools.

Mr. Marks · 03-26-2004, 01:41 AM

probably spider machine power.

Rorschach · 03-26-2004, 01:45 AM

http://www7.scu.edu.au/programme/ful...21/com1921.htm

woj · 03-26-2004, 01:56 AM

Quote:

Originally posted by Rorschach
http://www7.scu.edu.au/programme/ful...21/com1921.htm

I was about to post that... it's a good read

Jamief · 03-26-2004, 02:01 AM

Quote:

When a search query is submitted to Google, it is routed to a data coop where monitors flash result pages at blazing speeds. When a relevant result is observed by one of the pigeons in the cluster, it strikes a rubber-coated steel bar with its beak, which assigns the page a PigeonRank value of one. For each peck, the PigeonRank increases. Those pages receiving the most pecks, are returned at the top of the user's results page with the other results displayed in pecking order.

Is that just a stupid joke or something

Rorschach · 03-26-2004, 02:02 AM

Quote:

Originally posted by Jamief

Is that just a stupid joke or something

You saved up your first post to say something retarded?

johnbosh · 03-26-2004, 02:03 AM

the pigeon idea really works

Ash@phpFX · 03-26-2004, 02:11 AM

interesting read there

the Shemp · 03-26-2004, 02:13 AM

i always though they just had a ton of horsepower

beemk · 03-26-2004, 02:14 AM

Quote:

Originally posted by Jamief

Is that just a stupid joke or something

nope its real..... just like bangbus

DarkJedi · 03-26-2004, 02:19 AM

They got tens of thousands of servers.

vapewiz · 03-26-2004, 02:22 AM

Quote:

Originally posted by beemk
nope its real..... just like bangbus

Garou · 03-26-2004, 03:05 AM

wow! google was created by Russian programmist

)

valley hina · 03-26-2004, 03:09 AM

Quote:

Originally posted by Garou
wow! google was created by Russian programmist )

for real i didnt know that!!

Crazy you learn shit everyday

SomeCreep · 03-26-2004, 03:13 AM

Quote:

Originally posted by mrthumbs
dont give me the cache explanation because cache needs to be indexed to..

How does google search billions of pages within a fraction of a second.. not only that is amazing.. she does it for 10 thousands simultaneously.

Is there a secret or is it just machine power?

Yes, the algorithms are a secret. They use "advanced" computer science concepts and programming. Other than that, it gets very technical.

Serge Litehead · 03-26-2004, 03:22 AM

Quote:

Originally posted by DarkJedi
They got tens of thousands of servers.

literally

-----------------
http://google.com/bsd

foolio · 03-26-2004, 03:35 AM

google is one of thise things that as long as it works I dont care how it works

chrisp420 · 03-26-2004, 04:19 AM

Quote:

Originally posted by psyko514
<h3>The technology behind Google's great results</h3>
As a Google user, you're familiar with the speed and accuracy of a Google search. How exactly does Google manage to find the right results for every query as quickly as it does? The heart of Google's search technology is PigeonRank?, a system for ranking web pages developed by Google founders Larry Page and Sergey Brin at Stanford University.

Building upon the breakthrough work of B. F. Skinner, Page and Brin reasoned that low cost pigeon clusters (PCs) could be used to compute the relative value of web pages faster than human editors or machine-based algorithms. And while Google has dozens of engineers working to improve every aspect of our service on a daily basis, PigeonRank continues to provide the basis for all of our web search tools.

massive clusters will yeild anybody great results. their datacenter machine layouts are amazing.

nastyking · 03-26-2004, 04:52 AM

Writing Powerful Programs in C and C++

Using Technologies to avoid slow Hard disk Operations (Big Files)

Using Stuff like Lexicon, Word IDs, etc ..

Every word found in the document is turned into an integer number (wordID)

When somebody searches for "gay seals" it's turned for example into 123 and 653. Then Google looks for all documents matching this two Word IDs. This way they get around SLOW string comparisons ....

To rank the site Google uses a Hitlist.

For every document found it calculates the IR. If "seals" is for example in Hn tags it's a greater Hit as if it would be in normal P tag ...

Then it uses factors like PR, IBLs, anchor text, proximity of keywords and a lots of stuff we do not know to give it the final touch

This powerful piece of software is than run on 1000 of cheap servers ...

Full Paper

montel · 03-26-2004, 04:55 AM

The staff at google are extremely smart. They have more Phd's per square foot there than any institution I can think of.

sexsup · 03-26-2004, 04:58 AM

Quote:

Originally posted by Garou
wow! google was created by Russian programmist )

I always knew it

Mike-BP · 03-26-2004, 05:11 AM

it's magic

NetRodent · 03-26-2004, 06:17 AM

Quote:

Originally posted by SomeCreep

Yes, the algorithms are a secret. They use "advanced" computer science concepts and programming. Other than that, it gets very technical.

The ranking algorithims are secret, how they serve up results quickly has been published. There have been several articles published about how their clusters are organized, their use of ram disks, etc. I'm too lazy to dig them all up, here's one I happened to have handy:

http://www.cs.rochester.edu/sosp2003...5-ghemawat.pdf

If you want to find the others, try searching on Google.

mardigras · 03-26-2004, 08:10 AM

One of the prime time news magazines showed the Google servers awhile back. They were huge, went down a corridor and were behind chain-link fence.

mrthumbs · 03-27-2004, 09:43 AM

thank you all for the explanation. it makes sense now

Intrigue · 03-27-2004, 10:19 AM

There servers also use a properiatary filesystem known as GFS which is capable of searching and indexing multiple terabytes of data very effeciently, also developed by the magnificient bastards at google!

uvort · 03-27-2004, 10:26 AM

none the less, its pretty fuckin crazy how then can do that..

03-26-2004, 12:16 AM	#1
mrthumbs salad tossing sig guy Join Date: Apr 2002 Location: mrthumbs*gmail.com Posts: 11,702	how can google do all those quick searches? dont give me the cache explanation because cache needs to be indexed to.. How does google search billions of pages within a fraction of a second.. not only that is amazing.. she does it for 10 thousands simultaneously. Is there a secret or is it just machine power?

03-26-2004, 12:17 AM	#2
psyko514 See sig. Join Epic Cash. Join Date: Oct 2002 Location: Montreal, Quebec. ICQ: 214702014 Posts: 22,366	It's all about the Pigeon Rank technology. http://www.google.com/technology/pigeonrank.html __________________ Bad Girl Bucks - 50% Revshare through CCBill. Promote BrandyDDD, Pixie's Pillows, Action Allie and more! Phoenix Forum Pics \| Webmaster Access Montreal pics email: psyko514(a)gmail.com \| icq: 214-702-014

03-26-2004, 12:17 AM	#3
StuartD Sofa King Band Join Date: Jul 2002 Location: Outside the box Posts: 29,903	rooms and rooms full of servers... I figure they gotta have a ton of machines all working together to perform all those searches and indexing and everything else. __________________ This is me on facebook This is me on twitter

03-26-2004, 12:19 AM	#4
psyko514 See sig. Join Epic Cash. Join Date: Oct 2002 Location: Montreal, Quebec. ICQ: 214702014 Posts: 22,366	<h3>The technology behind Google's great results</h3> As a Google user, you're familiar with the speed and accuracy of a Google search. How exactly does Google manage to find the right results for every query as quickly as it does? The heart of Google's search technology is PigeonRank?, a system for ranking web pages developed by Google founders Larry Page and Sergey Brin at Stanford University. Building upon the breakthrough work of B. F. Skinner, Page and Brin reasoned that low cost pigeon clusters (PCs) could be used to compute the relative value of web pages faster than human editors or machine-based algorithms. And while Google has dozens of engineers working to improve every aspect of our service on a daily basis, PigeonRank continues to provide the basis for all of our web search tools. __________________ Bad Girl Bucks - 50% Revshare through CCBill. Promote BrandyDDD, Pixie's Pillows, Action Allie and more! Phoenix Forum Pics \| Webmaster Access Montreal pics email: psyko514(a)gmail.com \| icq: 214-702-014

03-26-2004, 02:13 AM	#12
the Shemp congrats to the winners Industry Role: Join Date: Nov 2001 Location: Echo Beach Posts: 10,891	i always though they just had a ton of horsepower __________________ i use Vacares...so should you Submit your picture galleries to my site...Outlaw TGP

03-26-2004, 01:41 AM	#5
Mr. Marks Confirmed User Join Date: Jun 2001 Location: Wherever I want Posts: 7,517	probably spider machine power.

03-26-2004, 01:45 AM	#6
Rorschach So Fucking Banned Join Date: Aug 2002 Posts: 5,579	http://www7.scu.edu.au/programme/ful...21/com1921.htm

03-26-2004, 02:03 AM	#10
johnbosh Confirmed User Join Date: Aug 2002 Location: The Netherlands, Rotterdam Posts: 8,965	the pigeon idea really works

03-26-2004, 02:11 AM	#11
Ash@phpFX Confirmed User Join Date: Nov 2003 Posts: 4,292	interesting read there

03-26-2004, 02:19 AM	#14
DarkJedi No Refunds Issued. Industry Role: Join Date: Feb 2001 Location: GFY Posts: 28,300	They got tens of thousands of servers.

03-26-2004, 03:05 AM	#16
Garou Confirmed User Join Date: Mar 2003 Posts: 850	wow! google was created by Russian programmist )

03-26-2004, 03:35 AM	#20
foolio ICQ: 178725656 Join Date: Nov 2002 Location: Sunny San Diego Posts: 12,366	google is one of thise things that as long as it works I dont care how it works __________________

03-26-2004, 04:52 AM	#22
nastyking Join Date: Nov 2002 Posts: 2,174	Writing Powerful Programs in C and C++ Using Technologies to avoid slow Hard disk Operations (Big Files) Using Stuff like Lexicon, Word IDs, etc .. Every word found in the document is turned into an integer number (wordID) When somebody searches for "gay seals" it's turned for example into 123 and 653. Then Google looks for all documents matching this two Word IDs. This way they get around SLOW string comparisons .... To rank the site Google uses a Hitlist. For every document found it calculates the IR. If "seals" is for example in Hn tags it's a greater Hit as if it would be in normal P tag ... Then it uses factors like PR, IBLs, anchor text, proximity of keywords and a lots of stuff we do not know to give it the final touch This powerful piece of software is than run on 1000 of cheap servers ... Full Paper __________________ Last edited by nastyking; 03-26-2004 at 04:54 AM..

03-26-2004, 04:55 AM	#23
montel Confirmed User Industry Role: Join Date: Jul 2001 Location: See sig! Posts: 2,606	The staff at google are extremely smart. They have more Phd's per square foot there than any institution I can think of. __________________ Barefootsies

03-26-2004, 05:11 AM	#25
Mike-BP Confirmed User Join Date: Mar 2003 Posts: 1,270	it's magic __________________ Thanks to: ISPrime.com - best hosting company without downtime

03-26-2004, 08:10 AM	#27
mardigras Bon temps! Join Date: Feb 2003 Location: down yonder Posts: 14,194	One of the prime time news magazines showed the Google servers awhile back. They were huge, went down a corridor and were behind chain-link fence. __________________ .

03-27-2004, 09:43 AM	#28
mrthumbs salad tossing sig guy Join Date: Apr 2002 Location: mrthumbs*gmail.com Posts: 11,702	thank you all for the explanation. it makes sense now

03-27-2004, 10:19 AM	#29
Intrigue Confirmed User Join Date: Feb 2004 Location: Ohio, US Posts: 662	There servers also use a properiatary filesystem known as GFS which is capable of searching and indexing multiple terabytes of data very effeciently, also developed by the magnificient bastards at google!

03-27-2004, 10:26 AM	#30
uvort Confirmed User Join Date: Mar 2003 Location: Metalcash.com Posts: 3,580	none the less, its pretty fuckin crazy how then can do that.. __________________ See Sig. 199907143