unix pagerank thingy

**WiredGuy** · 08-25-2003, 01:52 PM

Can I have it?
WG

**igneus** · 08-25-2003, 02:23 PM

hrm maybe not, lol

just ran it on a list of 500k domains and it went so fast i had to reset my router (router dies when i make a bunch of quick connections to something for some reason)

but i checked the list it came up with and its all there--sooo 500k in under 3mins is like 2777/sec

think i'll stop using this before the google police come and take me away 8-(

**Marc P** · 08-25-2003, 02:34 PM

Can't you slow it down, make it pause very now and then for a while, and make it query at random intervals?

The funny thing is we built the part that acts like a human "talking" to a web server, but the PR part of it we have not tried yet, it looks like quite a bit of work. We built it in java though.

Maybe we could help one another? Let me know if you're interested.

icq: 151902982

**igneus** · 08-25-2003, 02:39 PM

hmm yes, i'll have it pause...i was too busy doubting and trying to disprove the 2777/sec to start trying to slow it down hehe

all that time getting it to go fast, now i gotta work on making it slower, lol

**WiredGuy** · 08-25-2003, 02:40 PM

Why not just use it with lots and lots of proxies and spread the load over and reduce the number of threads to slow it down?

WG

**igneus** · 08-25-2003, 02:43 PM

another good option hehe

what i meant was "using it at these same settings"

proxies might slow it down too much, but ill look at it when i get back from my first awesome class of the fall semester, woo!!

education

**WiredGuy** · 08-25-2003, 02:52 PM

Originally posted by igneus
another good option hehe

what i meant was "using it at these same settings"

proxies might slow it down too much, but ill look at it when i get back from my first awesome class of the fall semester, woo!! education

How about you forward me that source code and I'll fix it up for ya

I'll even port it to perl for ya

WG

**goBigtime** · 08-25-2003, 02:58 PM

Raises hand to offer the php port

**Marc P** · 08-25-2003, 03:01 PM

I'm interested in licencing your application (when it works safely), I sent you an ICQ request about it.

**igneus** · 08-25-2003, 03:51 PM

Originally posted by goBigtime
Raises hand to offer the php port

i did the php version first, hehe...was quite slow 8-/

i'll get back to you when i get home Marc P, i'm in my awesome class right...NOW (gotta love computer classes and all the learning you do in them)

**igneus** · 08-25-2003, 05:28 PM

i didn't get your message Marc P (probably trillian's fault), so can you send again? i'm back on icq now

**TheJimmy** · 08-25-2003, 05:30 PM

Originally posted by WiredGuy

How about you forward me that source code and I'll fix it up for ya

I'll even port it to perl for ya
WG

what he said

**sacX** · 08-25-2003, 05:33 PM

make sure it's unreasonably expensive please.

**igneus** · 08-26-2003, 08:44 AM

alright, added in the pausing and even with a tiny pause it still slows it down a lot:

[chris ~/pagerank/c]$./load.sh split_\* 0; sleep 60; wc -l ranksplit_0*; killall pagerank
164201 total

[chris ~/pagerank/c]$./load.sh split_\* 10; sleep 60; wc -l ranksplit_0*; killall pagerank
72805 total

[chris ~/pagerank/c]$./load.sh split_\* 250; sleep 60; wc -l ranksplit_0*; killall pagerank
6215 total

(starts 50 processes, waits a minute then sees how many domains are in the output file--i tried 0ms, 10ms and 250)

even 250ms brings it down to about 100/sec, so i think it should be fine now

**Lane** · 08-26-2003, 08:57 AM

did u have to crack the encoding?

**igneus** · 08-26-2003, 09:04 AM

converted the checksum code from the toolbar disassembly to c++, yeah..but i did that a while ago with the windows version hehe

**fnet** · 08-26-2003, 09:47 AM

hey congrats

**Marc P** · 08-26-2003, 10:04 AM

I'll be the first one to support the "make it unreasonably expensive" comment.

**WiredGuy** · 08-26-2003, 10:36 AM

Still haven't received that email

Hehehe, just playing with ya man, congrats on breaking their algo.

WG

**nuclei** · 08-28-2003, 04:49 AM

did someone say unix page ranker?

bet this is gonna piss some people off:

http://4retards.com/pagerank

**nuclei** · 08-28-2003, 04:53 AM

igneus: question: how did you get around the fact that googles checksum uses diff starting digits for some major sites such as yahoo, altavista etc?

**nuclei** · 08-28-2003, 03:51 PM

igneus? =)

**igneus** · 08-29-2003, 02:09 AM

gfy moves too fast..didnt see 8-)

newest version of the toolbar only uses 6..unlike 5 and 6 on the older one

also gives 10 ranks if you do it without the info: before the url. not sure if that was always there..never tried 8-)

**nuclei** · 08-31-2003, 02:00 AM

Originally posted by igneus
gfy moves too fast..didnt see 8-)

newest version of the toolbar only uses 6..unlike 5 and 6 on the older one

also gives 10 ranks if you do it without the info: before the url. not sure if that was always there..never tried 8-)

yahoo and others still start out with 1's.

But it does look like I need to port this entire app to C. Even in perl with the checksum calc in C it is too slow for my taste. Ahh well, a release will out in days.

**JamesK** · 08-31-2003, 02:07 AM

Originally posted by TheJimmy

what he said

what he said

**igneus** · 09-01-2003, 03:43 PM

Originally posted by nuclei
yahoo and others still start out with 1's.

But it does look like I need to port this entire app to C. Even in perl with the checksum calc in C it is too slow for my taste. Ahh well, a release will out in days.

how do you get 1? mine still uses 6 on yahoo

http://www.google.com/search?client=...www.yahoo.com/

unix pagerank thingy

unix pagerank thingy

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment