A little GFY Background.... - GoFuckYourself.com

Big Ray · 05-20-2005, 01:52 PM

So, for the last week I have sat here and read several posts about search being down and other GFY related problems.

Some folks have been supportive, but others have kind of shit on Lensman and Jupiter.

First off, for those who have lent support, it is appreciated.

For those who have shit on why search is down, and why the site gets slow, I say before you open your mouth and profess to understand the workings of this board, read below.

GFY is no doubt a popular board. Several folks think they have something they can add by building tools like outside search engines and spiders to try and make a quick buck or earn a good rap.

To those who do that, do you realize that that this is a main cause of issues on the board?

For example, yesterday, we had over 15 spiders crawling the site. Clicking every thread for the last year and indexing every new post. Aggressive spiders and screen scrapers alone contribute to an estimated 15x load on the site as a whole over regular users.

Now lets take the search. Search is directly mapped to spiders who provide search services for keywords. Are you getting a clue why GFY Search maybe down at this time?

Let's add in the weekly DDoS attacks from some kiddie who was pissed he got called a "kiddie", it takes a full time well paid engineer to just keep the site up and running.

Ease up. Lensman has thrown some serious cash at this to both stop the problems from happening, and to shore up the infrastructure. It's a daily effort and I were you I would thank him.

AaronM · 05-20-2005, 01:55 PM

Juicy is one SEXY mother fucker!

Juicy D. Links · 05-20-2005, 02:02 PM

Quote:

Originally Posted by AaronM

Juicy is one SEXY mother fucker!

damn right

bigmack · 05-20-2005, 02:03 PM

No negative feedback on this end. Just by seeing the postcounts and watching topics get buried 2 pages back within 5 minutes, I can easily tell that this site takes some work to keep up. Had no idea about those details but its good to see a website with this much traffic to have a professional team and webmaster to keep the gears turning.

psili · 05-20-2005, 02:03 PM

It's sad, that for some, it's easier to spew criticism then anything positive.

Just wanted to say that and give a worthy post a bump.

Cheers.

Juicy D. Links · 05-20-2005, 02:07 PM

As long as the board is up its all good.

So we wait for the search.......chilllllllllllllllllllllllllllllllll

Be happy we have place to post holla back at me

RRRED · 05-20-2005, 02:11 PM

Big Ray nice post

That's exactly the kind of answer everyone needed... Now everyone STOP ICQING THE SHIT OUT OF ME ABOUT IT OR I'LL KILL YOUR PETS!!

whew... ok carry on...

Juicy D. Links · 05-20-2005, 02:13 PM

Quote:

Originally Posted by RRRED

Juicy , I want to make sweet love to you , I cant stop dreaming about you

Thanks RRRED i am blushing now

bringer · 05-20-2005, 02:15 PM

block all the spiders and setup a search server outside of the normal one
shutdown for 15minutes at 3am when nothing but 5 boardwhores posting ******* pictures are online and let it update.

sixxxthsense · 05-20-2005, 02:17 PM

biggups to GFY!

Juicy D. Links · 05-20-2005, 02:23 PM

Give me info on this "script kiddie"

I know people who know people.....holla back at me.

loverboy · 05-20-2005, 02:24 PM

For those haters, you cannot put a big board like GFY down.

GFY rocks!

Juicy D. Links · 05-20-2005, 02:26 PM

I plan on having my kids and grandchildren posting here!!!!!!!!

bigmack · 05-20-2005, 02:34 PM

I am just itching to know this. How much space in GB or TB does the databases for this board take up?

TexasDreams · 05-20-2005, 02:43 PM

Quote:

Originally Posted by Big Ray

Some folks have been supportive, but others have kind of shit on Lensman and Jupiter.

I appreciate the efforts, and can understand the nightmares. THANK YOU!!!

That banana and a nice link to Jupiter Hosting can be found at...

http://www.texasdreams.com/dance5.html

fris · 05-20-2005, 02:49 PM

Quote:

Originally Posted by Big Ray

So, for the last week I have sat here and read several posts about search being down and other GFY related problems.

Some folks have been supportive, but others have kind of shit on Lensman and Jupiter.

First off, for those who have lent support, it is appreciated.

For those who have shit on why search is down, and why the site gets slow, I say before you open your mouth and profess to understand the workings of this board, read below.

GFY is no doubt a popular board. Several folks think they have something they can add by building tools like outside search engines and spiders to try and make a quick buck or earn a good rap.

To those who do that, do you realize that that this is a main cause of issues on the board?

For example, yesterday, we had over 15 spiders crawling the site. Clicking every thread for the last year and indexing every new post. Aggressive spiders and screen scrapers alone contribute to an estimated 15x load on the site as a whole over regular users.

Now lets take the search. Search is directly mapped to spiders who provide search services for keywords. Are you getting a clue why GFY Search maybe down at this time?

Let's add in the weekly DDoS attacks from some kiddie who was pissed he got called a "kiddie", it takes a full time well paid engineer to just keep the site up and running.

Ease up. Lensman has thrown some serious cash at this to both stop the problems from happening, and to shore up the infrastructure. It's a daily effort and I were you I would thank him.

15x load, add more ram in the sql server. how many system admins do you have on staff. other than the ones that have quit?

what do you do at jupiter incase of a DDOS attack. anybody knows you nullroute the ip. but you ofcourse know that right?

i have worked in many datacenters before. with much bigger servers bigger applications. this is smalltime compared to what you are dealing with. when a fortune 500 company has 1 min of downtime they are up your ass. just imagine if you were hired to setup a forum for a companies internal lan. if you cant successfully do it on the outside. why would someone hire you to do it on the inside.

After Shock Media · 05-20-2005, 02:50 PM

Quote:

Originally Posted by RRRED

Big Ray nice post

That's exactly the kind of answer everyone needed... Now everyone STOP ICQING THE SHIT OUT OF ME ABOUT IT OR I'LL KILL YOUR PETS!!

whew... ok carry on...

I am telling loryn.

tranza · 05-20-2005, 02:55 PM

Any idea when the search will be up again?

Techie Media · 05-20-2005, 02:59 PM

Quote:

Originally Posted by AaronM

Juicy is one SEXY mother fucker!

Right On

Ray nice post bro

Judas_Hansen · 05-20-2005, 03:26 PM

have you hear about robot.txt? just tell the spiders what to search, and what not to search.. The perfect solution would be to build a "SE archive" from the daily backup located on an other server, and then let the Spiders look at that one, and not the live site

bawdy · 05-20-2005, 03:34 PM

Quote:

Originally Posted by Judas_Hansen

have you hear about robot.txt? just tell the spiders what to search, and what not to search.. The perfect solution would be to build a "SE archive" from the daily backup located on an other server, and then let the Spiders look at that one, and not the live site

do you think all spiders/site scrapper abide by the robots.txt file?

Big Ray · 05-20-2005, 03:54 PM

Quote:

Originally Posted by fris

15x load, add more ram in the sql server. how many system admins do you have on staff. other than the ones that have quit?

what do you do at jupiter incase of a DDOS attack. anybody knows you nullroute the ip. but you ofcourse know that right?

i have worked in many datacenters before. with much bigger servers bigger applications. this is smalltime compared to what you are dealing with. when a fortune 500 company has 1 min of downtime they are up your ass. just imagine if you were hired to setup a forum for a companies internal lan. if you cant successfully do it on the outside. why would someone hire you to do it on the inside.

LOL. Fris. You know nothing of us, myself included, or our employees (past or present). You no nothing of what we run here or how we run it.

Why don't you go back to scripting kiddie.

Violetta · 05-20-2005, 05:41 PM

thanks for the info!

Wiggles · 05-20-2005, 06:10 PM

if people are worried about tracking threads then they should just bookmark important threads and check them, not an easy issue for Lens and the rest to keep this going and still maintain the quality speeds of the board.

WiredGuy · 05-20-2005, 06:13 PM

Quote:

Originally Posted by Big Ray

LOL. Fris. You know nothing of us, myself included, or our employees (past or present). You no nothing of what we run here or how we run it.

Why don't you go back to scripting kiddie.

15x load coming up, lol.
WG

WiredGuy · 05-20-2005, 06:15 PM

On a serious note, I do like GFY and I have seen it proposed here a few times, what about installing some sort of OCR on the search page? From the description I read that you posted, the search is the most vulnerable area an attacker would do to bring GFY to a crawl, so putting an OCR would make the most sense to stop all scripts? I'm not sure how easy it would be to integrate into GFY but I imagine it would relieve efforts significantly.

WG

Serge Litehead · 05-20-2005, 06:54 PM

Big Ray, thank you for the post!

I'm concerned. Having search off is a definite drawback - massive amounts of resources that can not be accessed.
How do spiders affect the search? Normally siders go through the links, the existing pages. Spiders shouldn't and cannot perform direct searches unless spider is targeted to use the search. I don't know what the case is. But like WG said, image verification is a simple solution - has it been tried before?

If spiders cause heavy load on search, prevent spiders from using it.
If search by itself takes a big load then something can be done with the database and sql queries.

Jace · 05-20-2005, 06:58 PM

am I reading right, that sites like www.s e a r c h t h e b o a r d s.com don't help it? why not ban it's ip and not allow it to search on here?

notjoe · 05-20-2005, 07:16 PM

Quote:

Originally Posted by Big Ray

Now lets take the search. Search is directly mapped to spiders who provide search services for keywords. Are you getting a clue why GFY Search maybe down at this time?

Who are you talking about... AFAIK there is me, BT, and boardscanner and unless im missing anything they're not linked to your search function.

fuzzypeach · 05-20-2005, 07:17 PM

Quote:

Originally Posted by JaceXXX

am I reading right, that sites like www.s e a r c h t h e b o a r d s.com don't help it? why not ban it's ip and not allow it to search on here?

JaceXXX, if GFY were to ban your IP, would you:
a) Get another IP?
b) Stay banned?

Brujah · 05-20-2005, 07:20 PM

I just can't help but think that National Net's LBBV could easily fix this situation without always needing to throw piles of cash at it.

Brujah · 05-20-2005, 07:22 PM

Search could EASILY be approved for a certain membership level, say 5k+ posts for example. While being disallowed for all others.

A spider trap could be implemented to automatically block spiders that don't obey robots.txt

fuzzypeach · 05-20-2005, 07:22 PM

Quote:

Originally Posted by notjoe

Who are you talking about... AFAIK there is me, BT, and boardscanner and unless im missing anything they're not linked to your search function.

Like Big Ray has said many times to many people, you just don't know the whole picture...

And since you're a part of the problem, you're not part of the solution...

Says the guy with only 1 bot to the guy with 2 bots, "you're the problem."
Says the guy with only 2 bots to the guy with 3 bots, "you're the problem."
Says the guy with only 3 bots to the guy with 4 bots, "you're the problem."

^^^ notjoe, which guy are you?

notjoe · 05-20-2005, 07:24 PM

Quote:

Originally Posted by JaceXXX

am I reading right, that sites like www.s e a r c h t h e b o a r d s.com don't help it? why not ban it's ip and not allow it to search on here?

That is a misconception..the search on STB is completely independant of the GFY search issues and AFAIK had absolutely nothing to do with the problems happening recently..I dont consider 8.3k page loads (from memory) much..

If there was infact 15 spidering running on GFY all you would need to do is subtract 15 from the total views of each thread to get the amount of views not including the bots.

The search on STB spans about 7 different database servers which is why we're able to search 14+ million posts within a fairly reasonable amount of time. Thats also the reason we're spidering mainstream boards such as dbforums.com, maxbimmer.com, bodybuilding.com and a few others...

To date, our of the 40+ boards i've been spidering for the last few months i havent crashed any of them or had any admins accusing me of DOS'ing them, except for GFY.

notjoe · 05-20-2005, 07:29 PM

Quote:

Originally Posted by fuzzypeach

Like Big Ray has said many times to many people, you just don't know the whole picture...

And since you're a part of the problem, you're not part of the solution...

Says the guy with only 1 bot to the guy with 2 bots, "you're the problem."
Says the guy with only 2 bots to the guy with 3 bots, "you're the problem."
Says the guy with only 3 bots to the guy with 4 bots, "you're the problem."

^^^ notjoe, which guy are you?

Why dont you tell me. I can tell you that im not the guy who does "Now lets take the search. Search is directly mapped to spiders who provide search services for keywords. Are you getting a clue why GFY Search maybe down at this time? " as Ray put it

AlienQ - BANNED FOR LIFE · 05-20-2005, 07:30 PM

Quote:

Originally Posted by Big Ray

LOL. Fris. You know nothing of us, myself included, or our employees (past or present). You no nothing of what we run here or how we run it.

Why don't you go back to scripting kiddie.

rounders · 05-22-2005, 05:37 AM

good job

05-20-2005, 01:52 PM	#1
Big Ray Confirmed User Join Date: Dec 2003 Posts: 464	A little GFY Background.... So, for the last week I have sat here and read several posts about search being down and other GFY related problems. Some folks have been supportive, but others have kind of shit on Lensman and Jupiter. First off, for those who have lent support, it is appreciated. For those who have shit on why search is down, and why the site gets slow, I say before you open your mouth and profess to understand the workings of this board, read below. GFY is no doubt a popular board. Several folks think they have something they can add by building tools like outside search engines and spiders to try and make a quick buck or earn a good rap. To those who do that, do you realize that that this is a main cause of issues on the board? For example, yesterday, we had over 15 spiders crawling the site. Clicking every thread for the last year and indexing every new post. Aggressive spiders and screen scrapers alone contribute to an estimated 15x load on the site as a whole over regular users. Now lets take the search. Search is directly mapped to spiders who provide search services for keywords. Are you getting a clue why GFY Search maybe down at this time? Let's add in the weekly DDoS attacks from some kiddie who was pissed he got called a "kiddie", it takes a full time well paid engineer to just keep the site up and running. Ease up. Lensman has thrown some serious cash at this to both stop the problems from happening, and to shore up the infrastructure. It's a daily effort and I were you I would thank him. __________________ Five of the Top 10 Largest Programs Host with Jupiter. Find out why at http://www.jupiterhosting.com Last edited by Big Ray; 05-20-2005 at 02:02 PM..

05-20-2005, 02:03 PM	#5
psili Confirmed User Join Date: Apr 2003 Location: Loveland, CO Posts: 5,526	It's sad, that for some, it's easier to spew criticism then anything positive. Just wanted to say that and give a worthy post a bump. Cheers. __________________ Your post count means nothing.

05-20-2005, 02:15 PM	#9
bringer i have man boobies Join Date: Jul 2003 Location: van down by the river Posts: 13,082	block all the spiders and setup a search server outside of the normal one shutdown for 15minutes at 3am when nothing but 5 boardwhores posting ******* pictures are online and let it update. __________________ 333-765-551

05-20-2005, 02:55 PM	#18
tranza ICQ: 197-556-237 Join Date: Jun 2003 Location: BRASIL !!! Posts: 57,559	Any idea when the search will be up again? __________________ I'm just a newbie.

05-20-2005, 03:26 PM	#20
Judas_Hansen Confirmed User Join Date: Oct 2003 Location: Denmark / Hellerup Posts: 520	have you hear about robot.txt? just tell the spiders what to search, and what not to search.. The perfect solution would be to build a "SE archive" from the daily backup located on an other server, and then let the Spiders look at that one, and not the live site __________________ \| EUROPEAN HOSTING = FORGET 2257 \| IcQ 300-894038 \| anti-SPYWARE \|

05-20-2005, 01:55 PM	#2
AaronM Too lazy to set a custom title Industry Role: Join Date: Oct 2001 Location: ┌∩┐ ◣_◢ ┌∩┐ Posts: 46,909	Juicy is one SEXY mother fucker!

05-20-2005, 02:03 PM	#4
bigmack Confirmed User Join Date: Feb 2005 Posts: 329	No negative feedback on this end. Just by seeing the postcounts and watching topics get buried 2 pages back within 5 minutes, I can easily tell that this site takes some work to keep up. Had no idea about those details but its good to see a website with this much traffic to have a professional team and webmaster to keep the gears turning.

05-20-2005, 02:07 PM	#6
Juicy D. Links So Fucking Banned Industry Role: Join Date: Apr 2001 Location: N.Y. -Long Island -- Posts: 122,992	As long as the board is up its all good. So we wait for the search.......chilllllllllllllllllllllllllllllllll Be happy we have place to post holla back at me

05-20-2005, 02:11 PM	#7
RRRED Confirmed User Join Date: Jan 2001 Location: WA Posts: 6,754	Big Ray nice post That's exactly the kind of answer everyone needed... Now everyone STOP ICQING THE SHIT OUT OF ME ABOUT IT OR I'LL KILL YOUR PETS!! whew... ok carry on...

05-20-2005, 02:17 PM	#10
sixxxthsense Confirmed User Join Date: Aug 2004 Location: Toronto Posts: 2,421	biggups to GFY!

05-20-2005, 02:23 PM	#11
Juicy D. Links So Fucking Banned Industry Role: Join Date: Apr 2001 Location: N.Y. -Long Island -- Posts: 122,992	Give me info on this "script kiddie" I know people who know people.....holla back at me.

05-20-2005, 02:24 PM	#12
loverboy When it rains, it pours Industry Role: Join Date: May 2003 Posts: 20,609	For those haters, you cannot put a big board like GFY down. GFY rocks!

05-20-2005, 02:26 PM	#13
Juicy D. Links So Fucking Banned Industry Role: Join Date: Apr 2001 Location: N.Y. -Long Island -- Posts: 122,992	I plan on having my kids and grandchildren posting here!!!!!!!!

05-20-2005, 02:34 PM	#14
bigmack Confirmed User Join Date: Feb 2005 Posts: 329	I am just itching to know this. How much space in GB or TB does the databases for this board take up?

05-20-2005, 05:41 PM	#23
Violetta Affiliate Join Date: Jul 2004 Posts: 28,735	thanks for the info! __________________ M&A Queen

05-20-2005, 06:10 PM	#24
Wiggles Too lazy to set a custom title Join Date: May 2004 Posts: 14,423	if people are worried about tracking threads then they should just bookmark important threads and check them, not an easy issue for Lens and the rest to keep this going and still maintain the quality speeds of the board. __________________ no sig

05-20-2005, 06:15 PM	#26
WiredGuy Pounding Googlebot Industry Role: Join Date: Aug 2002 Location: Canada Posts: 34,497	On a serious note, I do like GFY and I have seen it proposed here a few times, what about installing some sort of OCR on the search page? From the description I read that you posted, the search is the most vulnerable area an attacker would do to bring GFY to a crawl, so putting an OCR would make the most sense to stop all scripts? I'm not sure how easy it would be to integrate into GFY but I imagine it would relieve efforts significantly. WG __________________ I play with Google.

05-20-2005, 06:54 PM	#27
Serge Litehead Confirmed User Industry Role: Join Date: Dec 2002 Location: Behind the scenes Posts: 5,190	Big Ray, thank you for the post! I'm concerned. Having search off is a definite drawback - massive amounts of resources that can not be accessed. How do spiders affect the search? Normally siders go through the links, the existing pages. Spiders shouldn't and cannot perform direct searches unless spider is targeted to use the search. I don't know what the case is. But like WG said, image verification is a simple solution - has it been tried before? If spiders cause heavy load on search, prevent spiders from using it. If search by itself takes a big load then something can be done with the database and sql queries. __________________ SilverCrocodile.com Adult web design since '05

05-20-2005, 06:58 PM	#28
Jace FBOP Class Of 2013 Industry Role: Join Date: Jan 2004 Location: bumfuck, ky Posts: 35,562	am I reading right, that sites like www.s e a r c h t h e b o a r d s.com don't help it? why not ban it's ip and not allow it to search on here?

05-20-2005, 07:20 PM	#31
Brujah Beer Money Baron Industry Role: Join Date: Jan 2001 Location: brujah / gmail Posts: 22,157	I just can't help but think that National Net's LBBV could easily fix this situation without always needing to throw piles of cash at it. __________________ Free Premium Domain Lists and Tools at Clickmojo.com For Sale: Obscenity.com

05-20-2005, 07:22 PM	#32
Brujah Beer Money Baron Industry Role: Join Date: Jan 2001 Location: brujah / gmail Posts: 22,157	Search could EASILY be approved for a certain membership level, say 5k+ posts for example. While being disallowed for all others. A spider trap could be implemented to automatically block spiders that don't obey robots.txt __________________ Free Premium Domain Lists and Tools at Clickmojo.com For Sale: Obscenity.com

05-22-2005, 05:37 AM	#37
rounders Confirmed User Join Date: Sep 2003 Location: p0rn0stars & h0es Posts: 2,931	good job __________________ ICQ#: 153923840