A little GFY Background....
 

So, for the last week I have sat here and read several posts about search being down and other GFY related problems.

Some folks have been supportive, but others have kind of shit on Lensman and Jupiter.

First off, for those who have lent support, it is appreciated.

For those who have shit on why search is down, and why the site gets slow, I say before you open your mouth and profess to understand the workings of this board, read below.

GFY is no doubt a popular board. Several folks think they have something they can add by building tools like outside search engines and spiders to try and make a quick buck or earn a good rap.

To those who do that, do you realize that that this is a main cause of issues on the board?

For example, yesterday, we had over 15 spiders crawling the site. Clicking every thread for the last year and indexing every new post. Aggressive spiders and screen scrapers alone contribute to an estimated 15x load on the site as a whole over regular users.

Now lets take the search. Search is directly mapped to spiders who provide search services for keywords. Are you getting a clue why GFY Search maybe down at this time?

Let's add in the weekly DDoS attacks from some kiddie who was pissed he got called a "kiddie", it takes a full time well paid engineer to just keep the site up and running.

Ease up. Lensman has thrown some serious cash at this to both stop the problems from happening, and to shore up the infrastructure. It's a daily effort and I were you I would thank him.

Juicy is one SEXY mother fucker! :Graucho

damn right

No negative feedback on this end. Just by seeing the postcounts and watching topics get buried 2 pages back within 5 minutes, I can easily tell that this site takes some work to keep up. Had no idea about those details but its good to see a website with this much traffic to have a professional team and webmaster to keep the gears turning.

It's sad, that for some, it's easier to spew criticism then anything positive.

Just wanted to say that and give a worthy post a bump.

Cheers.

As long as the board is up its all good.

So we wait for the search.......chilllllllllllllllllllllllllllllllll

Be happy we have place to post holla back at me

Big Ray nice post :thumbsup That's exactly the kind of answer everyone needed... Now everyone STOP ICQING THE SHIT OUT OF ME ABOUT IT OR I'LL KILL YOUR PETS!!

whew... ok carry on...

Thanks RRRED i am blushing now

block all the spiders and setup a search server outside of the normal one
shutdown for 15minutes at 3am when nothing but 5 boardwhores posting ******* pictures are online and let it update.

biggups to GFY! :thumbsup

Give me info on this "script kiddie"

I know people who know people.....holla back at me.

For those haters, you cannot put a big board like GFY down.

GFY rocks!

:smokin

I plan on having my kids and grandchildren posting here!!!!!!!!

I am just itching to know this. How much space in GB or TB does the databases for this board take up?

I appreciate the efforts, and can understand the nightmares. THANK YOU!!!

http://www.texasdreams.com/bananajupiter.gif http://www.texasdreams.com/bananajupiter.gif http://www.texasdreams.com/bananajupiter.gif http://www.texasdreams.com/bananajupiter.gif http://www.texasdreams.com/bananajupiter.gif

That banana and a nice link to Jupiter Hosting can be found at...

http://www.texasdreams.com/dance5.html

:thumbsup

15x load, add more ram in the sql server. how many system admins do you have on staff. other than the ones that have quit?

what do you do at jupiter incase of a DDOS attack. anybody knows you nullroute the ip. but you ofcourse know that right?

i have worked in many datacenters before. with much bigger servers bigger applications. this is smalltime compared to what you are dealing with. when a fortune 500 company has 1 min of downtime they are up your ass. just imagine if you were hired to setup a forum for a companies internal lan. if you cant successfully do it on the outside. why would someone hire you to do it on the inside.

I am telling loryn.

Any idea when the search will be up again?

Right On :thumbsup


Ray nice post bro :)

have you hear about robot.txt? just tell the spiders what to search, and what not to search.. The perfect solution would be to build a "SE archive" from the daily backup located on an other server, and then let the Spiders look at that one, and not the live site

do you think all spiders/site scrapper abide by the robots.txt file?

LOL. Fris. You know nothing of us, myself included, or our employees (past or present). You no nothing of what we run here or how we run it.

Why don't you go back to scripting kiddie.

thanks for the info!

if people are worried about tracking threads then they should just bookmark important threads and check them, not an easy issue for Lens and the rest to keep this going and still maintain the quality speeds of the board.

15x load coming up, lol.
WG

On a serious note, I do like GFY and I have seen it proposed here a few times, what about installing some sort of OCR on the search page? From the description I read that you posted, the search is the most vulnerable area an attacker would do to bring GFY to a crawl, so putting an OCR would make the most sense to stop all scripts? I'm not sure how easy it would be to integrate into GFY but I imagine it would relieve efforts significantly.

WG

Big Ray, thank you for the post!

I'm concerned. Having search off is a definite drawback - massive amounts of resources that can not be accessed.
How do spiders affect the search? Normally siders go through the links, the existing pages. Spiders shouldn't and cannot perform direct searches unless spider is targeted to use the search. I don't know what the case is. But like WG said, image verification is a simple solution - has it been tried before?

If spiders cause heavy load on search, prevent spiders from using it.
If search by itself takes a big load then something can be done with the database and sql queries.

am I reading right, that sites like www.s e a r c h t h e b o a r d s.com don't help it? why not ban it's ip and not allow it to search on here?

Who are you talking about... AFAIK there is me, BT, and boardscanner and unless im missing anything they're not linked to your search function.

JaceXXX, if GFY were to ban your IP, would you:
a) Get another IP?
b) Stay banned?

I just can't help but think that National Net's LBBV could easily fix this situation without always needing to throw piles of cash at it.

Search could EASILY be approved for a certain membership level, say 5k+ posts for example. While being disallowed for all others.

A spider trap could be implemented to automatically block spiders that don't obey robots.txt

Like Big Ray has said many times to many people, you just don't know the whole picture...

And since you're a part of the problem, you're not part of the solution... :321GFY

Says the guy with only 1 bot to the guy with 2 bots, "you're the problem."
Says the guy with only 2 bots to the guy with 3 bots, "you're the problem."
Says the guy with only 3 bots to the guy with 4 bots, "you're the problem."

^^^ notjoe, which guy are you?

That is a misconception..the search on STB is completely independant of the GFY search issues and AFAIK had absolutely nothing to do with the problems happening recently..I dont consider 8.3k page loads (from memory) much..

If there was infact 15 spidering running on GFY all you would need to do is subtract 15 from the total views of each thread to get the amount of views not including the bots.

The search on STB spans about 7 different database servers which is why we're able to search 14+ million posts within a fairly reasonable amount of time. Thats also the reason we're spidering mainstream boards such as dbforums.com, maxbimmer.com, bodybuilding.com and a few others...

To date, our of the 40+ boards i've been spidering for the last few months i havent crashed any of them or had any admins accusing me of DOS'ing them, except for GFY.

Why dont you tell me. I can tell you that im not the guy who does "Now lets take the search. Search is directly mapped to spiders who provide search services for keywords. Are you getting a clue why GFY Search maybe down at this time? " as Ray put it :321GFY

good job