A little GFY Background....

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Big Ray
    Confirmed User
    • Dec 2003
    • 464

    #1

    A little GFY Background....

    So, for the last week I have sat here and read several posts about search being down and other GFY related problems.

    Some folks have been supportive, but others have kind of shit on Lensman and Jupiter.

    First off, for those who have lent support, it is appreciated.

    For those who have shit on why search is down, and why the site gets slow, I say before you open your mouth and profess to understand the workings of this board, read below.

    GFY is no doubt a popular board. Several folks think they have something they can add by building tools like outside search engines and spiders to try and make a quick buck or earn a good rap.

    To those who do that, do you realize that that this is a main cause of issues on the board?

    For example, yesterday, we had over 15 spiders crawling the site. Clicking every thread for the last year and indexing every new post. Aggressive spiders and screen scrapers alone contribute to an estimated 15x load on the site as a whole over regular users.

    Now lets take the search. Search is directly mapped to spiders who provide search services for keywords. Are you getting a clue why GFY Search maybe down at this time?

    Let's add in the weekly DDoS attacks from some kiddie who was pissed he got called a "kiddie", it takes a full time well paid engineer to just keep the site up and running.

    Ease up. Lensman has thrown some serious cash at this to both stop the problems from happening, and to shore up the infrastructure. It's a daily effort and I were you I would thank him.
    Last edited by Big Ray; 05-20-2005, 01:02 PM.

    Five of the Top 10 Largest Programs Host with Jupiter. Find out why at http://www.jupiterhosting.com
  • AaronM
    GFY Royality ;)
    • Oct 2001
    • 46917

    #2
    Juicy is one SEXY mother fucker!

    Comment

    • Juicy D. Links
      So Fucking Banned
      • Apr 2001
      • 122992

      #3
      Originally posted by AaronM
      Juicy is one SEXY mother fucker!
      damn right

      Comment

      • bigmack
        Confirmed User
        • Feb 2005
        • 329

        #4
        No negative feedback on this end. Just by seeing the postcounts and watching topics get buried 2 pages back within 5 minutes, I can easily tell that this site takes some work to keep up. Had no idea about those details but its good to see a website with this much traffic to have a professional team and webmaster to keep the gears turning.
        Party Flings Blog
        Smokin Hot Hunnies

        I host with Lessel Web
        Hosting

        TrafficHolder.com - Buy and Sell Traffic

        Comment

        • psili
          Confirmed User
          • Apr 2003
          • 5526

          #5
          It's sad, that for some, it's easier to spew criticism then anything positive.

          Just wanted to say that and give a worthy post a bump.

          Cheers.
          Your post count means nothing.

          Comment

          • Juicy D. Links
            So Fucking Banned
            • Apr 2001
            • 122992

            #6
            As long as the board is up its all good.

            So we wait for the search.......chilllllllllllllllllllllllllllllllll

            Be happy we have place to post holla back at me

            Comment

            • RRRED
              Confirmed User
              • Jan 2001
              • 6754

              #7
              Big Ray nice post That's exactly the kind of answer everyone needed... Now everyone STOP ICQING THE SHIT OUT OF ME ABOUT IT OR I'LL KILL YOUR PETS!!

              whew... ok carry on...

              Comment

              • Juicy D. Links
                So Fucking Banned
                • Apr 2001
                • 122992

                #8
                Originally posted by RRRED
                Juicy , I want to make sweet love to you , I cant stop dreaming about you
                Thanks RRRED i am blushing now

                Comment

                • bringer
                  i have man boobies
                  • Jul 2003
                  • 13082

                  #9
                  block all the spiders and setup a search server outside of the normal one
                  shutdown for 15minutes at 3am when nothing but 5 boardwhores posting ******* pictures are online and let it update.
                  333-765-551

                  Comment

                  • sixxxthsense
                    Confirmed User
                    • Aug 2004
                    • 2419

                    #10
                    biggups to GFY!

                    Comment

                    • Juicy D. Links
                      So Fucking Banned
                      • Apr 2001
                      • 122992

                      #11
                      Give me info on this "script kiddie"

                      I know people who know people.....holla back at me.

                      Comment

                      • loverboy
                        When it rains, it pours
                        • May 2003
                        • 20609

                        #12
                        For those haters, you cannot put a big board like GFY down.

                        GFY rocks!

                        Comment

                        • Juicy D. Links
                          So Fucking Banned
                          • Apr 2001
                          • 122992

                          #13
                          I plan on having my kids and grandchildren posting here!!!!!!!!

                          Comment

                          • bigmack
                            Confirmed User
                            • Feb 2005
                            • 329

                            #14
                            I am just itching to know this. How much space in GB or TB does the databases for this board take up?
                            Party Flings Blog
                            Smokin Hot Hunnies

                            I host with Lessel Web
                            Hosting

                            TrafficHolder.com - Buy and Sell Traffic

                            Comment

                            • TexasDreams
                              former Miserable Admin :)
                              • Oct 2003
                              • 4700

                              #15
                              Originally posted by Big Ray
                              Some folks have been supportive, but others have kind of shit on Lensman and Jupiter.
                              I appreciate the efforts, and can understand the nightmares. THANK YOU!!!



                              That banana and a nice link to Jupiter Hosting can be found at...

                              http://www.texasdreams.com/dance5.html

                              ICQ: 168-914-369 >>> sysop [at] TexasDreams [dot] com

                              Comment

                              • fris
                                I have to go potty
                                • Aug 2002
                                • 55708

                                #16
                                Originally posted by Big Ray
                                So, for the last week I have sat here and read several posts about search being down and other GFY related problems.

                                Some folks have been supportive, but others have kind of shit on Lensman and Jupiter.

                                First off, for those who have lent support, it is appreciated.

                                For those who have shit on why search is down, and why the site gets slow, I say before you open your mouth and profess to understand the workings of this board, read below.

                                GFY is no doubt a popular board. Several folks think they have something they can add by building tools like outside search engines and spiders to try and make a quick buck or earn a good rap.

                                To those who do that, do you realize that that this is a main cause of issues on the board?

                                For example, yesterday, we had over 15 spiders crawling the site. Clicking every thread for the last year and indexing every new post. Aggressive spiders and screen scrapers alone contribute to an estimated 15x load on the site as a whole over regular users.

                                Now lets take the search. Search is directly mapped to spiders who provide search services for keywords. Are you getting a clue why GFY Search maybe down at this time?

                                Let's add in the weekly DDoS attacks from some kiddie who was pissed he got called a "kiddie", it takes a full time well paid engineer to just keep the site up and running.

                                Ease up. Lensman has thrown some serious cash at this to both stop the problems from happening, and to shore up the infrastructure. It's a daily effort and I were you I would thank him.

                                15x load, add more ram in the sql server. how many system admins do you have on staff. other than the ones that have quit?

                                what do you do at jupiter incase of a DDOS attack. anybody knows you nullroute the ip. but you ofcourse know that right?

                                i have worked in many datacenters before. with much bigger servers bigger applications. this is smalltime compared to what you are dealing with. when a fortune 500 company has 1 min of downtime they are up your ass. just imagine if you were hired to setup a forum for a companies internal lan. if you cant successfully do it on the outside. why would someone hire you to do it on the inside.
                                Since 1999: 69 Adult Industry awards for Best Hosting Company and professional excellence.

                                My Latest Theme

                                Comment

                                • After Shock Media
                                  It's coming look busy
                                  • Mar 2001
                                  • 35299

                                  #17
                                  Originally posted by RRRED
                                  Big Ray nice post That's exactly the kind of answer everyone needed... Now everyone STOP ICQING THE SHIT OUT OF ME ABOUT IT OR I'LL KILL YOUR PETS!!

                                  whew... ok carry on...
                                  I am telling loryn.

                                  [email protected] ICQ:135982156 AIM: Aftershockmed1a MSN: [email protected]

                                  Comment

                                  • tranza
                                    ICQ: 197-556-237
                                    • Jun 2003
                                    • 57559

                                    #18
                                    Any idea when the search will be up again?
                                    I'm just a newbie.

                                    Comment

                                    • Techie Media
                                      Confirmed User
                                      • Jan 2001
                                      • 3092

                                      #19
                                      Originally posted by AaronM
                                      Juicy is one SEXY mother fucker!
                                      Right On


                                      Ray nice post bro


                                      sales [AT] techiemedia.com

                                      Comment

                                      • Judas_Hansen
                                        Confirmed User
                                        • Oct 2003
                                        • 520

                                        #20
                                        have you hear about robot.txt? just tell the spiders what to search, and what not to search.. The perfect solution would be to build a "SE archive" from the daily backup located on an other server, and then let the Spiders look at that one, and not the live site

                                        | EUROPEAN HOSTING = FORGET 2257 | IcQ 300-894038 | anti-SPYWARE |

                                        Comment

                                        • bawdy
                                          Confirmed User
                                          • Feb 2002
                                          • 1424

                                          #21
                                          Originally posted by Judas_Hansen
                                          have you hear about robot.txt? just tell the spiders what to search, and what not to search.. The perfect solution would be to build a "SE archive" from the daily backup located on an other server, and then let the Spiders look at that one, and not the live site
                                          do you think all spiders/site scrapper abide by the robots.txt file?

                                          Comment

                                          • Big Ray
                                            Confirmed User
                                            • Dec 2003
                                            • 464

                                            #22
                                            Originally posted by fris
                                            15x load, add more ram in the sql server. how many system admins do you have on staff. other than the ones that have quit?

                                            what do you do at jupiter incase of a DDOS attack. anybody knows you nullroute the ip. but you ofcourse know that right?

                                            i have worked in many datacenters before. with much bigger servers bigger applications. this is smalltime compared to what you are dealing with. when a fortune 500 company has 1 min of downtime they are up your ass. just imagine if you were hired to setup a forum for a companies internal lan. if you cant successfully do it on the outside. why would someone hire you to do it on the inside.
                                            LOL. Fris. You know nothing of us, myself included, or our employees (past or present). You no nothing of what we run here or how we run it.

                                            Why don't you go back to scripting kiddie.

                                            Five of the Top 10 Largest Programs Host with Jupiter. Find out why at http://www.jupiterhosting.com

                                            Comment

                                            • Violetta
                                              Affiliate
                                              • Jul 2004
                                              • 28735

                                              #23
                                              thanks for the info!
                                              M&A Queen

                                              Comment

                                              • Wiggles
                                                Too lazy to set a custom title
                                                • May 2004
                                                • 14423

                                                #24
                                                if people are worried about tracking threads then they should just bookmark important threads and check them, not an easy issue for Lens and the rest to keep this going and still maintain the quality speeds of the board.
                                                no sig

                                                Comment

                                                • WiredGuy
                                                  Pounding Googlebot
                                                  • Aug 2002
                                                  • 34512

                                                  #25
                                                  Originally posted by Big Ray
                                                  LOL. Fris. You know nothing of us, myself included, or our employees (past or present). You no nothing of what we run here or how we run it.

                                                  Why don't you go back to scripting kiddie.
                                                  15x load coming up, lol.
                                                  WG
                                                  I play with Google.

                                                  Comment

                                                  • WiredGuy
                                                    Pounding Googlebot
                                                    • Aug 2002
                                                    • 34512

                                                    #26
                                                    On a serious note, I do like GFY and I have seen it proposed here a few times, what about installing some sort of OCR on the search page? From the description I read that you posted, the search is the most vulnerable area an attacker would do to bring GFY to a crawl, so putting an OCR would make the most sense to stop all scripts? I'm not sure how easy it would be to integrate into GFY but I imagine it would relieve efforts significantly.

                                                    WG
                                                    I play with Google.

                                                    Comment

                                                    • Serge Litehead
                                                      Confirmed User
                                                      • Dec 2002
                                                      • 5190

                                                      #27
                                                      Big Ray, thank you for the post!

                                                      I'm concerned. Having search off is a definite drawback - massive amounts of resources that can not be accessed.
                                                      How do spiders affect the search? Normally siders go through the links, the existing pages. Spiders shouldn't and cannot perform direct searches unless spider is targeted to use the search. I don't know what the case is. But like WG said, image verification is a simple solution - has it been tried before?

                                                      If spiders cause heavy load on search, prevent spiders from using it.
                                                      If search by itself takes a big load then something can be done with the database and sql queries.

                                                      Comment

                                                      • Jace
                                                        FBOP Class Of 2013
                                                        • Jan 2004
                                                        • 35562

                                                        #28
                                                        am I reading right, that sites like www.s e a r c h t h e b o a r d s.com don't help it? why not ban it's ip and not allow it to search on here?

                                                        Comment

                                                        • notjoe
                                                          Confirmed User
                                                          • May 2002
                                                          • 5599

                                                          #29
                                                          Originally posted by Big Ray
                                                          Now lets take the search. Search is directly mapped to spiders who provide search services for keywords. Are you getting a clue why GFY Search maybe down at this time?
                                                          Who are you talking about... AFAIK there is me, BT, and boardscanner and unless im missing anything they're not linked to your search function.

                                                          Comment

                                                          • fuzzypeach
                                                            Confirmed User
                                                            • May 2005
                                                            • 392

                                                            #30
                                                            Originally posted by JaceXXX
                                                            am I reading right, that sites like www.s e a r c h t h e b o a r d s.com don't help it? why not ban it's ip and not allow it to search on here?
                                                            JaceXXX, if GFY were to ban your IP, would you:
                                                            a) Get another IP?
                                                            b) Stay banned?
                                                            Sig too big!

                                                            Comment

                                                            • Brujah
                                                              Beer Money Baron
                                                              • Jan 2001
                                                              • 22157

                                                              #31
                                                              I just can't help but think that National Net's LBBV could easily fix this situation without always needing to throw piles of cash at it.

                                                              Comment

                                                              • Brujah
                                                                Beer Money Baron
                                                                • Jan 2001
                                                                • 22157

                                                                #32
                                                                Search could EASILY be approved for a certain membership level, say 5k+ posts for example. While being disallowed for all others.

                                                                A spider trap could be implemented to automatically block spiders that don't obey robots.txt

                                                                Comment

                                                                • fuzzypeach
                                                                  Confirmed User
                                                                  • May 2005
                                                                  • 392

                                                                  #33
                                                                  Originally posted by notjoe
                                                                  Who are you talking about... AFAIK there is me, BT, and boardscanner and unless im missing anything they're not linked to your search function.
                                                                  Like Big Ray has said many times to many people, you just don't know the whole picture...

                                                                  And since you're a part of the problem, you're not part of the solution...

                                                                  Says the guy with only 1 bot to the guy with 2 bots, "you're the problem."
                                                                  Says the guy with only 2 bots to the guy with 3 bots, "you're the problem."
                                                                  Says the guy with only 3 bots to the guy with 4 bots, "you're the problem."

                                                                  ^^^ notjoe, which guy are you?
                                                                  Sig too big!

                                                                  Comment

                                                                  • notjoe
                                                                    Confirmed User
                                                                    • May 2002
                                                                    • 5599

                                                                    #34
                                                                    Originally posted by JaceXXX
                                                                    am I reading right, that sites like www.s e a r c h t h e b o a r d s.com don't help it? why not ban it's ip and not allow it to search on here?
                                                                    That is a misconception..the search on STB is completely independant of the GFY search issues and AFAIK had absolutely nothing to do with the problems happening recently..I dont consider 8.3k page loads (from memory) much..

                                                                    If there was infact 15 spidering running on GFY all you would need to do is subtract 15 from the total views of each thread to get the amount of views not including the bots.

                                                                    The search on STB spans about 7 different database servers which is why we're able to search 14+ million posts within a fairly reasonable amount of time. Thats also the reason we're spidering mainstream boards such as dbforums.com, maxbimmer.com, bodybuilding.com and a few others...

                                                                    To date, our of the 40+ boards i've been spidering for the last few months i havent crashed any of them or had any admins accusing me of DOS'ing them, except for GFY.

                                                                    Comment

                                                                    • notjoe
                                                                      Confirmed User
                                                                      • May 2002
                                                                      • 5599

                                                                      #35
                                                                      Originally posted by fuzzypeach
                                                                      Like Big Ray has said many times to many people, you just don't know the whole picture...

                                                                      And since you're a part of the problem, you're not part of the solution...

                                                                      Says the guy with only 1 bot to the guy with 2 bots, "you're the problem."
                                                                      Says the guy with only 2 bots to the guy with 3 bots, "you're the problem."
                                                                      Says the guy with only 3 bots to the guy with 4 bots, "you're the problem."

                                                                      ^^^ notjoe, which guy are you?
                                                                      Why dont you tell me. I can tell you that im not the guy who does "Now lets take the search. Search is directly mapped to spiders who provide search services for keywords. Are you getting a clue why GFY Search maybe down at this time? " as Ray put it

                                                                      Comment

                                                                      • AlienQ - BANNED FOR LIFE
                                                                        best designer on GFY
                                                                        • Mar 2003
                                                                        • 30307

                                                                        #36
                                                                        Originally posted by Big Ray
                                                                        LOL. Fris. You know nothing of us, myself included, or our employees (past or present). You no nothing of what we run here or how we run it.

                                                                        Why don't you go back to scripting kiddie.

                                                                        Comment

                                                                        • rounders
                                                                          Confirmed User
                                                                          • Sep 2003
                                                                          • 2931

                                                                          #37
                                                                          good job
                                                                          ICQ#: 153923840

                                                                          Comment

                                                                          Working...