The reason why Search is down is because of the Board Leech sites.

[notjoe]

Claiming I am the cause of the board's misfortune is complete and utter nonsense. Let explain why.

Yesterday GFY had a total of 8356 posts which I've counted/spidered. That?s 8.3k page views. Might sound like a lot right? But its noting in comparison to how many page views GFY gets on a daily basis.

I don?t now how other spiders work but I can say mine is probably the most gentle of them all. I don?t download entire threads I download single posts. What does that mean? Quite simply put it means that we download an avg of maybe 5000-15000 bytes per post, depending on the length of the content and not the bloated 100-200K threads which you would come across as a surfer.

Obviously the search I did isn?t a piggyback system. I don?t search using GFY's search function and alter results to look like my own thus don?t add load to in that aspect.

The threads I download have been posted within the last few minutes. This means that I am NOT causing disk seeks and if I am causing a seek I'm willing to be its only one seek, maybe two...i'm betting the majority of the information is cached within MySQL's query cache/table indexes.


You say "Board republishers and search engines are two different things." How is this any different? I spider urls, I download the content and I create a searchable index out of it. People search and results are generated. Those results are then ranked and displayed, that sounds like an SE to me.

You say SEs are ok but STB is bad, unlike the SEs we don?t download the same content over and over again. It downloads it one time and moves on never to be back to that post ever again. I highly doubt that this would/could cause the kind of

You claim that these "leechers" are the root of the problem but why do you think most people run a bot to begin with? It isn?t to add login time to their account its probably to monitor posts for keyword. so instead of having one or two people running a service and scanning threads you're having 200+ people running their own bot to monitor threads.. Now instead of 17k page views from me and boardt-racker (8.3k*2) you're getting 200x8.3k= 1660000 hits per day from usless bot clients. Doesn?t compute.

However, all that being said, it is still your board and I will respect your wishes.

Best regards,

[azguy]

Originally Posted by Big Ray

Hey WG.

You make a good point, however, their is a big difference in the google and yahoo spider than the ones that try and pull 200 threads a second all day every day. What we have to contend with are aggressive screen scrapers that feel the need to index every post the second it's made. They are highly aggressive and are not throttled. They behavior more like a DoS attack than a bot.

People ask why we don't ban them. The answer is we try, however, the majority of them use anonymous proxies and I don't mean a few.. I am talking hundreds. When we block those, they get a new batch. It's a constant battle which we fight every day.

The search feature is widely used to attempt to create search notification systems. Some companies do this, and hit the site once a day to see what people are saying about them or "keyword". Others are constantly hitting search for keywords like "need content" or "need hosting". Ever wonder how the same folks always post first to a request for product or service thread? now ya know. lol

Ray

You're right, a SE must take into consideration the target server's load, etc. and use common sense and "SE spidering ethics". In this case, you should seek to stop this from happening or work with them to improve their efforts. That would save both sides a lot of time. After all, they are trying to send traffic to GFY (in the case of STBoards) and have no problem having people read the full thread here.

[Big Ray]

Originally Posted by WiredGuy

Wow, i had no idea they were pulling in data so frequently. I thought these feeds were just fetching your data daily or so. If they really are using hundreds of changeable proxies like this, I'd get ICS' attorneys involved at that point, a C&D should probably suffice. They obviously are not respecting your servers limits and circumventing your techniques to prevent their abuse. I really had no idea they were pulling data that frequently, that seems like an overkill where once per day suffices just as well.

WG

It's been done (C&D and DMCA notices). Some respect them, some don't. It's just part of the process.

Side note though. Many, not all, but many of these are overseas companies. You can send C&D's to them, or a roll of toilet paper. All the same to them.

[goBigtime]

Originally Posted by Babaganoosh

Lensman, what the search really needs is an image verification. That would stop this shit immediately. It wouldn't be tough for one of your geeks to whip up.

I was going to mention it again, but I figured it's been suggested many times now during the many months worth of search outages.

It's a pretty obvious solution to the problem mentioned here.

I don't understand why it wasn't implemented within like 72 hours tops from the onset of the original problem considering the size of the board and the amount of money it makes.

I have a feeling they'll listen this time, especially after this post.


IMO, the BEST thing GFY has going for it is its massive post archive. It's quite understandable why Lens doesn't want people leeching it and replicating it for their own gain.

What's not understandable is how the solution to protect that archive and restore functionality to the board was to remove the archive for the amount of time is has been removed.

[Satisfaction]

good luck with it all

[AlienQ - BANNED FOR LIFE]

NotJoe...
I want royalties for you taking my 150 or so posts yesterday.
I do not want you copying my work and you have no right to do so...

[notjoe]

Originally Posted by AlienQ

NotJoe...
I want royalties for you taking my 150 or so posts yesterday.
I do not want you copying my work and you have no right to do so...

You should be paying me for wasted resources in storing your posts

[Big Ray]

Originally Posted by notjoe

Claiming I am the cause of the board's misfortune is complete and utter nonsense. Let explain why.

Yesterday GFY had a total of 8356 posts which I've counted/spidered. That?s 8.3k page views. Might sound like a lot right? But its noting in comparison to how many page views GFY gets on a daily basis.

I don?t now how other spiders work but I can say mine is probably the most gentle of them all. I don?t download entire threads I download single posts. What does that mean? Quite simply put it means that we download an avg of maybe 5000-15000 bytes per post, depending on the length of the content and not the bloated 100-200K threads which you would come across as a surfer.

Obviously the search I did isn?t a piggyback system. I don?t search using GFY's search function and alter results to look like my own thus don?t add load to in that aspect.

The threads I download have been posted within the last few minutes. This means that I am NOT causing disk seeks and if I am causing a seek I'm willing to be its only one seek, maybe two...i'm betting the majority of the information is cached within MySQL's query cache/table indexes.


You say "Board republishers and search engines are two different things." How is this any different? I spider urls, I download the content and I create a searchable index out of it. People search and results are generated. Those results are then ranked and displayed, that sounds like an SE to me.

You say SEs are ok but STB is bad, unlike the SEs we don?t download the same content over and over again. It downloads it one time and moves on never to be back to that post ever again. I highly doubt that this would/could cause the kind of

You claim that these "leechers" are the root of the problem but why do you think most people run a bot to begin with? It isn?t to add login time to their account its probably to monitor posts for keyword. so instead of having one or two people running a service and scanning threads you're having 200+ people running their own bot to monitor threads.. Now instead of 17k page views from me and boardt-racker (8.3k*2) you're getting 200x8.3k= 1660000 hits per day from usless bot clients. Doesn?t compute.

However, all that being said, it is still your board and I will respect your wishes.

Best regards,

Like others, you make many assumptions on how this board works knowing what?s under the hood. It?s analogous to assuming an engine takes unleaded to find out that, opps, in fact it takes diesel.

No one said that your site is THE ONE that is causing the problems. What I am saying is that it contributes to the problem.

Seems everyone that spiders the site assumes they are the only one doing it. That computer resources are limitless and that what they are doing is ?small enough? not to cause a problem. Easy enough to crank up that spider a few more notches. Let?s see how far it can go!

The fact is that computer resources are not infinite and you are not the only one doing it. How many times an hour does your spider hit the site? A Minute? (we have logs).

When does a spider stop being a spider and become the source in a DoS attack. What?s the difference?

[notjoe]

Originally Posted by azguy

You're right, a SE must take into consideration the target server's load, etc. and use common sense and "SE spidering ethics". In this case, you should seek to stop this from happening or work with them to improve their efforts. That would save both sides a lot of time. After all, they are trying to send traffic to GFY (in the case of STBoards) and have no problem having people read the full thread here.

You hit the nail on the head..

All i add is 8.3k page views per day

[Big Ray]

Originally Posted by azguy

You're right, a SE must take into consideration the target server's load, etc. and use common sense and "SE spidering ethics". In this case, you should seek to stop this from happening or work with them to improve their efforts. That would save both sides a lot of time. After all, they are trying to send traffic to GFY (in the case of STBoards) and have no problem having people read the full thread here.

Not my rules. I run the hosting. But I will say this.

It's Lensman's board. He has been running it for years his way. He has every right to run it the way HE wants to. If he doesn't want it, then isn't it his call?

[Theo]

Originally Posted by AlienQ

NotJoe...
I want royalties for you taking my 150 or so posts yesterday.
I do not want you copying my work and you have no right to do so...

royalties for the stupidity? you have to patent this. Talk to your buddy Evan lol

[Big Ray]

Originally Posted by goBigtime

What's not understandable is how the solution to protect that archive and restore functionality to the board was to remove the archive for the amount of time is has been removed.

Sometimes you need to take a step backwards to make a two steps forward.

[AlienQ - BANNED FOR LIFE]

Originally Posted by notjoe

You should be paying me for wasted resources in storing your posts

Your are a Copyright thief and taking my intellectual property.
You have no right to redistribute it.

[chupachups]

Originally Posted by Soul_Rebel

royalties for the stupidity? you have to patent this. Talk to your buddy Evan lol

[goBigtime]

Originally Posted by Big Ray

Sometimes you need to take a step backwards to make a two steps forward.

I can understand why you wouldn't want the solution to be something so easy.

I am *very* familiar with how VB works and how networking at this level and beyond works.

If the problem is what Lens mentions in the first post, then a well coded image verification routine will solve it. Period.

Technically, yes. Assuming that usability had nothing to do with it.

[notjoe]

Originally Posted by AlienQ

Your are a Copyright thief and taking my intellectual property.
You have no right to redistribute it.

Riiiight...Why dont you go sue google you moron

[4Pics]

Why not allow search for certain users?

Or just add a image function to it, that the spiders can't beat?

[azguy]

Originally Posted by Big Ray

Not my rules. I run the hosting. But I will say this.

It's Lensman's board. He has been running it for years his way. He has every right to run it the way HE wants to. If he doesn't want it, then isn't it his call?

Like I said earlier, I understand that. It is his call to some extent. I'm just raising the issue that on GFY the posts belong to the authors and therefore DMCA notices coming from Lensman are unlikely to hold in court. They might scare some hosts or even the "leecher", but that's as far as it will go if it ever gets to that point. Especially if the "leecher" is merely a SE (publishing a very short excerpt along with a link directly to the GFY thread), which is exactly what STboards seems to be. As Joe said, his bot's activity equals to about 10 GFY regulars, each viewing 10% of the the daily posts. Clearly not enough to crash GFY.

[AlienQ - BANNED FOR LIFE]

[CENTER]

Originally Posted by notjoe

Riiiight...Why dont you go sue google you moron

Technically I could

So could George Lucas...

[azguy]

Originally Posted by AlienQ

Your are a Copyright thief and taking my intellectual property.
You have no right to redistribute it.

This is again not true. By posting you knew that the post will appear on GFY and that there are search engines out there, such as Google and archive.org, that cache pages and that your post is likely to be cached. I told you, it's not black and white. Message board posts are the hardest to enforce under the DMCA.

[goBigtime]

It shouldn't have taken more than 72 hours to code in a custom routine including image verification that would knock the fuck out of anyone doing anything they aren't supposed to.

And no more than 4 hours if you just used the stock VB image verification routine.

How long has GFY search been down again total?

This really isn't Lens's fault here at all... I'm guessing he plays the role more of an organizer and project leader here. But I assume he has some tech guys hired that should have came up with these solutions LOOOOOOOOONG ago.

If I were Lens I would be one pissed off CEO - and not at NotJoe or other people spidering.

[Big Ray]

Originally Posted by azguy

Like I said earlier, I understand that. It is his call to some extent. I'm just raising the issue that on GFY the posts belong to the authors and therefore DMCA notices coming from Lensman are unlikely to hold in court. They might scare some hosts or even the "leecher", but that's as far as it will go if it ever gets to that point. Especially if the "leecher" is merely a SE (publishing a very short excerpt along with a link directly to the GFY thread), which is exactly what STboards seems to be. As Joe said, his bot's activity equals to about 10 GFY regulars, each viewing 10% of the the daily posts. Clearly not enough to crash GFY.

Please read my previous post.

[AlienQ - BANNED FOR LIFE]

Azguy it does not matter.

SE's have the option of indexing or not indexing a site, just like a site owner has the right to deny whom or what access thier site.

Its a two street.

Personally I have always found Search Engines and the like to be invasive of intellectual properties and overly opportunistic.

Such as even illustrated above.

That link leads to Bit Torrent.

[azguy]

Originally Posted by goBigtime

It shouldn't have taken more than 72 hours to code in a custom routine including image verification that would knock the fuck out of anyone doing anything they aren't supposed to.

And no more than 4 hours if you just used the stock VB image verification routine.

How long has GFY search been down again total?

This really isn't Lens's fault here at all... I'm guessing he plays the role more of an organizer here. But I assume he has some tech guys hired that should have came up with these solutions LOOOOOOOOONG ago.

If I were Lens I would be one pissed off CEO - and not at NotJoe or other people spidering.

What are you talking about? An image verification test has nothing to do with this. Those so-called leechers do not use the search feature on-the-fly or anything like that. You think they should use it on the login screen? The the bot owner will do that manually and have the bot do the rest. It doesn't solve anything, as supposedly the problem is the bots downloading so many pages.

[Juicy D. Links]

Originally Posted by Lensman

Funny to think about, eh?

I know the game , you know the game all the matters is the $$$$ in each of our pockets in the end

[Juicy D. Links]

Serious Note, give me info the the kiddie trying to ddos or whatever . i know a freind of a freind of a friend :-)

[azguy]

Originally Posted by Big Ray

Like others, you make many assumptions on how this board works knowing what?s under the hood. It?s analogous to assuming an engine takes unleaded to find out that, opps, in fact it takes diesel.

No one said that your site is THE ONE that is causing the problems. What I am saying is that it contributes to the problem.

Seems everyone that spiders the site assumes they are the only one doing it. That computer resources are limitless and that what they are doing is ?small enough? not to cause a problem. Easy enough to crank up that spider a few more notches. Let?s see how far it can go!

The fact is that computer resources are not infinite and you are not the only one doing it. How many times an hour does your spider hit the site? A Minute? (we have logs).

When does a spider stop being a spider and become the source in a DoS attack. What?s the difference?

I understand the need to decrease the amount of traffic and server load used by spiders. What I was originally saying is that the 'content theft' excuse is not the right one. At the time of the actual crawling, the sites did not republish anything as far as you know. Since they use proxies, you have no idea who they are doing all that work for.

[Big Ray]

Originally Posted by goBigtime

It shouldn't have taken more than 72 hours to code in a custom routine including image verification that would knock the fuck out of anyone doing anything they aren't supposed to.

And no more than 4 hours if you just used the stock VB image verification routine.

How long has GFY search been down again total?

This really isn't Lens's fault here at all... I'm guessing he plays the role more of an organizer and project leader here. But I assume he has some tech guys hired that should have came up with these solutions LOOOOOOOOONG ago.

If I were Lens I would be one pissed off CEO - and not at NotJoe or other people spidering.

You make the assumption that some sort of block is wanted in the search area. You are assuming that someone is incapable of doing that.

Let me ask you this. If it were your site, would you want the functionality dictated to you by 1% of the people that use your site?

Why would Lensman put an image verification system in front of search that impacts 99% of the legitimate people that use search? I?m sure no one is going to to get upset about that.

The key to a successful GFY is to be accountable for the functionality and take care of the people that make site great. Users shouldn?t have to change the way they interact with the site. Apply your same logic to google.

[Big Ray]

Originally Posted by azguy

Those so-called leechers do not use the search feature on-the-fly or anything like that.

How do you know what others do and do not do with / to this board?

[smutx]

ahh now i know why the search is gone... DAMN YOU LEECHERS DAMN YOU ALL TO HELL!!

[notjoe]

Originally Posted by Big Ray

Like others, you make many assumptions on how this board works knowing what?s under the hood. It?s analogous to assuming an engine takes unleaded to find out that, opps, in fact it takes diesel.

With the way the search has been you might want to consider installing a supercharger?

Originally Posted by Big Ray

No one said that your site is THE ONE that is causing the problems. What I am saying is that it contributes to the problem.

Seems everyone that spiders the site assumes they are the only one doing it.
That computer resources are limitless and that what they are doing is ?small enough? not to cause a problem. Easy enough to crank up that spider a few more notches. Let?s see how far it can go!

I guess having hundreds of members trying to run their own keyword alert/monitoring service for themselves which relies on your search function is better than 1 or 2 sites who provide that service and don't touch the search function?

As for cranking it up a notch, the spider will only spider as many posts as the board has to offer, nothing more and nothing less...sounds like a more traditional SE bot that you're talking about[QUOTE=Big Ray]

Originally Posted by Big Ray

The fact is that computer resources are not infinite and you are not the only one doing it. How many times an hour does your spider hit the site? A Minute? (we have logs).

The bot continuously downloads posts until it's up to date at which point it goes back to sleep for 5 minutes. This is so that instead of hitting your site once a month and downloading hundreds of thousands of pages over a couple of days it's balanced out over the month.

Originally Posted by Big Ray

When does a spider stop being a spider and become the source in a DoS attack. What?s the difference?

is 8300 page views considered a DOS attack?

[Juicy D. Links]

On a side note:

I am sexy

[Big Ray]

Originally Posted by azguy

I understand the need to decrease the amount of traffic and server load used by spiders. What I was originally saying is that the 'content theft' excuse is not the right one. At the time of the actual crawling, the sites did not republish anything as far as you know. Since they use proxies, you have no idea who they are doing all that work for.

We know more than you think we do. We have been fighting this for years. Longer than you have been a member. Man, you make a lot of assumptions and it's humorous to see your posts applied to the "real" working knowledge of this site.

[azguy]

Originally Posted by Big Ray

How do you know what others do and do not do with / to this board?

I assumed so, but I was obviously wrong

[AlienQ - BANNED FOR LIFE]

But you still have no right to take peoples contributions.
Thats the bottom line you are side stepping.

Your stealing content and attemping to capitalise on it without the contributors consent, nor GFY;s consent.

What makes you think you have that right?

[Juicy D. Links]

btw i am looking for a postbot compatible with GFy and has artificial Intelliengce in it

[azguy]

Originally Posted by Big Ray

We know more than you think we do. We have been fighting this for years. Longer than you have been a member. Man, you make a lot of assumptions and it's humorous to see your posts applied to the "real" working knowledge of this site.

I'm talking specifically re STBoards.com and its position as a SE.

[Juicy D. Links]

Originally Posted by AlienQ

But you still have no right to take peoples contributions.
Thats the bottom line you are side stepping.

Your stealing content.

gfy or the spiders?

[azguy]

Originally Posted by AlienQ

But you still have no right to take peoples contributions.
Thats the bottom line you are side stepping.

Your stealing content.

In theory you may be right, but this is not true. How do you think Google gets away with caching every god damn page? Because it's never been proven to be illegal.

[Big Ray]

Originally Posted by azguy

I'm talking specifically re STBoards.com and its position as a SE.

So you are saying that the owner of the board should make exceptions for some sites, and ban others?

[Juicy D. Links]

BTW Lens make me a Mod please

[AlienQ - BANNED FOR LIFE]

Originally Posted by azguy

In theory you may be right, but this is not true. How do you think Google gets away with caching every god damn page? Because it's never been proven to be illegal.

I have the perogative to turn away the Google engine.
An option.

Buts since Lensman is the go between and houses my content here your merely stealing it without my consent nor his.

[Juicy D. Links]

Originally Posted by Juicy D. Links

BTW Lens make me a Mod please

4+ years here. Ill give you admin passes to my servers as insurance.

[Juicy D. Links]

Fact is if you spider or crawl GFY and use it without Tender Lens's consent then I feel its not Juicy

[goBigtime]

Originally Posted by Big Ray

You make the assumption that some sort of block is wanted in the search area. You are assuming that someone is incapable of doing that.

Let me ask you this. If it were your site, would you want the functionality dictated to you by 1% of the people that use your site?

Why would Lensman put an image verification system in front of search that impacts 99% of the legitimate people that use search? I?m sure no one is going to to get upset about that.

The key to a successful GFY is to be accountable for the functionality and take care of the people that make site great. Users shouldn?t have to change the way they interact with the site. Apply your same logic to google.

You just need to think about it a little more Ray. If you can't come up with anything, then brainstorm with people who you think can. If they can't, then find some new people to brainstorm with.

But there are plenty of answers to this problem.

If this were 12 months ago, I would have gladly explained some of them to you. I can't give away too many ideas away anymore though. I really wish I could, I enjoy GFY as much as the next guy.


I'll give you a clue though... even if you had image verification on the search 100% of the time, I don't think GFY members would give two fucks about it when faced with your alternative - no search at all.

Maybe you should do VB's image-verification based search as a temporary fix then? Since it's obviously a better solution than no search at all. No?


And this isn't google, so it's illogical to apply the same logic to it.

But maybe you should look to google for solutions for your problem.

[azguy]

Originally Posted by Big Ray

So you are saying that the owner of the board should make exceptions for some sites, and ban others?

I assume that makes no sense to you? Obviously that should be the case. If Lensman is okay with SEs that simply publishes 1 or 2 lines as an excerpt and link directly to the thread - than that SE should be permitted to crawl GFY. Of course, both sides would agree what times the bot would be active, from which IPs, etc.

On the other hand, if another site/bot is crawling GFY every 2 minutes and publishes entire threads and Lensman feels this is wrong - that bot should be banned.

It all boils down to what Lensman wants to do and what is technologically possible to do in order to accomplish that. From what it seems, he's not okay even with sites such as STboards.

The only reason I posted my first post is because the issue of copyrights and other legal matters fascinate me and I like to participate in such discussions. This thread is not going in the direction I originally thought it would - so I may have to call it a day soon

[notjoe]

Originally Posted by Big Ray

So you are saying that the owner of the board should make exceptions for some sites, and ban others?

Didnt lens already do that? Whether i'm a "leecher" or a legitimate SE i'm still prohibited and google is still allowed?

[Juicy D. Links]

100//////////

[Juicy D. Links]

oops 100

[Juicy D. Links]

shit 100