Claiming I am the cause of the board's misfortune is complete and utter nonsense. Let explain why.
Yesterday GFY had a total of 8356 posts which I've counted/spidered. That?s 8.3k page views. Might sound like a lot right? But its noting in comparison to how many page views GFY gets on a daily basis.
I don?t now how other spiders work but I can say mine is probably the most gentle of them all. I don?t download entire threads I download single posts. What does that mean? Quite simply put it means that we download an avg of maybe 5000-15000 bytes per post, depending on the length of the content and not the bloated 100-200K threads which you would come across as a surfer.
Obviously the search I did isn?t a piggyback system. I don?t search using GFY's search function and alter results to look like my own thus don?t add load to in that aspect.
The threads I download have been posted within the last few minutes. This means that I am NOT causing disk seeks and if I am causing a seek I'm willing to be its only one seek, maybe two...i'm betting the majority of the information is cached within MySQL's query cache/table indexes.
You say "Board republishers and search engines are two different things." How is this any different? I spider urls, I download the content and I create a searchable index out of it. People search and results are generated. Those results are then ranked and displayed, that sounds like an SE to me.
You say SEs are ok but STB is bad, unlike the SEs we don?t download the same content over and over again. It downloads it one time and moves on never to be back to that post ever again. I highly doubt that this would/could cause the kind of
You claim that these "leechers" are the root of the problem but why do you think most people run a bot to begin with? It isn?t to add login time to their account its probably to monitor posts for keyword. so instead of having one or two people running a service and scanning threads you're having 200+ people running their own bot to monitor threads.. Now instead of 17k page views from me and boardt-racker (8.3k*2) you're getting 200x8.3k= 1660000 hits per day from usless bot clients. Doesn?t compute.
However, all that being said, it is still your board and I will respect your wishes.
Best regards,
Yesterday GFY had a total of 8356 posts which I've counted/spidered. That?s 8.3k page views. Might sound like a lot right? But its noting in comparison to how many page views GFY gets on a daily basis.
I don?t now how other spiders work but I can say mine is probably the most gentle of them all. I don?t download entire threads I download single posts. What does that mean? Quite simply put it means that we download an avg of maybe 5000-15000 bytes per post, depending on the length of the content and not the bloated 100-200K threads which you would come across as a surfer.
Obviously the search I did isn?t a piggyback system. I don?t search using GFY's search function and alter results to look like my own thus don?t add load to in that aspect.
The threads I download have been posted within the last few minutes. This means that I am NOT causing disk seeks and if I am causing a seek I'm willing to be its only one seek, maybe two...i'm betting the majority of the information is cached within MySQL's query cache/table indexes.
You say "Board republishers and search engines are two different things." How is this any different? I spider urls, I download the content and I create a searchable index out of it. People search and results are generated. Those results are then ranked and displayed, that sounds like an SE to me.
You say SEs are ok but STB is bad, unlike the SEs we don?t download the same content over and over again. It downloads it one time and moves on never to be back to that post ever again. I highly doubt that this would/could cause the kind of
You claim that these "leechers" are the root of the problem but why do you think most people run a bot to begin with? It isn?t to add login time to their account its probably to monitor posts for keyword. so instead of having one or two people running a service and scanning threads you're having 200+ people running their own bot to monitor threads.. Now instead of 17k page views from me and boardt-racker (8.3k*2) you're getting 200x8.3k= 1660000 hits per day from usless bot clients. Doesn?t compute.
However, all that being said, it is still your board and I will respect your wishes.
Best regards,
Comment