Anyone out there running hosted gallery' figure out a method to keep download spiders off there gallery's?
I am just getting reamed by download spiders they are just sucking down my shit like there was nothing. there keeping 30mbs pegged now for days. I am pushing over 300k uniques through the gallery's daily, but I am seeing alot of spider traffic that isn't SE related, user agent banning only works to a degree. I also tried the standard spider trap hidden link, but this apperently pissed off a few of the TGP spiders as having bad or broken links or popups/redirects. which I don't unless you went to a hidden link (yes trully hidden unless you read the code or you were a spider just looking for href tags) some help me with a solution here?
I am just getting reamed by download spiders they are just sucking down my shit like there was nothing. there keeping 30mbs pegged now for days. I am pushing over 300k uniques through the gallery's daily, but I am seeing alot of spider traffic that isn't SE related, user agent banning only works to a degree. I also tried the standard spider trap hidden link, but this apperently pissed off a few of the TGP spiders as having bad or broken links or popups/redirects. which I don't unless you went to a hidden link (yes trully hidden unless you read the code or you were a spider just looking for href tags) some help me with a solution here?
Comment