Quote:
Originally posted by Armed & Hammered
That's the problem. Most of those programs bullshit the server. Most look like a legit browser.
|
No shit! Unless they're someone out there with a browser that's actually named "Mozilla by Wal-Mart Made in China" who can do a GET on 100 files a second, I do believe some will let you tell it you want it to look like anything you want it to.
Since in your robots.txt file you have to describe the default for EACH and EVERY damn spider they ever came up with, you're only going to stop people who are too dumb to look at the options in their spider app.
Anyone who knows enough to spider you is probably going to be able to change the name to something you don't already have in there within 2 minutes.
I gave up on the spiders. They're not the ones who REALLY jurt. The ones that piss me off are the ones that hot link to files. One fucker in Holland linked to one of my files and ate up about 5 gigs in one day.
I think I came up with a way to turn that around though.
Oh well...
Refresh
__________________
"If it floats, flies or fucks... rent it"
<a href="http://www.goodpussygonebad.com" target="_blank">
<img border="0" src="http://www.goodpussygonebad.com/images/site/GPGBLogo_sm.jpg" width="73" height="74">