Quote:
Originally Posted by Marshal
Adding thousands of rules in robots.txt won't slow down a website, since robots.txt is only fetched once, or once in a while.
By the way, that script is written for Apache. Basically a blacklist of rules and IPs.
|
I just quickly looked in the file and saw robots.txt. I plan on using this thanks for info.