Quote:
Originally Posted by money biz
1500+ records in robots.txt does that make site slower?
|
Adding thousands of rules in robots.txt won't slow down a website, since robots.txt is only fetched once, or once in a while.
By the way, that script is written for Apache. Basically a blacklist of rules and IPs.