Quote:
|
Originally Posted by digifan
The funny thing is the google pr... wondering why so low. Google must be blocked in the robots.txt me thinks.
|
/robots.txt
User-agent: msnbot
Crawl-delay: 900
Disallow: /cgi-bin
User-agent: slurp
Crawl-delay: 900
User-agent: Slurp
Disallow: /
User-agent: hahahahahahahahahahahaha
Disallow: /