Quote:
|
Originally Posted by spunkmaster
Put this in your robots.txt file:
User-agent: WebZip
User-agent: larbin
User-agent: b2w/0.1
......
Disallow: /
|
The prob with real harvester software , they use real browser User Agent , so you cant find or control them.