Quote:
|
Originally Posted by FightThisPatent
If t3report spiders the adult web...
|
Does your spidering technology adhere to 'well behaved robot' protocols.
ie. Do you respect ROBOTS.TXT directives to ban your spider from sites not interested in being 'exposed' this way?
If so, what is your UserAgent ID to include in our ROBOTS.TXT to keep your spider out?
Or are we going to have to ban all your IP blocks?
-Dino