Quote:
|
Originally Posted by fireorange
Since you use a crawler bot, what do I put in my robots.txt to block your robots?
|
you can't block a web browser via a robots.txt file
we use specialized spiders based on web browsers to do the indexing, so it looks and behaves just like any surfer...
Fight the Do not Enter!