Quote:
|
Originally Posted by V_RocKs
Exactly... Google looks for a robots text file and only goes where you want it to... Board leeching spyders do not follow a robots file and usually do thier handy work by abusing the search function.
|
Not true. The new generation of spiders is going to ignore those.
www.dipsie.com is spidering the 'Deep Web' and are not following the robots.txt anymore. They log on to sites such as GFY with a login/pass and cache everything. That I know for a fact, I've met with them a while ago.