View Single Post
Old 09-12-2007, 05:49 PM  
Elli
Reach for those stars!
 
Industry Role:
Join Date: Apr 2003
Location: Vancouver, BC
Posts: 17,991
Quote:
Originally Posted by Tempest View Post
I hear ya... Google is partly to blame.. They could very easily flag those types of sites for aditional crawling of external javascript files etc. but they choose not to for some reason.

Actually, there is a potential option but it's not 100%. You set up a bot trap on your site. It needs to be a more advanced type because these guys are getting more brazen and are setting up their crawlers to pretend to be googlebot, slurp etc.
Yeah I don't want to stop valid bots from crawling, just the scrapers... sigh.
__________________
email: [email protected]
Elli is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote