Originally Posted by Jesus H Christ

True. I clued in to something interesting about 19 months ago and chased it down and was surprised on what I found. I'll give hints. Normally it's a good thing when SE's bot crawls your sites. I was pissed when Cuil kept crawling my smaller servers/sites and the bots were relentless hitting hourly even 404's and a few times shutting down the server. I was like what the fuck do they have to gain from this?. Then spin3r started across all my sites and then I started to get Corporate bots from well known companies in my niche. Then it hit me, my niche/sites are not alone in this and (the hint) why are they data-mining?

I got a smile on my face and never looked back from there..