Quote:
Originally Posted by 10 Inch Dick
I want crawler like that. Where do I get one?
|
You write it yourself ;)
There are a lot of good frameworks out there you could build off of (Scrapy and Nutch both come to mind), but it's not too hard to get a basic crawler up and running from scratch.