Quote:
Originally Posted by 10 Inch Dick
Spent 3 hour today looking at open source codes. ;)
You gave me some great ideas.
|
If you're interested in building a crawler of your own, check out the paper "Crawling the web" by Gautam Pant, Padmini Srinivasan, and Filippo Menczer. It's a great read that describes common web crawler architectures:
http://citeseerx.ist.psu.edu/viewdoc...10.1.1.14.1830