Quote:
Originally Posted by AdultKing
PHP is too slow.
Most spiders are built out of C, Java or Python.
Spiders apart from very few functions should be decoupled from the index you are building.
|
thanks., looking in to./
i managed to crawl 100k pages in 15 minutes while using 250 threads.
for now, this is enough for me.,