Quote:
Originally Posted by baddog
So you are suggesting that the spider takes the time to look at all the links, then sort in order of length?
|
exactly.
basically if it finds 200 new pages it records the urls , then schedules the order they will be spidered by length.