are there any spiders/crawlers out there like "zeus"?
 

except a tad more spiffy and a lot more configurable as far as blocking rules... done in perl so it can run on a server 24/7

what about wget ??

Interesting... not looking to save the webpages via HTTP. Merely want it to generate a list of URLs with descriptions and load it into a page kind of like zeus does (but not for the purposes of building a directory or farm, just to get the URLs).