Ive had to setup spiders to level out bandwidth graphs, and with one machine can fill a gig pipe. Havent tried specifically for images, at that level but would be more then willing to talk with you. for $500 i want a harem of desi bitches and an appliance hehe
DESCRIPTION
GNU Wget is a free utility for non-interactive download of files from the Web. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies.
Wget is non-interactive, meaning that it can work in the background, while the user is not logged on. This allows you to start a retrieval and disconnect from the system, letting Wget finish the work. By contrast, most of the Web browsers require constant user's presence, which can be a great hindrance when transferring a lot of data.
Originally posted by hybrid What I want is not a file grabber, a picture stealer or a TGP cruiser. Shit, not even really a URL Spider, more of a description-finder.
as someone said before, wget without giving a proper description of what a description finder is I'll recommend grep for finding descriptions, maybe you might want to do something with perl instead...
My suggestion, run wget to suck down the files that you want to spider, then grep them to look for descriptions...
Comment