|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
12-28-2003, 10:17 PM
|
#1 |
|
Confirmed User
Join Date: Jul 2003
Posts: 278
|
are there any spiders/crawlers out there like "zeus"?
except a tad more spiffy and a lot more configurable as far as blocking rules... done in perl so it can run on a server 24/7
|
|
|
|
12-28-2003, 10:20 PM
|
#2 |
|
So Fucking Banned
Join Date: Sep 2001
Location: shell beach
Posts: 7,938
|
what about wget ??
|
|
|
|
|
12-28-2003, 10:23 PM
|
#3 | |
|
Confirmed User
Join Date: Jul 2003
Posts: 278
|
Quote:
Interesting... not looking to save the webpages via HTTP. Merely want it to generate a list of URLs with descriptions and load it into a page kind of like zeus does (but not for the purposes of building a directory or farm, just to get the URLs). |
|
|
|
|
