Sitemap maker? Lists all pages in excel or text - GoFuckYourself.com

mkx · 09-14-2010, 11:54 AM

I am looking to get a list of all url's in a website. There are about 100000 or so product ID's that are spread out in no specific order. Example www.website.com/productid=554887 can be a product and www.website.com/productid=554888 can be a 404 error but then www.website.com/productid=554899 can be a product again. Basically I want to rip the website by providing the urls to all the products but since the id's are random I need to build a list of links first. Hope I explained this right. Anyone know of a good software I can use to build this list?

woj · 09-14-2010, 12:14 PM

You should just pull it straight out of the db, rather than use some software to crawl your site... if you are interested in some custom solution, hit me up icq:33375924

mkx · 09-14-2010, 12:46 PM

i don't have access to the db, it's hosted on another server, one of those turnkey websites. will hit you up later though

harvey · 09-14-2010, 12:53 PM

Quote:

Originally Posted by mkx

I am looking to get a list of all url's in a website. There are about 100000 or so product ID's that are spread out in no specific order. Example www.website.com/productid=554887 can be a product and www.website.com/productid=554888 can be a 404 error but then www.website.com/productid=554899 can be a product again. Basically I want to rip the website by providing the urls to all the products but since the id's are random I need to build a list of links first. Hope I explained this right. Anyone know of a good software I can use to build this list?

try http://www.drk.com.ar/spider.php . It's free and pretty good, not sure if it will work with that amount of pages, but you can give it a try, it's 100% free

fris · 09-14-2010, 01:09 PM

download the sitemap, and parse the urls and grab them based on that.

09-14-2010, 11:54 AM	#1
mkx Confirmed User Industry Role: Join Date: Nov 2003 Location: Toronto Posts: 4,001	Sitemap maker? Lists all pages in excel or text I am looking to get a list of all url's in a website. There are about 100000 or so product ID's that are spread out in no specific order. Example www.website.com/productid=554887 can be a product and www.website.com/productid=554888 can be a 404 error but then www.website.com/productid=554899 can be a product again. Basically I want to rip the website by providing the urls to all the products but since the id's are random I need to build a list of links first. Hope I explained this right. Anyone know of a good software I can use to build this list?

09-14-2010, 12:14 PM	#2
woj <&(©¿©)&> Industry Role: Join Date: Jul 2002 Location: Chicago Posts: 47,882	You should just pull it straight out of the db, rather than use some software to crawl your site... if you are interested in some custom solution, hit me up icq:33375924 __________________ Custom Software Development, email: woj#at#wojfun#.#com to discuss details or skype: wojl2000 or gchat: wojfun or telegram: wojl2000 Affiliate program tools: Hosted Galleries Manager Banner Manager Video Manager Wordpress Affiliate Plugin Pic/Movie of the Day Fansign Generator Zip Manager

09-14-2010, 01:09 PM	#5
fris Too lazy to set a custom title Industry Role: Join Date: Aug 2002 Posts: 55,372	download the sitemap, and parse the urls and grab them based on that. __________________ Since 1999: 69 Adult Industry awards for Best Hosting Company and professional excellence. WP Stuff

09-14-2010, 12:46 PM	#3
mkx Confirmed User Industry Role: Join Date: Nov 2003 Location: Toronto Posts: 4,001	i don't have access to the db, it's hosted on another server, one of those turnkey websites. will hit you up later though