Quote:
Originally Posted by mkx
I am looking to get a list of all url's in a website. There are about 100000 or so product ID's that are spread out in no specific order. Example www.website.com/productid=554887 can be a product and www.website.com/productid=554888 can be a 404 error but then www.website.com/productid=554899 can be a product again. Basically I want to rip the website by providing the urls to all the products but since the id's are random I need to build a list of links first. Hope I explained this right. Anyone know of a good software I can use to build this list? |
try http://www.drk.com.ar/spider.php . It's free and pretty good, not sure if it will work with that amount of pages, but you can give it a try, it's 100% free
__________________
This post is endorsed by CIA, KGB, MI6, the Mafia, Illuminati, Kim Jong Il, Worldwide Ninjas Association, Klingon Empire and lolcats. Don't mess around with it, just accept it and embrace the truth
|