I am looking to get a list of all url's in a website. There are about 100000 or so product ID's that are spread out in no specific order. Example
www.website.com/productid=554887 can be a product and
www.website.com/productid=554888 can be a 404 error but then
www.website.com/productid=554899 can be a product again. Basically I want to rip the website by providing the urls to all the products but since the id's are random I need to build a list of links first. Hope I explained this right. Anyone know of a good software I can use to build this list?