I have a small project to do. haha.
I need to verify the status of about 1800 domains.
Here's what I need to know in order of importance:
1. http status code. Does it return a 200 ok? 404? redirect?
2. if redirect:
grab the ip of the domain itself
grab the domain name and html title and meta description of the site it redirects to
3. for sites that exist:
grab the html title and meta descriptions or report none/blank
grab the IP the domain is on, even if it redirects.
grab the first few hundred characters of body text, with html stripped
4. if 404 - mark the record as does not exist
5. It would be cool if the script could check the whois data to be sure my client still owns the domain. Not a priority.
This is a project where my client owns the domains. No funny business/text scraping, etc.
That's pretty much it.
I can do this by hand, of course, but it takes way too long and I really need to get this done quickly.
I need to verify the status of about 1800 domains.
Here's what I need to know in order of importance:
1. http status code. Does it return a 200 ok? 404? redirect?
2. if redirect:
grab the ip of the domain itself
grab the domain name and html title and meta description of the site it redirects to
3. for sites that exist:
grab the html title and meta descriptions or report none/blank
grab the IP the domain is on, even if it redirects.
grab the first few hundred characters of body text, with html stripped
4. if 404 - mark the record as does not exist
5. It would be cool if the script could check the whois data to be sure my client still owns the domain. Not a priority.
This is a project where my client owns the domains. No funny business/text scraping, etc.
That's pretty much it.
I can do this by hand, of course, but it takes way too long and I really need to get this done quickly.
Comment