Thread: Ever Have a Look at the Whitehouse.gov Robots.txt?
View Single Post
Old 12-18-2007, 10:24 AM  
rowan
Too lazy to set a custom title
 
Join Date: Mar 2002
Location: Australia
Posts: 17,393
Quote:
Originally Posted by minusonebit View Post
Still, why would they want to keep people off the text versions?
Duplicate content?

I checked a few of those exclusions and they're 404 anyway.
rowan is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote