Ever Have a Look at the Whitehouse.gov Robots.txt?
 

Its quite interesting. Now why wouldn't they want the search engines indexing some of that stuff? Makes you wonder...

http://www.whitehouse.gov/robots.txt

Very interesting!

That's fucking crazy..

interesting ....:)

Hmm, looks like I am not the first one to notice or write about it... looks like they blocked all the Iraq crap way back when.

http://www.news.com/8301-13578_3-9773721-38.html

indeed interesting stuff..

Disallow: /terrorists

If you read the article it says that it's to stop text versions of articles being indexed. Just about every path listed there ends with /text

Here are the remaining exclusions

User-agent: *
Disallow: /cgi-bin
Disallow: /search
Disallow: /query.html
Disallow: /help
Disallow: /slideshow/gallery/iraq

User-agent: whsearch
Disallow: /cgi-bin
Disallow: /search
Disallow: /query.html
Disallow: /help
Disallow: /sitemap.html
Disallow: /privacy.html
Disallow: /accessibility.html

Still, why would they want to keep people off the text versions?

I smoke crack.

Duplicate content?

I checked a few of those exclusions and they're 404 anyway.