|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
12-18-2007, 08:51 AM
|
#1 |
|
So Fucking Banned
|
Ever Have a Look at the Whitehouse.gov Robots.txt?
Its quite interesting. Now why wouldn't they want the search engines indexing some of that stuff? Makes you wonder...
http://www.whitehouse.gov/robots.txt |
|
|
|
12-18-2007, 08:55 AM
|
#2 |
|
Confirmed User
Join Date: Aug 2006
Location: Half way to HELL!
Posts: 301
|
Very interesting!
|
|
|
|
|
12-18-2007, 08:58 AM
|
#3 |
|
"Assassins"
Industry Role:
Join Date: Dec 2001
Location: At home
Posts: 17,277
|
That's fucking crazy..
__________________
|
|
|
|
|
12-18-2007, 09:04 AM
|
#4 |
|
www.scarlettcontent.net
Industry Role:
Join Date: Mar 2006
Location: Pleiades
Posts: 6,031
|
interesting ....
__________________
 Scarlett Content - Adult Content Provider - High Quality Adult Stock Content for your Websites, Mobile Media and Print.  Over 3 Million Images (14,000 photo sets) over 5000 Videos - Many Niches, US-2257, Awesome Prices. Over 40 years in the adult industry.  [email protected] Follow us on twitter. |
|
|
|
|
12-18-2007, 09:17 AM
|
#5 |
|
So Fucking Banned
|
Hmm, looks like I am not the first one to notice or write about it... looks like they blocked all the Iraq crap way back when.
http://www.news.com/8301-13578_3-9773721-38.html |
|
|
|
|
12-18-2007, 09:31 AM
|
#6 |
|
Hmm
Industry Role:
Join Date: Sep 2005
Location: On an endless road around the world for rock and roll.
Posts: 12,642
|
indeed interesting stuff..
__________________
|
|
|
|
|
12-18-2007, 09:35 AM
|
#7 |
|
Too lazy to set a custom title
Join Date: Dec 2006
Posts: 23,400
|
Disallow: /terrorists
__________________
i like waffles |
|
|
|
|
12-18-2007, 09:51 AM
|
#8 |
|
Too lazy to set a custom title
Join Date: Mar 2002
Location: Australia
Posts: 17,393
|
If you read the article it says that it's to stop text versions of articles being indexed. Just about every path listed there ends with /text
Here are the remaining exclusions User-agent: * Disallow: /cgi-bin Disallow: /search Disallow: /query.html Disallow: /help Disallow: /slideshow/gallery/iraq User-agent: whsearch Disallow: /cgi-bin Disallow: /search Disallow: /query.html Disallow: /help Disallow: /sitemap.html Disallow: /privacy.html Disallow: /accessibility.html |
|
|
|
|
12-18-2007, 09:54 AM
|
#9 | |
|
So Fucking Banned
|
Quote:
|
|
|
|
|
|
12-18-2007, 09:57 AM
|
#10 |
|
Confirmed User
Join Date: Mar 2007
Posts: 922
|
__________________
Vote Bill Cosby 2012 |
|
|
|
|
12-18-2007, 10:06 AM
|
#11 |
|
Damn Right I Kiss Ass!
Industry Role:
Join Date: Dec 2003
Location: Cowtown, USA
Posts: 32,409
|
I smoke crack.
|
|
|
|
|
12-18-2007, 10:24 AM
|
#12 | |
|
Too lazy to set a custom title
Join Date: Mar 2002
Location: Australia
Posts: 17,393
|
Quote:
I checked a few of those exclusions and they're 404 anyway. |
|
|
|
|
