|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
02-28-2003, 10:36 AM
|
#1 |
|
Registered User
Join Date: Feb 2003
Posts: 48
|
Robots.txt
What do YOU have in your robots file?
__________________
<a href="http://www.wethoney.com"><img src="http://wethoney.com/images/banners/wethoney_120x60_02.gif" border="0"></a> |
|
|
|
02-28-2003, 10:46 AM
|
#2 |
|
Confirmed User
Join Date: Feb 2002
Location: Canada
Posts: 2,370
|
Mine is pretty small at the moment. Had better ones in the past, not sure where they went though.
--- User-Agent: GetRight Disallow: / User-Agent: Go!Zilla Disallow: / User-Agent: Teleport Disallow: /
__________________
"People can have the Model T in any color - so long as it's black." - Henry Ford |
|
|
|
|
02-28-2003, 12:33 PM
|
#3 |
|
Confirmed User
Industry Role:
Join Date: Oct 2002
Location: lalaland
Posts: 2,120
|
User-agent: URL2File
Disallow: / User-agent: TE Disallow: / User-agent: BaiDuSpider Disallow: / User-agent: Offline Explorer Disallow: / User-agent: WebSauger Disallow: / User-agent: Mister PiX Disallow: / User-agent: Internet Ninja Disallow: / User-agent: Microsoft URL Control Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Teleport Disallow: / User-agent: asterias Disallow: / User-agent: wget Disallow: / User-agent: Web Downloader Disallow: / User-agent: nearsite Disallow: / User-agent: netattache Disallow: / User-agent: sitesnagger Disallow: / User-agent: webwhacker Disallow: / User-agent: digout4u Disallow: / User-agent: extractorpro Disallow: / User-agent: go-ahead-got-it Disallow: / User-agent: HTTPClient Disallow: / User-agent: TovekTools Web Indexer Disallow: / User-agent: LinkWalker Disallow: / User-agent: NEWT ActiveX Disallow: / User-agent: GetRight Disallow: / User-agent: WebZIP Disallow: / User-agent: WebReaper Disallow: / User-agent: HTTrack Disallow: / |
|
|
|
|
02-28-2003, 12:50 PM
|
#4 |
|
Confirmed User
Join Date: Jan 2002
Location: In the walls of your house.
Posts: 3,985
|
Um... Do you really think any of those site-sucking programs pay attention to robots.txt directives?
__________________
"Every normal man must be tempted, at times, to spit on his hands, hoist the black flag, and begin slitting throats." --H.L. Mencken |
|
|
|
|
03-29-2003, 09:22 PM
|
#5 |
|
Confirmed User
Join Date: Mar 2003
Posts: 133
|
<H1>COCK SMOKERS!!</H1>
|
|
|
|
