|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
09-12-2005, 11:39 PM
|
#1 |
|
Internet Entrepeneur
Join Date: Jun 2004
Posts: 586
|
Robots.txt
What to put in it ?
__________________
[IMG] |
|
|
|
09-12-2005, 11:42 PM
|
#2 |
|
Confirmed User
Join Date: Apr 2004
Location: Very close to that Jack Daniels bottle..
Posts: 6,341
|
if you really want to block some parts of your site, do a google.
It's not neccessary if you don't need to block anything from crawling. |
|
|
|
|
09-13-2005, 12:07 AM
|
#3 |
|
So Fucking Banned
Industry Role:
Join Date: Apr 2001
Location: the beach, SoCal
Posts: 107,089
|
|
|
|
|
|
09-13-2005, 12:12 AM
|
#4 |
|
►SouthOfHeaven
Join Date: Jun 2004
Location: PlanetEarth MyBoardRank: GerbilMaster My-Penis-Size: extralarge MyWeapon: Computer
Posts: 28,609
|
put sexy messages in it to turn the robots on..
__________________
hatisblack at yahoo.com |
|
|
|
|
09-13-2005, 12:13 AM
|
#5 |
|
aliasx
Join Date: Apr 2001
Posts: 19,010
|
r2d2 ascii art
__________________
https://porncorporation.com |
|
|
|
|
09-13-2005, 12:52 AM
|
#6 | |
|
Too lazy to set a custom title
Join Date: Oct 2002
Location: Far far away - as possible
Posts: 14,956
|
Quote:
__________________
XXX TLD's - Another mosquito to swat. |
|
|
|
|
|
09-13-2005, 01:08 AM
|
#7 |
|
Confirmed User
Join Date: Dec 2004
Location: it's like: beam me up scottie. i control your body.
Posts: 147
|
unless you're using google products, otherwise don't worry about it.
|
|
|
|
|
09-13-2005, 01:11 AM
|
#8 |
|
So Fucking Banned
Join Date: Aug 2005
Location: dirty south
Posts: 997
|
robots give good head
|
|
|
|
|
09-13-2005, 02:27 AM
|
#9 |
|
Confirmed User
Industry Role:
Join Date: Oct 2002
Location: Upstate, New York
Posts: 8,187
|
User-agent: *
Disallow: /
__________________
Skype: j3nn.com ICQ 160370494 My current favorite high-converting sponsor: CrakRevenue 
|
|
|
|
|
09-13-2005, 02:37 AM
|
#10 | |
|
<&(©¿©)&>
Industry Role:
Join Date: Jul 2002
Location: Chicago
Posts: 47,882
|
Quote:
User-agent: * Disallow: / Google_PriorityID: d41d8cd98f00b204e9800998ecf8427e This google Priority ID is used by 100s of webmasters already, so it's not as effective as using a fresh ID. It's still better than using no ID at all, but I would look into obtaining your own ID.
__________________
Custom Software Development, email: woj#at#wojfun#.#com to discuss details or skype: wojl2000 or gchat: wojfun or telegram: wojl2000 Affiliate program tools: Hosted Galleries Manager Banner Manager Video Manager  Wordpress Affiliate Plugin Pic/Movie of the Day Fansign Generator Zip Manager |
|
|
|
|
|
09-13-2005, 02:39 AM
|
#11 |
|
Confirmed User
Industry Role:
Join Date: Dec 2004
Location: OGG / PDX
Posts: 456
|
I just list all the robots who are allowed to visit if they should wish to.
ASIMO C3PO HAL etc...
__________________
|
|
|
|
|
09-13-2005, 04:33 AM
|
#12 |
|
there's no $$$ in porn
Industry Role:
Join Date: Jul 2005
Location: icq: 195./568.-230 (btw: not getting offline msgs)
Posts: 33,063
|
robots.txt is a waste of bandwith
 |
|
|
|
|
09-13-2005, 06:13 AM
|
#13 | |
|
Confirmed User
Join Date: Aug 2002
Location: Brazil
Posts: 1,474
|
Quote:
__________________
 Trust the "WOW" effect. |
|
|
|
|
|
09-13-2005, 06:41 AM
|
#14 | |
|
Confirmed User
Join Date: Jun 2005
Location: rolling for CASH
Posts: 2,983
|
Quote:
|
|
|
|
|
