|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
04-02-2003, 01:32 PM
|
#1 |
|
I need a beer
 
Industry Role:
Join Date: Jun 2002
Location: ♠ Toiletville ♠
Posts: 133,944
|
What Do I Put In My Robot.txt File?
I'm trying to optimize my SE traffic. I was told to put this in my main directory
# /robots.txt for http://www.yoursite.com/ # comments to [email protected]/ User-agent: * Disallow: Allow: * Im cofused as to what I place in for user-agent and disallow/allow parts.And does this help with robots spidering your other pages?Can anybody help with this?.Much appreciated.
__________________
|
|
|
|
04-02-2003, 01:43 PM
|
#2 |
|
Confirmed User
Join Date: Dec 2002
Location: Austin, TX
Posts: 2,115
|
__________________
1monkey0cup.com |
|
|
|
|
04-02-2003, 01:53 PM
|
#3 |
|
Confirmed User
Join Date: Jan 2002
Location: In the walls of your house.
Posts: 3,985
|
robots.txt files aren't particularly important, unless you get annoyed with the 404 entries in your logs, or you're trying to keep a particular search engine out of part of your site.
For example, if you built a set of pages specifically for Inktomi on http://www.yoursite.com/inktomi/ and another set for Altavista on http://www.yoursite.com/altavista/ you could do the following: User-agent: * Disallow: User-agent: Slurp Disallow: /altavista/ User-agent: Scooter Disallow: /inktomi/ This will keep Scooter out of the inktomi files and Slurp out of the Altavista files. For more information check out the following website: http://www.robotstxt.org/wc/robots.html
__________________
"Every normal man must be tempted, at times, to spit on his hands, hoist the black flag, and begin slitting throats." --H.L. Mencken |
|
|
|
|
04-02-2003, 01:57 PM
|
#4 |
|
I need a beer
Industry Role:
Join Date: Jun 2002
Location: ♠ Toiletville ♠
Posts: 133,944
|
Cool..Thanks for the information
__________________
|
|
|
|
