|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
10-06-2002, 07:41 AM
|
#1 |
|
Confirmed User
Join Date: Jan 2001
Location: footmaniac.com
Posts: 2,880
|
robots.txt, how important?
i noticed on some of my servers that the most common document not found was robots.txt...
how important is it? whats the use / benefits of it and what should it contain? thanks in advance sabai
__________________
|
|
|
|
10-06-2002, 07:46 AM
|
#2 |
|
Confirmed User
Join Date: Apr 2002
Posts: 9,736
|
I think you only need to use robots.txt when you don't want certain sites, or parts of your sites to be spidered. But i could be wrong.
|
|
|
|
|
10-06-2002, 08:10 AM
|
#3 |
|
Confirmed User
Join Date: Jan 2001
Location: footmaniac.com
Posts: 2,880
|
thanks.....
people bring in more info plz.... 
__________________
|
|
|
|
|
10-06-2002, 09:03 AM
|
#4 |
|
Confirmed User
Join Date: Jan 2001
Location: footmaniac.com
Posts: 2,880
|
bumpy dooo
__________________
|
|
|
|
|
10-06-2002, 10:29 AM
|
#5 |
|
Registered User
Join Date: Jul 2002
Location: Toronto
Posts: 97
|
robots.txt is entirely optional - as said above, it can instruct spiders to ignore certain parts of your site if you desire.
The 404 errors on that file will almost entirely be spiders checking for the file before examining your site. It won't stop them from indexing your site.
__________________
<p><font size="2" face="Arial, Helvetica, sans-serif"><a href="http://www.maddoghouse.com"><img src="http://www.maddoghouse.com/images_global/mdhlogo60x45_GFYbutton_black.jpg" width="60" height="45" border="0">MadDogHouse.com</a></font></p> |
|
|
|
|
10-06-2002, 02:00 PM
|
#6 |
|
Confirmed User
Join Date: May 2001
Posts: 135
|
SABAI,
I´ve seen googlebot acting weird when she doesn´t find a robots.txt, particularly when there is a .htaccess file redirecting the 404s. I wouldn´t take chances. Robots.txt are simple to learn, and there are some scripts online which can generate it for you. I make sure all my sites have a robots.txt |
|
|
|
|
10-06-2002, 02:03 PM
|
#7 |
|
Confirmed User
Join Date: May 2001
Posts: 135
|
|
|
|
|
