robots.txt, how important?

**chodadog** · 10-06-2002, 06:46 AM

I think you only need to use robots.txt when you don't want certain sites, or parts of your sites to be spidered. But i could be wrong.

**SABAI** · 10-06-2002, 07:10 AM

thanks.....

people bring in more info plz....

**SABAI** · 10-06-2002, 08:03 AM

bumpy dooo

**MadDog** · 10-06-2002, 09:29 AM

robots.txt is entirely optional - as said above, it can instruct spiders to ignore certain parts of your site if you desire.

The 404 errors on that file will almost entirely be spiders checking for the file before examining your site. It won't stop them from indexing your site.

**brazz** · 10-06-2002, 01:00 PM

SABAI,

I´ve seen googlebot acting weird when she doesn´t find a robots.txt, particularly when there is a .htaccess file redirecting the 404s.

I wouldn´t take chances. Robots.txt are simple to learn, and there are some scripts online which can generate it for you.

I make sure all my sites have a robots.txt

**brazz** · 10-06-2002, 01:03 PM

Check out:

http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

robots.txt, how important?

robots.txt, how important?

Comment

Comment

Comment

Comment

Comment

Comment