leaving out robots.txt ? - GoFuckYourself.com

rapmaster · 06-02-2007, 12:30 PM

What are the negative consequences of having a missing robots.txt file?

I understand that it could waste a tiny bit of bandwidth, but could this hurt in the SE rankings?

Oracle Porn · 06-02-2007, 12:35 PM

no......

rapmaster · 06-02-2007, 12:41 PM

Quote:

Originally Posted by Oracle Porn

no......

Alright thanks

TheDoc · 06-02-2007, 01:04 PM

Not every site needs a robots.txt file - not every site would benefit by having one. Most sites however do benefit by having a robots.txt file

The largest benefit is the ability to tell google, yahoo, and ask which sections/pages (like affiliate code redirect pages) to not list in google. Doing this keeps rank values focused and not on needless pages or pages that could get you into trouble (like redirects, popup codes, iframes)

I block ai archiver too. You can use it to block mass download bots for TGP's, or include the url to your sitemap.

Tempest · 06-02-2007, 01:05 PM

You probably want to get around to using robots.txt. Apparently there's a new command that lets you tell the SE bots where your sitemap is.

JamesK2 · 06-02-2007, 01:07 PM

Quote:

Originally Posted by Tempest

You probably want to get around to using robots.txt. Apparently there's a new command that lets you tell the SE bots where your sitemap is.

That's some nice info, thanks

CIVMatt · 06-02-2007, 01:56 PM

You should have one even if it's blank

06-02-2007, 12:30 PM	#1
rapmaster Confirmed User Join Date: Jan 2007 Posts: 1,576	leaving out robots.txt ? What are the negative consequences of having a missing robots.txt file? I understand that it could waste a tiny bit of bandwidth, but could this hurt in the SE rankings? __________________ http://en.wikipedia.org/wiki/SIG_Sauer_P226

06-02-2007, 12:35 PM	#2
Oracle Porn Affiliate Industry Role: Join Date: Oct 2002 Location: Icq: 94-399-723 Posts: 24,433	no...... __________________

06-02-2007, 01:04 PM	#4
TheDoc Too lazy to set a custom title Industry Role: Join Date: Jul 2001 Location: Currently Incognito Posts: 13,827	Not every site needs a robots.txt file - not every site would benefit by having one. Most sites however do benefit by having a robots.txt file The largest benefit is the ability to tell google, yahoo, and ask which sections/pages (like affiliate code redirect pages) to not list in google. Doing this keeps rank values focused and not on needless pages or pages that could get you into trouble (like redirects, popup codes, iframes) I block ai archiver too. You can use it to block mass download bots for TGP's, or include the url to your sitemap. __________________ ~TheDoc - ICQ7765825 It's all disambiguation

06-02-2007, 01:56 PM	#7
CIVMatt Amateur Pimpin Industry Role: Join Date: Aug 2004 Location: Orlando, FL Posts: 13,075	You should have one even if it's blank __________________ Make easy money with Webcams

06-02-2007, 01:05 PM	#5
Tempest Too lazy to set a custom title Industry Role: Join Date: May 2004 Location: West Coast, Canada. Posts: 10,217	You probably want to get around to using robots.txt. Apparently there's a new command that lets you tell the SE bots where your sitemap is.