GoFuckYourself.com - Adult Webmaster Forum

GoFuckYourself.com - Adult Webmaster Forum (https://gfy.com/index.php)
-   Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)
-   -   leaving out robots.txt ? (https://gfy.com/showthread.php?t=738941)

rapmaster 06-02-2007 12:30 PM

leaving out robots.txt ?
 
What are the negative consequences of having a missing robots.txt file?

I understand that it could waste a tiny bit of bandwidth, but could this hurt in the SE rankings?

Oracle Porn 06-02-2007 12:35 PM

no......

rapmaster 06-02-2007 12:41 PM

Quote:

Originally Posted by Oracle Porn (Post 12532004)
no......

Alright thanks

TheDoc 06-02-2007 01:04 PM

Not every site needs a robots.txt file - not every site would benefit by having one. Most sites however do benefit by having a robots.txt file

The largest benefit is the ability to tell google, yahoo, and ask which sections/pages (like affiliate code redirect pages) to not list in google. Doing this keeps rank values focused and not on needless pages or pages that could get you into trouble (like redirects, popup codes, iframes)

I block ai archiver too. You can use it to block mass download bots for TGP's, or include the url to your sitemap.

Tempest 06-02-2007 01:05 PM

You probably want to get around to using robots.txt. Apparently there's a new command that lets you tell the SE bots where your sitemap is.

JamesK2 06-02-2007 01:07 PM

Quote:

Originally Posted by Tempest (Post 12532097)
You probably want to get around to using robots.txt. Apparently there's a new command that lets you tell the SE bots where your sitemap is.

That's some nice info, thanks :thumbsup

CIVMatt 06-02-2007 01:56 PM

You should have one even if it's blank


All times are GMT -7. The time now is 05:39 AM.

Powered by vBulletin® Version 3.8.8
Copyright ©2000 - 2026, vBulletin Solutions, Inc.
©2000-, AI Media Network Inc123