GoFuckYourself.com - Adult Webmaster Forum - View Single Post

TripleXPrint · 10-27-2008, 09:29 AM

Robot text files are pretty much useless unless you're trying to get Google to not index a page. Google will crawl your page and index your content so just keep the robot.txt file out of your header. You really don't need it.

10-27-2008, 09:29 AM
TripleXPrint Confirmed User Join Date: Apr 2007 Posts: 983	Robot text files are pretty much useless unless you're trying to get Google to not index a page. Google will crawl your page and index your content so just keep the robot.txt file out of your header. You really don't need it. __________________ Skype: Triplexprint