Thread: Huh? Google is indexing GFY?
View Single Post
Old 08-23-2004, 09:42 AM  
rowan
Too lazy to set a custom title
 
Join Date: Mar 2002
Location: Australia
Posts: 17,393
Quote:
Originally posted by beergood
I think that the robots.txt file for gfy only causes the robot not to index the index page.

http://www.google.com/search?hl=en&l...ckyourself.com


User-agent: *
Disallow: *

for the whole site
No, the robots.txt entry for GFY is correct. "Disallow: /" will exclude the entire site from crawling.
rowan is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote