Thread: Huh? Google is indexing GFY?
View Single Post
Old 08-23-2004, 09:45 AM  
beergood
Confirmed User
 
Join Date: Jun 2003
Location: United States
Posts: 2,918
Quote:
Originally posted by rowan
No, the robots.txt entry for GFY is correct. "Disallow: /" will exclude the entire site from crawling.

See I wasn't entirely sure because I never blocked their bot. You are absolutely right. I looked it up.
beergood is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote