Thread: Why does Google decide Robots.txt no longer works?
View Single Post
Old 12-12-2013, 06:51 PM  
sandman!
Icq: 14420613
 
sandman!'s Avatar
 
Industry Role:
Join Date: Mar 2001
Location: chicago
Posts: 15,432
if a script your running generates a 500 error on a not found robots.txt google wont spider the site.

i have had this happen to me before.
__________________
Need WebHosting ? Email me for some great deals [email protected]
sandman! is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote