Thread: Why does Google decide Robots.txt no longer works?
View Single Post
Old 12-12-2013, 07:19 PM  
wehateporn
Promoting Debate on GFY
 
wehateporn's Avatar
 
Industry Role:
Join Date: Apr 2007
Posts: 27,176
Quote:
Originally Posted by sandman! View Post
if a script your running generates a 500 error on a not found robots.txt google wont spider the site.

i have had this happen to me before.
__________________
wehateporn is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote