GoFuckYourself.com - Adult Webmaster Forum - View Single Post - Why does Google decide Robots.txt no longer works?

rowan · 12-12-2013, 07:01 PM

Googlebot is probably assuming (reasonably) that if a fetch attempt returns a 500, your site has problems.

I'd be more concerned about why your server is generating a 500 internal server error for a simple 404 file not found.

12-12-2013, 07:01 PM
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Googlebot is probably assuming (reasonably) that if a fetch attempt returns a 500, your site has problems. I'd be more concerned about why your server is generating a 500 internal server error for a simple 404 file not found.