Why does Google decide Robots.txt no longer works?
 

On a shared server you have no robots.txt for your sites, Google has always been happy, then one day Google decides it's no longer happy with any of the sites on that server, it claims it can't access the robots.txt for them. I see it expects 404 if there's no robots.txt but my server gives Error 500, according to forums that causes a problem.

Has anyone else had this type of problem, it should be easy enough to fix, but anyone know what likely caused the change? Change of settings on the shared server?

Cheers :thumbsup

When did robots.txt become a server issue? Just add it in your public_html folder

I've done that to fix it, but I never used to use robots.txt and Google was fine with that :2 cents:

if a script your running generates a 500 error on a not found robots.txt google wont spider the site.

i have had this happen to me before.

Googlebot is probably assuming (reasonably) that if a fetch attempt returns a 500, your site has problems.

I'd be more concerned about why your server is generating a 500 internal server error for a simple 404 file not found.

:thumbsup

Good point :thumbsup

never question GOD!

Blame the Illuminate.

500 is your issue, you have to solve it, either give him 404 or 200


if google is asking for it then give it one.
create a blank robots.txt

i done this before. or the bot starts hitting it and it floods the fucking error logs.

Added blank robots, will see what Mr G thinks :2 cents:

google fucking sucks.

I won't comment on that point in case their bots are reading :winkwink:

Still not certain what caused the change, but have used .htaccess to switch the 500 to 404 :2 cents: :thumbsup

Your thread title is backwards.