Why does Google decide Robots.txt no longer works?

**baddog** · 12-12-2013, 04:02 PM

When did robots.txt become a server issue? Just add it in your public_html folder

**wehateporn** · 12-12-2013, 04:04 PM

Originally posted by baddog

When did robots.txt become a server issue? Just add it in your public_html folder

I've done that to fix it, but I never used to use robots.txt and Google was fine with that

**sandman!** · 12-12-2013, 04:51 PM

if a script your running generates a 500 error on a not found robots.txt google wont spider the site.

i have had this happen to me before.

**rowan** · 12-12-2013, 05:01 PM

Googlebot is probably assuming (reasonably) that if a fetch attempt returns a 500, your site has problems.

I'd be more concerned about why your server is generating a 500 internal server error for a simple 404 file not found.

**wehateporn** · 12-12-2013, 05:19 PM

Originally posted by sandman!

if a script your running generates a 500 error on a not found robots.txt google wont spider the site.

i have had this happen to me before.

**wehateporn** · 12-12-2013, 05:20 PM

Originally posted by rowan

Googlebot is probably assuming (reasonably) that if a fetch attempt returns a 500, your site has problems.

I'd be more concerned about why your server is generating a 500 internal server error for a simple 404 file not found.

Good point

**postalsex** · 12-12-2013, 06:49 PM

never question GOD!

**newB** · 12-12-2013, 07:10 PM

**ottopottomouse** · 12-12-2013, 08:08 PM

Blame the Illuminate.

**freecartoonporn** · 12-12-2013, 10:05 PM

500 is your issue, you have to solve it, either give him 404 or 200

if google is asking for it then give it one.
create a blank robots.txt

i done this before. or the bot starts hitting it and it floods the fucking error logs.

**wehateporn** · 12-13-2013, 01:22 AM

Originally posted by freecartoonporn

500 is your issue, you have to solve it, either give him 404 or 200

if google is asking for it then give it one.
create a blank robots.txt

i done this before. or the bot starts hitting it and it floods the fucking error logs.

Added blank robots, will see what Mr G thinks

**nico-t** · 12-13-2013, 06:26 AM

google fucking sucks.

**wehateporn** · 12-13-2013, 06:56 AM

Originally posted by nico-t

google fucking sucks.

I won't comment on that point in case their bots are reading

**wehateporn** · 12-13-2013, 06:57 AM

Originally posted by rowan

I'd be more concerned about why your server is generating a 500 internal server error for a simple 404 file not found.

Still not certain what caused the change, but have used .htaccess to switch the 500 to 404

**PornDiscounts-V** · 12-13-2013, 07:02 AM

Your thread title is backwards.

Why does Google decide Robots.txt no longer works?

Why does Google decide Robots.txt no longer works?

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment