Why does Google decide Robots.txt no longer works? - GoFuckYourself.com

wehateporn · 12-12-2013, 06:00 PM

On a shared server you have no robots.txt for your sites, Google has always been happy, then one day Google decides it's no longer happy with any of the sites on that server, it claims it can't access the robots.txt for them. I see it expects 404 if there's no robots.txt but my server gives Error 500, according to forums that causes a problem.

Has anyone else had this type of problem, it should be easy enough to fix, but anyone know what likely caused the change? Change of settings on the shared server?

Cheers

baddog · 12-12-2013, 06:02 PM

When did robots.txt become a server issue? Just add it in your public_html folder

wehateporn · 12-12-2013, 06:04 PM

Quote:

Originally Posted by baddog

When did robots.txt become a server issue? Just add it in your public_html folder

I've done that to fix it, but I never used to use robots.txt and Google was fine with that

sandman! · 12-12-2013, 06:51 PM

if a script your running generates a 500 error on a not found robots.txt google wont spider the site.

i have had this happen to me before.

rowan · 12-12-2013, 07:01 PM

Googlebot is probably assuming (reasonably) that if a fetch attempt returns a 500, your site has problems.

I'd be more concerned about why your server is generating a 500 internal server error for a simple 404 file not found.

wehateporn · 12-12-2013, 07:19 PM

Quote:

Originally Posted by sandman!

if a script your running generates a 500 error on a not found robots.txt google wont spider the site.

i have had this happen to me before.

wehateporn · 12-12-2013, 07:20 PM

Quote:

Originally Posted by rowan

Googlebot is probably assuming (reasonably) that if a fetch attempt returns a 500, your site has problems.

I'd be more concerned about why your server is generating a 500 internal server error for a simple 404 file not found.

Good point

postalsex · 12-12-2013, 08:49 PM

never question GOD!

newB · 12-12-2013, 09:10 PM

ottopottomouse · 12-12-2013, 10:08 PM

Blame the Illuminate.

freecartoonporn · 12-13-2013, 12:05 AM

500 is your issue, you have to solve it, either give him 404 or 200

if google is asking for it then give it one.
create a blank robots.txt

i done this before. or the bot starts hitting it and it floods the fucking error logs.

wehateporn · 12-13-2013, 03:22 AM

Quote:

Originally Posted by freecartoonporn

500 is your issue, you have to solve it, either give him 404 or 200

if google is asking for it then give it one.
create a blank robots.txt

i done this before. or the bot starts hitting it and it floods the fucking error logs.

Added blank robots, will see what Mr G thinks

nico-t · 12-13-2013, 08:26 AM

google fucking sucks.

wehateporn · 12-13-2013, 08:56 AM

Quote:

Originally Posted by nico-t

google fucking sucks.

I won't comment on that point in case their bots are reading

wehateporn · 12-13-2013, 08:57 AM

Quote:

Originally Posted by rowan

I'd be more concerned about why your server is generating a 500 internal server error for a simple 404 file not found.

Still not certain what caused the change, but have used .htaccess to switch the 500 to 404

PornDiscounts-V · 12-13-2013, 09:02 AM

Your thread title is backwards.

12-12-2013, 06:00 PM	#1
wehateporn Promoting Debate on GFY Industry Role: Join Date: Apr 2007 Posts: 27,173	Why does Google decide Robots.txt no longer works? On a shared server you have no robots.txt for your sites, Google has always been happy, then one day Google decides it's no longer happy with any of the sites on that server, it claims it can't access the robots.txt for them. I see it expects 404 if there's no robots.txt but my server gives Error 500, according to forums that causes a problem. Has anyone else had this type of problem, it should be easy enough to fix, but anyone know what likely caused the change? Change of settings on the shared server? Cheers __________________ Sexy Girls Teasing

12-12-2013, 06:51 PM	#4
sandman! Icq: 14420613 Industry Role: Join Date: Mar 2001 Location: chicago Posts: 15,432	if a script your running generates a 500 error on a not found robots.txt google wont spider the site. i have had this happen to me before. __________________ Need WebHosting ? Email me for some great deals [email protected]

12-12-2013, 09:10 PM	#9
newB Confirmed User Industry Role: Join Date: Jul 2006 Location: Somewhere between reality and total ape-shit bonkers. Posts: 2,870	__________________ The best Adult Affiliate Programs reviewed and indexed by niche and feature. Easily find the sponsors that suit your needs.

12-12-2013, 10:08 PM	#10
ottopottomouse She is ugly, bad luck. Industry Role: Join Date: Jan 2010 Posts: 13,177	Blame the Illuminate. __________________ ↑ see post ↑ 13101

12-13-2013, 12:05 AM	#11
freecartoonporn Confirmed User Industry Role: Join Date: Jan 2012 Location: NC Posts: 7,683	500 is your issue, you have to solve it, either give him 404 or 200 if google is asking for it then give it one. create a blank robots.txt i done this before. or the bot starts hitting it and it floods the fucking error logs. __________________ SSD Cloud Server, VPS Server, Simple Cloud Hosting \| DigitalOcean

12-12-2013, 06:02 PM	#2
baddog So Fucking Banned Industry Role: Join Date: Apr 2001 Location: the beach, SoCal Posts: 107,089	When did robots.txt become a server issue? Just add it in your public_html folder

12-12-2013, 07:01 PM	#5
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Googlebot is probably assuming (reasonably) that if a fetch attempt returns a 500, your site has problems. I'd be more concerned about why your server is generating a 500 internal server error for a simple 404 file not found.

12-12-2013, 08:49 PM	#8
postalsex Confirmed User Industry Role: Join Date: Dec 2013 Posts: 400	never question GOD!

12-13-2013, 08:26 AM	#13
nico-t emperor of my world Join Date: Aug 2004 Location: nethalands Posts: 29,903	google fucking sucks.

12-13-2013, 09:02 AM	#16
PornDiscounts-V Confirmed User Industry Role: Join Date: Oct 2003 Location: L.A. Posts: 5,744	Your thread title is backwards. __________________ Blog Posts - Contextual Links - Hardlinks on 600+ Blog Network * Handwritten * 180 C Class IPs * Permanent! * Many Niches! * Bulk Discounts! GFYPosts /at/ J2Media.net