|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
12-12-2013, 06:00 PM
|
#1 |
|
Promoting Debate on GFY
Industry Role:
Join Date: Apr 2007
Posts: 27,173
|
Why does Google decide Robots.txt no longer works?
On a shared server you have no robots.txt for your sites, Google has always been happy, then one day Google decides it's no longer happy with any of the sites on that server, it claims it can't access the robots.txt for them. I see it expects 404 if there's no robots.txt but my server gives Error 500, according to forums that causes a problem.
Has anyone else had this type of problem, it should be easy enough to fix, but anyone know what likely caused the change? Change of settings on the shared server? Cheers 
__________________
|
|
|
|
12-12-2013, 06:02 PM
|
#2 |
|
So Fucking Banned
Industry Role:
Join Date: Apr 2001
Location: the beach, SoCal
Posts: 107,089
|
When did robots.txt become a server issue? Just add it in your public_html folder
|
|
|
|
|
12-12-2013, 06:04 PM
|
#3 | |
|
Promoting Debate on GFY
Industry Role:
Join Date: Apr 2007
Posts: 27,173
|
Quote:
__________________
|
|
|
|
|
|
12-12-2013, 06:51 PM
|
#4 |
|
Icq: 14420613
Industry Role:
Join Date: Mar 2001
Location: chicago
Posts: 15,432
|
if a script your running generates a 500 error on a not found robots.txt google wont spider the site.
i have had this happen to me before.
__________________
Need WebHosting ? Email me for some great deals [email protected] |
|
|
|
|
12-12-2013, 07:01 PM
|
#5 |
|
Too lazy to set a custom title
Join Date: Mar 2002
Location: Australia
Posts: 17,393
|
Googlebot is probably assuming (reasonably) that if a fetch attempt returns a 500, your site has problems.
I'd be more concerned about why your server is generating a 500 internal server error for a simple 404 file not found. |
|
|
|
|
12-12-2013, 07:19 PM
|
#6 | |
|
Promoting Debate on GFY
Industry Role:
Join Date: Apr 2007
Posts: 27,173
|
Quote:
__________________
|
|
|
|
|
|
12-12-2013, 07:20 PM
|
#7 | |
|
Promoting Debate on GFY
Industry Role:
Join Date: Apr 2007
Posts: 27,173
|
Quote:
__________________
|
|
|
|
|
|
12-12-2013, 08:49 PM
|
#8 |
|
Confirmed User
Industry Role:
Join Date: Dec 2013
Posts: 400
|
never question GOD!
|
|
|
|
|
12-12-2013, 09:10 PM
|
#9 |
|
Confirmed User
Industry Role:
Join Date: Jul 2006
Location: Somewhere between reality and total ape-shit bonkers.
Posts: 2,870
|
__________________
The best Adult Affiliate Programs reviewed and indexed by niche and feature. Easily find the sponsors that suit your needs. |
|
|
|
|
12-12-2013, 10:08 PM
|
#10 |
|
She is ugly, bad luck.
Industry Role:
Join Date: Jan 2010
Posts: 13,177
|
Blame the Illuminate.
__________________
↑ see post ↑ 13101 |
|
|
|
|
12-13-2013, 12:05 AM
|
#11 |
|
Confirmed User
Industry Role:
Join Date: Jan 2012
Location: NC
Posts: 7,683
|
500 is your issue, you have to solve it, either give him 404 or 200
if google is asking for it then give it one. create a blank robots.txt i done this before. or the bot starts hitting it and it floods the fucking error logs.
__________________
SSD Cloud Server, VPS Server, Simple Cloud Hosting | DigitalOcean
|
|
|
|
|
12-13-2013, 03:22 AM
|
#12 | |
|
Promoting Debate on GFY
Industry Role:
Join Date: Apr 2007
Posts: 27,173
|
Quote:
Added blank robots, will see what Mr G thinks
__________________
|
|
|
|
|
|
12-13-2013, 08:26 AM
|
#13 |
|
emperor of my world
Join Date: Aug 2004
Location: nethalands
Posts: 29,903
|
google fucking sucks.
|
|
|
|
|
12-13-2013, 08:56 AM
|
#14 | |
|
Promoting Debate on GFY
Industry Role:
Join Date: Apr 2007
Posts: 27,173
|
Quote:
__________________
|
|
|
|
|
|
12-13-2013, 08:57 AM
|
#15 | |
|
Promoting Debate on GFY
Industry Role:
Join Date: Apr 2007
Posts: 27,173
|
Quote:
__________________
|
|
|
|
|
|
12-13-2013, 09:02 AM
|
#16 |
|
Confirmed User
Industry Role:
Join Date: Oct 2003
Location: L.A.
Posts: 5,744
|
Your thread title is backwards.
__________________
 Blog Posts - Contextual Links - Hardlinks on 600+ Blog Network * Handwritten * 180 C Class IPs * Permanent! * Many Niches! * Bulk Discounts! GFYPosts /at/ J2Media.net |
|
|
|
