how do I stop google from including my RSS feed in normal SERPs? - GoFuckYourself.com

rowan · 10-21-2007, 07:04 AM

Google is including my RSS feed as a search result - it doesn't seem to recognise/care that it's in XML...

The obvious solution would be to exclude that file with robots.txt, but I'm concerned that it may affect some services which should be accessing that file.

Anyone run into this problem before?

beta-tester · 10-21-2007, 07:06 AM

you don't have to worry about google's including your rss feeds. I asked Matt Cutts on his blog and he answered that there's no reason to block spiders from indexing your rss feeds. They'll simply distinguish between real page and the feed, and display original page in serps.

rowan · 10-21-2007, 07:17 AM

Quote:

Originally Posted by beta-tester

you don't have to worry about google's including your rss feeds. I asked Matt Cutts on his blog and he answered that there's no reason to block spiders from indexing your rss feeds. They'll simply distinguish between real page and the feed, and display original page in serps.

Hmmm... this is not a hypothetical concern, Google is actually linking to rss.php in some of its results. Other search queries link to / as expected.

144.139.37.*- - [21/Oct/2007:14:03:38 +1000] "GET /rss.php HTTP/1.1" 200 31210 http://www.google.com.au/search?hl=en&q=***&btnG=Search&meta=cr%3DcountryAU

rowan · 10-21-2007, 07:29 AM

I could exclude rss.php for user-agent Googlebot only, but I'm not sure if their bloggy and RSS specific stuff also uses Googlebot.

It's not really a big problem, but it does mean there's potential for someone unfamiliar with XML to see "junk" when they click on a result, and get confused/scared off...

beta-tester · 10-21-2007, 08:14 AM

excluding rss.php for user-agent Googlebot is against Google's quality guidelines and such behavior will be seen as spam, hence the site will be probably banned. So I don't recommend this solution.

If that rss thing really bothers you, just exclude rss.php (or whatever) via simple robots.txt Disallow clause and you're all set.

beta-tester · 10-21-2007, 08:19 AM

just to clear up... when i wrote that excluding a page only for certain user-agent sting i meant excluding page via php, javascript and other 'sneaky' technique.

It's of course legitimate and legal to exclude the page via robots.txt and specify User-Agent: clause to address to certain search engines bot.

You could do that w/o problems. I am pretty much certain other google bots (media, images, etc) will keep indexing that page.

Bird · 10-21-2007, 08:21 AM

exclude that shit in robots.txt file

10-21-2007, 07:04 AM	#1
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	how do I stop google from including my RSS feed in normal SERPs? Google is including my RSS feed as a search result - it doesn't seem to recognise/care that it's in XML... The obvious solution would be to exclude that file with robots.txt, but I'm concerned that it may affect some services which should be accessing that file. Anyone run into this problem before?

10-21-2007, 07:06 AM	#2
beta-tester Rock 'n Roll Baby! Join Date: Sep 2004 Location: USA, temporarly Posts: 22,562	you don't have to worry about google's including your rss feeds. I asked Matt Cutts on his blog and he answered that there's no reason to block spiders from indexing your rss feeds. They'll simply distinguish between real page and the feed, and display original page in serps. __________________ Sig for sale. Affordable prices. Contact me and get a great deal ;) My contact: ICQ: 944-320-46 e-mail: manca {AT} HotFreeSex4All.com

10-21-2007, 08:14 AM	#5
beta-tester Rock 'n Roll Baby! Join Date: Sep 2004 Location: USA, temporarly Posts: 22,562	excluding rss.php for user-agent Googlebot is against Google's quality guidelines and such behavior will be seen as spam, hence the site will be probably banned. So I don't recommend this solution. If that rss thing really bothers you, just exclude rss.php (or whatever) via simple robots.txt Disallow clause and you're all set. __________________ Sig for sale. Affordable prices. Contact me and get a great deal ;) My contact: ICQ: 944-320-46 e-mail: manca {AT} HotFreeSex4All.com

10-21-2007, 08:19 AM	#6
beta-tester Rock 'n Roll Baby! Join Date: Sep 2004 Location: USA, temporarly Posts: 22,562	just to clear up... when i wrote that excluding a page only for certain user-agent sting i meant excluding page via php, javascript and other 'sneaky' technique. It's of course legitimate and legal to exclude the page via robots.txt and specify User-Agent: clause to address to certain search engines bot. You could do that w/o problems. I am pretty much certain other google bots (media, images, etc) will keep indexing that page. __________________ Sig for sale. Affordable prices. Contact me and get a great deal ;) My contact: ICQ: 944-320-46 e-mail: manca {AT} HotFreeSex4All.com

10-21-2007, 08:21 AM	#7
Bird Confirmed User Join Date: Jan 2005 Location: Stockton Posts: 4,365	exclude that shit in robots.txt file __________________ ICQ:268731675

10-21-2007, 07:29 AM	#4
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	I could exclude rss.php for user-agent Googlebot only, but I'm not sure if their bloggy and RSS specific stuff also uses Googlebot. It's not really a big problem, but it does mean there's potential for someone unfamiliar with XML to see "junk" when they click on a result, and get confused/scared off...