SE experts, need advice on pages with feeds - GoFuckYourself.com - Adult Webmaster Forum

X

Dravyk

Confirmed User

Join Date: Jan 2001

Posts: 1202
Share

Post
#1

SE experts, need advice on pages with feeds

06-23-2004, 11:03 AM

I have a site that uses RSS feeds generated via J avascript and other embedded execuatbles. As these pages are generated "within browser" -- much like an SSI does -- a look through a few search engine simulators suggests that SE spiders would find mostly blank pages there. While a browser would see constantly-updating content.

Anyone know of a fix or a script that will turn these into "static" pages that SEs will spider?

All Of 'Em
Tags: None
WiredGuy

Pounding Googlebot

Join Date: Aug 2002

Posts: 34517
Share

Post
#2

06-23-2004, 11:05 AM

Originally posted by Dravyk
I have a site that uses RSS feeds generated via J avascript and other embedded execuatbles. As these pages are generated "within browser" -- much like an SSI does -- a look through a few search engine simulators suggests that SE spiders would find mostly blank pages there. While a browser would see constantly-updating content.

Anyone know of a fix or a script that will turn these into "static" pages that SEs will spider?

Here's an example: http://www.submitcorner.com/
The feeds on the right side of the screen under latest headlines come from Moreover. The way I do it, is I get their data through an XML feed and have a script create the HTML code that will be used on the right panel. The html code is then saved into an SSI include and loaded on runtime. This way, browsers and bots will see the same text and nobody needs to know its an xml feed.

WG

I play with Google.
Comment
Basic_man

Programming King Pin

Join Date: Oct 2003

Posts: 27360
Share

Post
#3

06-23-2004, 11:20 AM

Originally posted by WiredGuy
Here's an example: http://www.submitcorner.com/
The feeds on the right side of the screen under latest headlines come from Moreover. The way I do it, is I get their data through an XML feed and have a script create the HTML code that will be used on the right panel. The html code is then saved into an SSI include and loaded on runtime. This way, browsers and bots will see the same text and nobody needs to know its an xml feed.

WG

Wow, nice explanation!

UUGallery Builder - automated photo/video gallery plugin for Wordpress!
Stop looking! Checkout Naked Hosting, online since 1999 !
Comment
Dravyk

Confirmed User

Join Date: Jan 2001

Posts: 1202
Share

Post
#4

06-23-2004, 11:30 AM

Very nice, WG!!

I had both an RSS converter built awhile back for me, as well as a site cacher for when those feeds sometimes have Net congestion probs and hold up the pageloads.

Looks like (which is what I pretty much figured) I'll also need to get another one made up to do the static thing. Glad to see it's possible and it exists. Again, many thanks.

All Of 'Em
Comment
WiredGuy

Pounding Googlebot

Join Date: Aug 2002

Posts: 34517
Share

Post
#5

06-23-2004, 11:34 AM

Originally posted by Dravyk
Very nice, WG!!

I had both an RSS converter built awhile back for me, as well as a site cacher for when those feeds sometimes have Net congestion probs and hold up the pageloads.

Looks like (which is what I pretty much figured) I'll also need to get another one made up to do the static thing. Glad to see it's possible and it exists. Again, many thanks.

This also cuts back on load time and bandwidth significantly. If you're using XML feeds on the fly, you download the data, parse it and spit out the results in realtime. Using the method above, the load time is static since your just opening a file and you eliminate constantly requesting the feed. I just refresh the data every 4 hours. So from a couple thousand times per day to 6 data loads.

WG

I play with Google.
Comment
Namzo

Registered User

Join Date: Jun 2004

Posts: 83
Share

Post
#6

06-25-2004, 08:24 PM

Originally posted by WiredGuy
Here's an example: http://www.submitcorner.com/
The feeds on the right side of the screen under latest headlines come from Moreover. The way I do it, is I get their data through an XML feed and have a script create the HTML code that will be used on the right panel. The html code is then saved into an SSI include and loaded on runtime. This way, browsers and bots will see the same text and nobody needs to know its an xml feed.

WG

Is this technically considered leeching content by some feed providers?
Comment
Webby

Too lazy to set a custom title

Join Date: Oct 2002

Posts: 14956
Share

Post
#7

06-25-2004, 08:30 PM

WiredGuy

Nice idea!!

XXX TLD's - Another mosquito to swat.
Comment
WiredGuy

Pounding Googlebot

Join Date: Aug 2002

Posts: 34517
Share

Post
#8

06-25-2004, 08:38 PM

Originally posted by Namzo
Is this technically considered leeching content by some feed providers?

If you're licensing the content it's perfectly fine. In fact, if you're doing the implementation I suggested above, you'd be using extremely little bandwidth from the feed provider since you only request it a couple times per day versus per every surfer.

WG

I play with Google.
Comment
smack

Push Porn Like Weight.

Join Date: Mar 2002

Posts: 10652
Share

Post
#9

06-25-2004, 09:40 PM

thanks for the tip WG.

*bump for some good info*

Cry havoc and let slip the dogs of war.
Comment
Abyss_Vee

Confirmed User

Join Date: Sep 2003

Posts: 5208
Share

Post
#10

06-25-2004, 09:44 PM

good stuff wg
Comment
Namzo

Registered User

Join Date: Jun 2004

Posts: 83
Share

Post
#11

06-25-2004, 10:09 PM

Originally posted by WiredGuy
If you're licensing the content it's perfectly fine. In fact, if you're doing the implementation I suggested above, you'd be using extremely little bandwidth from the feed provider since you only request it a couple times per day versus per every surfer.

WG

So do you think doing it with a free feed is kinda a gray area and possibly ok, especially if you are using not using noticible bandwidth from them depending on each provider's TOS or a definite no?

Thanks
Comment

Previous template Next

Working...