How to make script which crawl posts from blogs/forums? - GoFuckYourself.com

Klen · 02-13-2007, 06:33 AM

Since there is now millions of forums and blogs,i think it will be good to make script which will crawl posts from there,something like rss feed.With blogs that should'nt be problem since most blogs include rss feed,but forums not.Any idea how to do it?

Adultnet · 02-13-2007, 06:36 AM

this is old and was made..

fris · 02-13-2007, 06:53 AM

blogs you can do, but why would you want forum posts?

tolik · 02-13-2007, 07:15 AM

use libnet if you know perl

Kimo · 02-13-2007, 07:24 AM

good luck....

Klen · 02-13-2007, 08:23 AM

Quote:

Originally Posted by Fris

blogs you can do, but why would you want forum posts?

Well there are several big forums which have buy/sell sections,so that would be easier for tracking those sections.

Jace · 02-13-2007, 08:26 AM

vbulletin

external.php?type=rss2

Klen · 02-13-2007, 09:20 AM

Quote:

Originally Posted by Jace

vbulletin

external.php?type=rss2

That is default function or mod?

StuartD · 02-13-2007, 09:23 AM

board tracker.com

Klen · 02-13-2007, 09:29 AM

Hmm it seems that command working on few board which i tryed,but it show all new posts/threads.I would more preffer if i can select specific categories,but that probaly depend on settings of rss feed.

Damian_Maxcash · 02-13-2007, 09:34 AM

Its ez to crawl the forums and get them into a database - phpDig for example - Its getting that info into a usable format that is hard.

Jace · 02-13-2007, 09:43 AM

Quote:

Originally Posted by KlenTelaris

That is default function or mod?

look man, I know asking question is great and all to learn, but try and figure some stuff out for yourself, it sinks in longer that way

rowan · 02-13-2007, 10:10 AM

Quote:

Originally Posted by KlenTelaris

Since there is now millions of forums and blogs,i think it will be good to make script which will crawl posts from there,something like rss feed.With blogs that should'nt be problem since most blogs include rss feed,but forums not.Any idea how to do it?

Is this so you can find newer posts instead of bumping 3 year old threads?

Klen · 02-13-2007, 10:20 AM

Quote:

Originally Posted by Jace

look man, I know asking question is great and all to learn, but try and figure some stuff out for yourself, it sinks in longer that way

Ok hehe.Not a problem.

Klen · 02-13-2007, 10:22 AM

Quote:

Originally Posted by rowan

Is this so you can find newer posts instead of bumping 3 year old threads?

I just want to see all news at one place.It is much faster then checking site one by one.Also i can add feature which will if specific word apper to alert me over mail.

darksoul · 02-13-2007, 10:56 AM

Quote:

Originally Posted by Jace

vbulletin

external.php?type=rss2

pretty neat
thanks

LustyBucks · 02-13-2007, 11:17 AM

Take a look here - http://php.resourceindex.com.

02-13-2007, 06:33 AM	#1
Klen Industry Role: Join Date: Aug 2006 Location: Little Vienna Posts: 32,235	How to make script which crawl posts from blogs/forums? Since there is now millions of forums and blogs,i think it will be good to make script which will crawl posts from there,something like rss feed.With blogs that should'nt be problem since most blogs include rss feed,but forums not.Any idea how to do it?

02-13-2007, 06:36 AM	#2
Adultnet Confirmed User Join Date: Sep 2003 Posts: 8,713	this is old and was made.. __________________ TrafficCashGold Paying Webmasters Since 1996! Awesome Conversions! Fast Weekly Payments! Over 125 Tours!

02-13-2007, 07:15 AM	#4
tolik Confirmed User Join Date: Jun 2001 Location: inside my brain Posts: 2,035	use libnet if you know perl __________________ LONGBUCKS-Teen, Mature, Reality, Gay sites. Free hosting (cgi/php/mysql/cron etc). FHGs. Free content.Great rebills. Support icq - 313-882-945 Webmaster friendly high recommended programs to promote (personal experience): RoyalCash-Teen ProfitX -Video SmokinCash-Amateur,Lesbo,Squirting

02-13-2007, 07:24 AM	#5
Kimo ... Join Date: Jan 2006 Location: Maryland ICQ:87038677 Posts: 11,542	good luck.... __________________ ...

02-13-2007, 09:23 AM	#9
StuartD Sofa King Band Join Date: Jul 2002 Location: Outside the box Posts: 29,903	board tracker.com __________________ This is me on facebook This is me on twitter

02-13-2007, 06:53 AM	#3
fris Too lazy to set a custom title Industry Role: Join Date: Aug 2002 Posts: 55,382	blogs you can do, but why would you want forum posts?

02-13-2007, 08:26 AM	#7
Jace FBOP Class Of 2013 Industry Role: Join Date: Jan 2004 Location: bumfuck, ky Posts: 35,562	vbulletin external.php?type=rss2

02-13-2007, 09:29 AM	#10
Klen Industry Role: Join Date: Aug 2006 Location: Little Vienna Posts: 32,235	Hmm it seems that command working on few board which i tryed,but it show all new posts/threads.I would more preffer if i can select specific categories,but that probaly depend on settings of rss feed.

02-13-2007, 09:34 AM	#11
Damian_Maxcash So Fucking Banned Join Date: Oct 2002 Location: MaxCash.com Posts: 12,745	Its ez to crawl the forums and get them into a database - phpDig for example - Its getting that info into a usable format that is hard.

02-13-2007, 11:17 AM	#17
LustyBucks Confirmed User Join Date: Oct 2006 Posts: 722	Take a look here - http://php.resourceindex.com. __________________ LustyBucks Affiliate Program - earn $40-$100 Per Join on Live Cams URL: http://www.lustybucks.com ICQ: 475-703-766, 205-825-691