How to make script which crawl posts from blogs/forums?

[Klen]

Since there is now millions of forums and blogs,i think it will be good to make script which will crawl posts from there,something like rss feed.With blogs that should'nt be problem since most blogs include rss feed,but forums not.Any idea how to do it?

[Adultnet]

this is old and was made..

[fris]

blogs you can do, but why would you want forum posts?

[tolik]

use libnet if you know perl

[Kimo]

good luck....

[Klen]

blogs you can do, but why would you want forum posts?

Well there are several big forums which have buy/sell sections,so that would be easier for tracking those sections.

[Jace]

vbulletin

external.php?type=rss2

[Klen]

vbulletin

external.php?type=rss2

That is default function or mod?

[StuartD]

board tracker.com

[Klen]

Hmm it seems that command working on few board which i tryed,but it show all new posts/threads.I would more preffer if i can select specific categories,but that probaly depend on settings of rss feed.

[Damian_Maxcash]

Its ez to crawl the forums and get them into a database - phpDig for example - Its getting that info into a usable format that is hard.

[Jace]

That is default function or mod?

look man, I know asking question is great and all to learn, but try and figure some stuff out for yourself, it sinks in longer that way

[rowan]

Since there is now millions of forums and blogs,i think it will be good to make script which will crawl posts from there,something like rss feed.With blogs that should'nt be problem since most blogs include rss feed,but forums not.Any idea how to do it?

Is this so you can find newer posts instead of bumping 3 year old threads?

[Klen]

look man, I know asking question is great and all to learn, but try and figure some stuff out for yourself, it sinks in longer that way

Ok hehe.Not a problem.

[Klen]

Is this so you can find newer posts instead of bumping 3 year old threads?

I just want to see all news at one place.It is much faster then checking site one by one.Also i can add feature which will if specific word apper to alert me over mail.

[darksoul]

vbulletin

external.php?type=rss2

pretty neat
thanks

[LustyBucks]

Take a look here - http://php.resourceindex.com.