Quote:
Originally Posted by Fris
pussy.org traffic rocks, sucks that you have to stop spidering cause of trojan fuckers, wonder who else will do this, i know catlist and all his sites spider.
|
I don't see problem with spidering though, a year ago when I was involved in adult search engine project we had our spider build in a way to check for "suspicious activity" of indexed pages. It would check if page is up, if it has any popups, redirects, jacavscripts, activex and other embedded objects. The challenge was once indexed pages database builds to a large amount, to effectively revisit indexed sites so we had to come up with some dynamic scheduling techniques but that's a matter of different topic.
What I?m saying is if you have a spider collecting urls technically it?s possible to have it revisit and recheck pages automatically or have a different spider doing specifically just that. Even if you collect hand submissions it?s better to have something on the background periodically checking listed urls.