GoFuckYourself.com - Adult Webmaster Forum - I have several different crawlers

- Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)

- - I have several different crawlers (https://gfy.com/showthread.php?t=996520)

theking

11-08-2010 04:36 AM

I have several different crawlers

...crawling my sites every day. Is this typical for everyone?

redwhiteandblue

11-08-2010 05:09 AM

Block the little fuckers in your htaccess, most of them are up to no good or at least a waste of bandwidth.

CaptainHowdy

11-08-2010 05:14 AM

Emil	11-08-2010 05:18 AM

Yeah, "Join Date: Sep 2002". You started your first site 2010?

All times are GMT -7. The time now is 08:39 PM.