.htaccess IP banning script - bans robots, spiders, etc - GoFuckYourself.com

mkx · 10-13-2010, 11:22 AM

I am looking to protect my websites data and images against spiders, bots, data extraction etc. I figure this can be done by limiting the number of requests and/or data transfer an IP can make in a certain amount of time. I do not however want to block google or other major search engines from spidering my site which I guess makes things more complicated. Does anyone know of any such script or method?

Cyber Fucker · 10-13-2010, 11:33 AM

http://www.google.com/#hl=en&q=bad+b...ccess+ban+list

mkx · 10-13-2010, 11:53 AM

thanks88

urbanpimp · 10-13-2010, 11:54 AM

Quote:

Originally Posted by Cyber Fucker

http://www.google.com/#hl=en&q=bad+b...ccess+ban+list

thanks

RycEric · 10-13-2010, 11:55 AM

Quote:

Originally Posted by mkx

I am looking to protect my websites data and images against spiders, bots, data extraction etc. I figure this can be done by limiting the number of requests and/or data transfer an IP can make in a certain amount of time. I do not however want to block google or other major search engines from spidering my site which I guess makes things more complicated. Does anyone know of any such script or method?

This can be superceded by using bots with proxy databases. Just an FYI.

Cyber Fucker · 10-13-2010, 01:37 PM

Well, I can't post links to other forums here... but if you browse the link I gave you there will be one on the 1st search results page that has plenty of htaccess configs including lists of bad bots, website scrapers etc. You're welcome.

mkx · 10-13-2010, 01:38 PM

i know httrack can make it look like the browser is internet explorer or firefox, would this still catch it>?

10-13-2010, 11:22 AM	#1
mkx Confirmed User Industry Role: Join Date: Nov 2003 Location: Toronto Posts: 4,001	.htaccess IP banning script - bans robots, spiders, etc I am looking to protect my websites data and images against spiders, bots, data extraction etc. I figure this can be done by limiting the number of requests and/or data transfer an IP can make in a certain amount of time. I do not however want to block google or other major search engines from spidering my site which I guess makes things more complicated. Does anyone know of any such script or method?

10-13-2010, 11:33 AM	#2
Cyber Fucker Hmm Industry Role: Join Date: Sep 2005 Location: On an endless road around the world for rock and roll. Posts: 12,642	http://www.google.com/#hl=en&q=bad+b...ccess+ban+list __________________

10-13-2010, 01:37 PM	#6
Cyber Fucker Hmm Industry Role: Join Date: Sep 2005 Location: On an endless road around the world for rock and roll. Posts: 12,642	Well, I can't post links to other forums here... but if you browse the link I gave you there will be one on the 1st search results page that has plenty of htaccess configs including lists of bad bots, website scrapers etc. You're welcome. __________________

10-13-2010, 11:53 AM	#3
mkx Confirmed User Industry Role: Join Date: Nov 2003 Location: Toronto Posts: 4,001	thanks88

10-13-2010, 01:38 PM	#7
mkx Confirmed User Industry Role: Join Date: Nov 2003 Location: Toronto Posts: 4,001	i know httrack can make it look like the browser is internet explorer or firefox, would this still catch it>?