Free Trials + Spidering. - GoFuckYourself.com

Darren · 03-22-2002, 08:41 AM

When offering a FREE trial how does/would the webmaster go about protecting is content from software such as spiders that could download everything?

What about people just coping text to word? I know this can be prevented via no right click but u can still get round it, is there another way??? more safe.

as anyone wrote any scripts to safe guard against people spidering the entire site to their hard drive? etc?

much advice is needed on this subject and I am sure that everyone will benefit !

so.....?

Darren · 03-22-2002, 08:47 AM

an example of the spidering program I am refering to would be:
www.webattack.com/get/superbot.shtml

Darren · 03-22-2002, 08:50 AM

and apparently this one is the best:
http://www.spidersoft.com/

spacedog · 03-22-2002, 08:55 AM

I'm not adept with scripts myself, but I THINK that you can use htaccess to deny these spiders access??

Or via your robots.txt file?? Maybe?

Darren · 03-22-2002, 08:58 AM

would that work?

anyone got a defintive solution for this problem?

Snake Doctor · 03-22-2002, 09:03 AM

Darren don't put a no right click in your members area, it'll piss people off.
You can use cgi to serve your images, that's one way.

Also you can put a user agent allow/deny in your htaccess to prevent all of the known web sucking programs.

You can also set a bandwidth threshhold per username/pass. Figure about 4 or 5 times what anyone could reasonably download in a day and then cut them off. This kicks out the password traders as well

(Granted you MIGHT piss off a member who's just in there right clicking his ass off, but if he's trying to get that much that quick he's probably not planning on sticking around anyways.

Of course, you should probably use a host that handles some of the bigger paysites and their tech crew will most probably set up a solution for you.

Darren · 03-22-2002, 09:50 AM

good ideas - are there any actual hosts which do offer solutions if so speak up! or spacedog spam me one!

any other thoughts

spacedog · 03-22-2002, 09:52 AM

Ask your current host to help you with setting up your htaccess..

I know there's articles & examples of how to use htaccess to block the user agents, I seen them, just dont remember the link..
I am sure http://www.google.com/search?hl=en&i...with+.htaccess

you could find something useful here..

spacedog · 03-22-2002, 09:54 AM

actually, the first listing on this page may be exactly what you need...

http://www.google.com/search?hl=en&i...with+.htaccess

playa · 03-22-2002, 10:12 AM

here is nice .htaccess file,,,,
http://chokinchicken.com/webmasters/htaccess.htm

i added it to my root server and found out some of my pics wouldn't load,, but added it to my domain directory and it worked fine

maybe something in the httpd.conf configuration would be better?

pr0 · 03-22-2002, 10:46 AM

Limit user to 2 simultaneous connections

Which would mean he can load index.htm & pic1.jpg at one time...but no more.

Most spidering software makes use of 100's if not 1000's of threads to spider the site.

You can effectively eliminate that with bandwidth throttling, or limiting the user connections/threads.

I have used such technology while using win2k server, not sure how to set that up on unix, linux, or apache, but im sure there is a way to do it.

You can also set a bandwidth cap...IE total transfer limit per-ip address, or per username/password.

The per-ip can be bypassed by a person using a proxy-spider...however if you set a total download limit for a user/pass that would prob be your best bet

hope that helpz

Lensman · 03-22-2002, 10:54 AM

OK, here's how to block spiders like Teleport Pro, which is the #1 spider software.

Make your navigation pages a form submit and use the images as submit buttons, instead of the usual href. The spiders cannot get past them! The action will simply be the page you want to go to. And you can do mouseovers so it looks just like a normal link.

This works great for member's areas.

pr0 · 03-22-2002, 10:59 AM

Quote:

Originally posted by Lensman
OK, here's how to block spiders like Teleport Pro, which is the #1 spider software.

Make your navigation pages a form submit and use the images as submit buttons, instead of the usual href. The spiders cannot get past them! The action will simply be the page you want to go to. And you can do mouseovers so it looks just like a normal link.

This works great for member's areas.

quit makin me look bad lensman ; )

darius · 03-22-2002, 11:03 AM

Lensman

That sounds great. Can you give us a site url that is doing that? I would love to see it.

Darius

ReFresh · 08-14-2002, 09:39 PM

I don't know if it did when this was first posted, but when I looked, it was pointing to superscripts.com, psybercore.com or whatever.. (They and drewstar.com and no telling how many other sites are all the same guys sites).

If you plan on laying out any cash for these scripts this guys sells, you might want to do a search on "psybercore" "superscripts" etc. first. It would have same some people some money if they'd have done that before they sent their cash to this company. See:

http://www.bitchdomain.com/superscriptmeoff.htm

Quote:

Originally posted by spacedog
actually, the first listing on this page may be exactly what you need...

http://www.google.com/search?hl=en&i...with+.htaccess

Wizzo · 08-14-2002, 09:50 PM

Quote:

Originally posted by darius
Lensman

That sounds great. Can you give us a site url that is doing that? I would love to see it.

Darius

You might try amateurpages...

03-22-2002, 10:46 AM	#11
pr0 rockin tha trailerpark Industry Role: Join Date: May 2001 Location: ~Coastal~ Posts: 23,088	Limit user to 2 simultaneous connections Which would mean he can load index.htm & pic1.jpg at one time...but no more. Most spidering software makes use of 100's if not 1000's of threads to spider the site. You can effectively eliminate that with bandwidth throttling, or limiting the user connections/threads. I have used such technology while using win2k server, not sure how to set that up on unix, linux, or apache, but im sure there is a way to do it. You can also set a bandwidth cap...IE total transfer limit per-ip address, or per username/password. The per-ip can be bypassed by a person using a proxy-spider...however if you set a total download limit for a user/pass that would prob be your best bet hope that helpz __________________ __________ Loadedca$h - get sum! - Revengebucks - mmm rebills! - webair (gotz sErVrz)

03-22-2002, 08:41 AM	#1
Darren Confirmed User Join Date: Sep 2001 Posts: 5,994	Free Trials + Spidering. When offering a FREE trial how does/would the webmaster go about protecting is content from software such as spiders that could download everything? What about people just coping text to word? I know this can be prevented via no right click but u can still get round it, is there another way??? more safe. as anyone wrote any scripts to safe guard against people spidering the entire site to their hard drive? etc? much advice is needed on this subject and I am sure that everyone will benefit ! so.....? __________________ Penis enlargement / male enhancement pills - earn up to $229.48 per signup ICQ: 46335817

03-22-2002, 08:47 AM	#2
Darren Confirmed User Join Date: Sep 2001 Posts: 5,994	an example of the spidering program I am refering to would be: www.webattack.com/get/superbot.shtml

03-22-2002, 08:50 AM	#3
Darren Confirmed User Join Date: Sep 2001 Posts: 5,994	and apparently this one is the best: http://www.spidersoft.com/

03-22-2002, 08:55 AM	#4
spacedog Yes that IS me. Bitch. Industry Role: Join Date: Nov 2001 Posts: 14,149	I'm not adept with scripts myself, but I THINK that you can use htaccess to deny these spiders access?? Or via your robots.txt file?? Maybe?

03-22-2002, 08:58 AM	#5
Darren Confirmed User Join Date: Sep 2001 Posts: 5,994	would that work? anyone got a defintive solution for this problem?

03-22-2002, 09:03 AM	#6
Snake Doctor I'm Lenny2 Bitch Join Date: Mar 2001 Location: On top of my soapbox Posts: 13,449	Darren don't put a no right click in your members area, it'll piss people off. You can use cgi to serve your images, that's one way. Also you can put a user agent allow/deny in your htaccess to prevent all of the known web sucking programs. You can also set a bandwidth threshhold per username/pass. Figure about 4 or 5 times what anyone could reasonably download in a day and then cut them off. This kicks out the password traders as well (Granted you MIGHT piss off a member who's just in there right clicking his ass off, but if he's trying to get that much that quick he's probably not planning on sticking around anyways. Of course, you should probably use a host that handles some of the bigger paysites and their tech crew will most probably set up a solution for you.

03-22-2002, 09:50 AM	#7
Darren Confirmed User Join Date: Sep 2001 Posts: 5,994	good ideas - are there any actual hosts which do offer solutions if so speak up! or spacedog spam me one! any other thoughts

03-22-2002, 09:52 AM	#8
spacedog Yes that IS me. Bitch. Industry Role: Join Date: Nov 2001 Posts: 14,149	Ask your current host to help you with setting up your htaccess.. I know there's articles & examples of how to use htaccess to block the user agents, I seen them, just dont remember the link.. I am sure http://www.google.com/search?hl=en&i...with+.htaccess you could find something useful here..

03-22-2002, 09:54 AM	#9
spacedog Yes that IS me. Bitch. Industry Role: Join Date: Nov 2001 Posts: 14,149	actually, the first listing on this page may be exactly what you need... http://www.google.com/search?hl=en&i...with+.htaccess

03-22-2002, 10:12 AM	#10
playa Confirmed User Join Date: Feb 2001 Location: atlanta, GA Posts: 6,432	here is nice .htaccess file,,,, http://chokinchicken.com/webmasters/htaccess.htm i added it to my root server and found out some of my pics wouldn't load,, but added it to my domain directory and it worked fine maybe something in the httpd.conf configuration would be better?

03-22-2002, 10:54 AM	#12
Lensman GFY Chaperone Join Date: Jan 2001 Location: Adult.com Posts: 9,846	OK, here's how to block spiders like Teleport Pro, which is the #1 spider software. Make your navigation pages a form submit and use the images as submit buttons, instead of the usual href. The spiders cannot get past them! The action will simply be the page you want to go to. And you can do mouseovers so it looks just like a normal link. This works great for member's areas.

03-22-2002, 11:03 AM	#14
darius Registered User Join Date: Sep 2001 Posts: 51	Lensman That sounds great. Can you give us a site url that is doing that? I would love to see it. Darius