How can you stop THIEVES SCRAPING your SITES? - GoFuckYourself.com - Adult Webmaster Forum

X

CunningStunt

Confirmed User

Join Date: Aug 2006

Posts: 5594
Share

Post
#1

How can you stop THIEVES SCRAPING your SITES?

08-28-2009, 12:26 AM

I have a content rich site that this is happening to more and more.

You can ban their IP addresses, but they soon pop up with another.

Is their any way to lock down the server better, yet still allow the friendly SE bots? It's becoming an increasing problem.
Tags: None
cLin

Registered User

Join Date: May 2009

Posts: 94
Share

Post
#2

08-28-2009, 12:34 AM

We use strongbox which says it has an anti scraping technology but to be honest, I've never dealt with it. The logic they use makes sense though.

http://bettercgi.com/strongbox/features.html#antislurp

Chris
The Ex Girlfriend Pics
Comment
CunningStunt

Confirmed User

Join Date: Aug 2006

Posts: 5594
Share

Post
#3

08-28-2009, 01:59 AM

Thanks Chris.

I forgot to mention the site is written in asp.net, don't know how different methods work exactly with different server setups.

Knowing nothing much about server security, I would just have a list of all known "friendly" search engine bots, and everything else gets fooked out if there are multiple session attempts. Is that how it works?

I should contact my dedicated host, I'm sure they'll be able to recommend something, that's their speciality after all.
Comment
Klen

Join Date: Aug 2006

Posts: 32234
Share

Post
#4

08-28-2009, 02:59 AM

Well,one of methods which i use it is to delete old content and replaced it with completely new content on different location.
Comment
CunningStunt

Confirmed User

Join Date: Aug 2006

Posts: 5594
Share

Post
#5

08-28-2009, 03:07 AM

Originally posted by KlenTelaris

Well,one of methods which i use it is to delete old content and replaced it with completely new content on different location.

Almost all my content is ranking and linked to. How can doing 301's etc help? I don't want to be shifting stuff around all the time.

I'd like to simply shut off programmatic attempts to access my sites from unknown IP addresses, period.
Comment
Dirty Dane

Sick Fuck

Join Date: Feb 2004

Posts: 9491
Share

Post
#6

08-28-2009, 03:28 AM

Is it a paysite or free? And when you say "thieves".. are they duplicating your website or just download content?

Last edited by Dirty Dane; 08-28-2009, 03:32 AM.
Comment
CunningStunt

Confirmed User

Join Date: Aug 2006

Posts: 5594
Share

Post
#7

08-28-2009, 03:41 AM

Originally posted by Dirty Dane

Is it a paysite or free? And when you say "thieves".. are they duplicating your website or just download content?

Free site.

First instance they had an autoscraper on a daily scrape to pull the new entries and posting to a cough **new** site, which I soon had closed down via dmca. Now I've caught it at an earlier stage.
Comment
SeanLEE

Confirmed User

Join Date: Feb 2006

Posts: 1556
Share

Post
#8

08-28-2009, 03:46 AM

Just put advertisements in your player- then they are advertising for you.

I spammed in threads!
Comment
quantum-x

Confirmed User

Join Date: Feb 2002

Posts: 6863
Share

Post
#9

08-28-2009, 03:49 AM

Honestly, there's not much you can do.
If you can see it, you can steal it.... cookies, restrictions, captchyas etc can all be defeated.

The best suggestions are: watermark, adverts, etc.

One trick that really does stump most spiders, however, is to link to your content via JS or CSS.

PrettyInCash.com - BoozedGFs.com - TeenGFs.com - JizzGFs.com- MilfUploads.com -
Comment
Dirty Dane

Sick Fuck

Join Date: Feb 2004

Posts: 9491
Share

Post
#10

08-28-2009, 04:27 AM

Originally posted by CunningStunt

Free site.

First instance they had an autoscraper on a daily scrape to pull the new entries and posting to a cough **new** site, which I soon had closed down via dmca. Now I've caught it at an earlier stage.

Ok....

Banning IPs won't help much, because they might just use rotating IPs.

Banning known offline user-agents could help, but that is also easy to override (they are sending fake user-agent info).

If you have movies, put it in javascript (the agent usually can't read that) - and text outside (for SE). The downside of this is surfers who disabled javascript.

If you have a decent CPU on your server, then trick their "browser" into fake links with long delays (like a cgi link) or fake targets that temporarily kill and ban too many attempts. This will lag or cut off their agent temporarily. (might be a very good idea to use robots.txt on those links because you do not want to trick google..)

You can also create something that their agent doesn't understand to sort out. The more garbage, the better. For instance, if you have invisible links to the "tubegirl" , they will end up with all kinds of shit, they have to sort out manually.
You can also structure your site in a way that has no logic and is hard to restructure for a software.

You can also watermark your stuff with "licensed to...", but talk with your sponsors before doing that. And if they promote same sponsor, then you should talk about that too

Last edited by Dirty Dane; 08-28-2009, 04:31 AM.
Comment
faxxaff

Confirmed User

Join Date: Dec 2002

Posts: 2134
Share

Post
#11

08-28-2009, 04:30 AM

You can use trap files ... like 1x1 pix big files named 6tjgTTvtfgh.jpg or something like that. If it get's downloaded, you know it's an illegal bot. Now, write a script that will block that user based on IP and session ID or have him download some malicious bullshit ...

Asian Babes
Comment
rowan

Too lazy to set a custom title

Join Date: Mar 2002

Posts: 17393
Share

Post
#12

08-28-2009, 05:11 AM

I'm facing this problem myself as I have a site with millions of pages. Currently if an IP downloads too many pages within a short period of time and it's NOT on the whitelist (eg the IPs of Google) it gets firewalled for a period of time. It's a pretty aggressive approach but it works (for now)

Most of the people trying to scrape don't bother trying to hide it, so their 3 fetches a second gets picked up pretty quickly.
Comment
Dirty Dane

Sick Fuck

Join Date: Feb 2004

Posts: 9491
Share

Post
#13

08-28-2009, 05:52 AM

Please post the domains of these bastards. Without the http
Comment
Agent 488

Registered User

Join Date: Feb 2006

Posts: 22511
Share

Post
#14

08-28-2009, 06:05 AM

i was going to say dmca but looks like you did it.

please no idiots come in here saying that is your site is online you are consenting for it to be scraped. fuck off.
Comment
SmokeyTheBear

►SouthOfHeaven

Join Date: Jun 2004

Posts: 28609
Share

Post
#15

08-28-2009, 08:20 AM

Originally posted by budsbabes

please no idiots come in here saying that is your site is online you are consenting for it to be scraped. fuck off.

lol well i hate to do this but....

Have you given consent for google to scrape you ? yahoo ? bing ? etc.

Basically you have an apple tree in a public place. You aren't stopping anyone from picking your apples, infact you like some people picking your apples ( google ) even though they never asked to pick your apples and furthermore they are showing your apples on their site and making money from it. shitloads of money.

Can't be too suprised when some fatty comes by and picks all your apples one day, after watching everyone else pick them and you not stopping them

hatisblack at yahoo.com
Comment
DonovanTrent

Confirmed User

Join Date: Dec 2006

Posts: 968
Share

Post
#16

08-28-2009, 08:28 AM

Originally posted by CunningStunt

I have a content rich site that this is happening to more and more.

I'm just kind of wondering why you have a "content rich site" that is free. But that's just me, I may be missing something here.

Donovan Trent
Comment
Agent 488

Registered User

Join Date: Feb 2006

Posts: 22511
Share

Post
#17

08-28-2009, 08:32 AM

search engines link to my content. in order for someone to read it they have to click through to my site. there is a difference.

Originally posted by SmokeyTheBear

lol well i hate to do this but....

Have you given consent for google to scrape you ? yahoo ? bing ? etc.

Basically you have an apple tree in a public place. You aren't stopping anyone from picking your apples, infact you like some people picking your apples ( google ) even though they never asked to pick your apples and furthermore they are showing your apples on their site and making money from it. shitloads of money.

Can't be too suprised when some fatty comes by and picks all your apples one day, after watching everyone else pick them and you not stopping them
Comment
seeandsee

Check SIG!

Join Date: Mar 2006

Posts: 50945
Share

Post
#18

08-28-2009, 08:32 AM

Originally posted by Dirty Dane

Ok....

Banning IPs won't help much, because they might just use rotating IPs.

Banning known offline user-agents could help, but that is also easy to override (they are sending fake user-agent info).

If you have movies, put it in javascript (the agent usually can't read that) - and text outside (for SE). The downside of this is surfers who disabled javascript.

If you have a decent CPU on your server, then trick their "browser" into fake links with long delays (like a cgi link) or fake targets that temporarily kill and ban too many attempts. This will lag or cut off their agent temporarily. (might be a very good idea to use robots.txt on those links because you do not want to trick google..)

You can also create something that their agent doesn't understand to sort out. The more garbage, the better. For instance, if you have invisible links to the "tubegirl" , they will end up with all kinds of shit, they have to sort out manually.
You can also structure your site in a way that has no logic and is hard to restructure for a software.

You can also watermark your stuff with "licensed to...", but talk with your sponsors before doing that. And if they promote same sponsor, then you should talk about that too

nice tips

BUY MY SIG - 50$/Year

Contact here
Comment
Agent 488

Registered User

Join Date: Feb 2006

Posts: 22511
Share

Post
#19

08-28-2009, 08:33 AM

what is so hard to understand. you can't think of one content rich free site?

Originally posted by DonovanTrent

I'm just kind of wondering why you have a "content rich site" that is free. But that's just me, I may be missing something here.
Comment
Agent 488

Registered User

Join Date: Feb 2006

Posts: 22511
Share

Post
#20

08-28-2009, 08:35 AM

i understand where you are coming from, but there is a difference between a content preview and a full scrape.

Originally posted by SmokeyTheBear

lol well i hate to do this but....

Have you given consent for google to scrape you ? yahoo ? bing ? etc.

Basically you have an apple tree in a public place. You aren't stopping anyone from picking your apples, infact you like some people picking your apples ( google ) even though they never asked to pick your apples and furthermore they are showing your apples on their site and making money from it. shitloads of money.

Can't be too suprised when some fatty comes by and picks all your apples one day, after watching everyone else pick them and you not stopping them
Comment
SmokeyTheBear

►SouthOfHeaven

Join Date: Jun 2004

Posts: 28609
Share

Post
#21

08-28-2009, 08:44 AM

Originally posted by budsbabes

i understand where you are coming from, but there is a difference between a content preview and a full scrape.

google would like you to think that anyways did you know google offers a service that allows users to browse your site without most of your ads? they scrape the entire page on the fly and only display the text.

even without that , all google is doing is cutting your page up and displaying it as seperate items, the only difference is they get to show way more ads in the process than someone who just scrapes the page and repukes it up.

hatisblack at yahoo.com
Comment
DonovanTrent

Confirmed User

Join Date: Dec 2006

Posts: 968
Share

Post
#22

08-28-2009, 08:58 AM

Originally posted by budsbabes

what is so hard to understand. you can't think of one content rich free site?

Depends on the content. I've seen plenty of content-rich sites that should be nowhere near free.

Donovan Trent
Comment
raymor

Confirmed User

Join Date: Oct 2002

Posts: 3745
Share

Post
#23

08-28-2009, 09:11 AM

Originally posted by cLin

We use strongbox which says it has an anti scraping technology but to be honest, I've never dealt with it. The logic they use makes sense though.

http://bettercgi.com/strongbox/features.html#antislurp

We have Throttlebox, specifically designed for this type of thing.
Throttlebox is an Apache module. The OP says he used ASP.net.
I wonder if that means he's hosting on a Windows desktop instead
of a server OS running Apache.

For historical display only. This information is not current:
support@bettercgi.com ICQ 7208627
Strongbox - The next generation in site security
Throttlebox - The next generation in bandwidth control
Clonebox - Backup and disaster recovery on steroids
Comment
CunningStunt

Confirmed User

Join Date: Aug 2006

Posts: 5594
Share

Post
#24

08-28-2009, 01:47 PM

Originally posted by DonovanTrent

I'm just kind of wondering why you have a "content rich site" that is free. But that's just me, I may be missing something here.

Yes you are

There's some good advice in here, thanks for taking the time folks.
Comment
raymor

Confirmed User

Join Date: Oct 2002

Posts: 3745
Share

Post
#25

08-29-2009, 03:30 PM

Originally posted by faxxaff

You can use trap files ... like 1x1 pix big files named 6tjgTTvtfgh.jpg or something like that. If it get's downloaded, you know it's an illegal bot. Now, write a script that will block that user based on IP and session ID or have him download some malicious bullshit ...

That's one of several techniques we use. That technique works sometimes, but
often does not. We don't send people malicious files of course, we're not criminals,
but we do use traps. It's a useful part of a multi-layered approach, but not at all
sufficient on it's own.

For historical display only. This information is not current:
support@bettercgi.com ICQ 7208627
Strongbox - The next generation in site security
Throttlebox - The next generation in bandwidth control
Clonebox - Backup and disaster recovery on steroids
Comment
raymor

Confirmed User

Join Date: Oct 2002

Posts: 3745
Share

Post
#26

08-29-2009, 03:38 PM

Originally posted by SmokeyTheBear

lol well i hate to do this but....

Have you given consent for google to scrape you ? yahoo ? bing ? etc.

Yes, he's given Google permission to index, not scrape, the site, and thereby promote it.
If you've been a webmaster for more than a few days, you know about robots.txt.
By choosing not to put up a "no indexing" sign (robots.txt), you've given implied
permission for Google to promote you by adding you to their index. I'd bet the people
scraping (not indexing) the site don't check for robots.txt.

Besides, use a ounce or so of common sense. Obviously webmasters want their
porno sites listed in search engines. Duh.

For historical display only. This information is not current:
support@bettercgi.com ICQ 7208627
Strongbox - The next generation in site security
Throttlebox - The next generation in bandwidth control
Clonebox - Backup and disaster recovery on steroids
Comment
Angry Jew Cat - Banned for Life

(felis madjewicus)

Join Date: Jul 2006

Posts: 20368
Share

Post
#27

08-29-2009, 03:56 PM

the cat thinks most of the people in this thread have a skewed definition of content.
Comment
DonovanTrent

Confirmed User

Join Date: Dec 2006

Posts: 968
Share

Post
#28

08-29-2009, 04:27 PM

Originally posted by CunningStunt

Yes you are

I guess I was just asking, based on the content being so valuable to you as to be concerned about protecting it. That's all.

Donovan Trent
Comment
GrouchyAdmin

Now choke yourself!

Join Date: Apr 2006

Posts: 12085
Share

Post
#29

08-29-2009, 05:25 PM

One of the ways I've dealt with it is with custom webserver-level applications - never post a direct link to the content. Use a custom hash, decode, and sendfile() the bitch. Otherwise, I've used trivial timestamping and other simple methods to break fuskers. Don't forget to disable supporting HTTP Trace.

AIM: GrouchyGfy
Comment
SmokeyTheBear

►SouthOfHeaven

Join Date: Jun 2004

Posts: 28609
Share

Post
#30

08-30-2009, 12:50 AM

Originally posted by raymor

Yes, he's given Google permission to index, not scrape, the site, and thereby promote it.

he did ? i didnt see the part where he defined his submittion to google, i thought google just scraped his site like they scrape every site..

Originally posted by raymor

If you've been a webmaster for more than a few days, you know about robots.txt.

not a few , just a couple days

Originally posted by raymor

By choosing not to put up a "no indexing" sign (robots.txt), you've given implied
permission for Google to promote you by adding you to their index.

lol so by not posting a sign saying " do not break my car windows" you are implying that it's okay to smash your car windows ? ok got it..

by that theory everyone has permission , why would it be implied for google but not implied for others ? is it called the googlerobots.txt ?

Originally posted by raymor

Besides, use a ounce or so of common sense. Obviously webmasters want their
porno sites listed in search engines. Duh.

now they do .. before google it was just software downloading everything on your server

maybe thats what he wants to do is become so rich and well known you will beg him to come scrape your site just like google.

hatisblack at yahoo.com
Comment

Previous template Next

Working...