Warning with this technique (Google)

[roganoli]

I'm still an inexperienced webmaster...

I was having good placements on Google for my new site... He was even making good sales!

Suddenly something happened. My website disappeared from search! Today I found out what caused it... My site was totally copied by the proxy service Google! See the link below:

https://www.google.com/#q=%22appspot...w=1280&bih=663

I believe this was done intentionally by another webmaster and I believe I have solved this problem using my htaccess!

But be warned for other amateurs like me!

[Klen]

Had same happening to me,totally copied site to three different domains.Which is why i have now in my settings protection against all kind of downloaders.

[roganoli]

I put it on my htaccess:

RewriteCond %{HTTP_USER_AGENT} AppEngine [NC]
RewriteRule .* - [F]

Let me know if there is better solution...

[baryl]

Wait..what? Can someone explain what this is?

[Klen]

Search for article like this:
http://www.htaccess-guide.com/blocki...-and-bad-bots/
Had one with huge list of those downloaders,but dont know what was exact url.

[roganoli]

Wait..what? Can someone explain what this is?

Sorry if my English is not helpful...

[baryl]

It's fine. Thanks for pointing that out and sorry about your site.

[RayBonga]

I put it on my htaccess:

RewriteCond %{HTTP_USER_AGENT} AppEngine [NC]
RewriteRule .* - [F]

Let me know if there is better solution...

Thanks for the warning What does this code do?

[brassmonkey]

damn sorry to hear that

[redwhiteandblue]

Thanks for the warning What does this code do?

It blocks access to any client which presents a user agent with the string "AppEngine" in it. But there are a whole load of others you need to block too, and that won't even stop all scrapers. I'd love to know the UA updowner.com is using and block that fucker.

[medicatedghost]

check out google.com/safebrowsing/diagnostic?site=updowner.com/
interesting analysis

[jakeuk]

The mod rewrite you suggested offer some protection. When I had similar problem a few years ago, here are other things I did in addition to the mod_rewrite:

1. Get strong links to your site so that your site is at least stronger than the copycat.
2. Rewrite some of the contents of your site so that its not a duplicate any more but a different version of the one that was copied.
3. Update you site regularly so that the copycat cannot keep up.
4. use a more aggressive exclusion robots.txt file, allowing ONLY the big 3 crawlers.

[rowan]

I'd love to know the UA updowner.com is using and block that fucker.

Any half decent scraper probably uses a common UA, and possibly a random selection of common UAs. I've seen loads of multiple pages from the same IP use a different UA each time.