EBORG9

I have been getting a crap load of bandwidth sucking hits from these ip addresses .
The problem is, my stats don't reflect this in unique visitors, just page views and hits.
Anybody know what is going on?
Here are the addresses:
216.109.121.44 11594 11594 400.13 MB 19 May 2007 - 23:13
216.109.121.41 11560 11561 405.01 MB 19 May 2007 - 23:12
216.109.121.42 11374 11374 398.89 MB 19 May 2007 - 23:13

These are all Yahoo!

I am perplexed to say the least.

<edited>
Now when I look at my stats,
awstats shows 150 uniques, and over 2000 page views
active meter (for shits and giggles) shows 370 visitors today, which is double from yesterday.

also here is the big one, the referring host with the largest number and most band width...IS MINE!
Over 12,000 pages and 32,000 hits. WTF is that and why is my own IP address throwing so much crap at my site?

Any help is appreciated.
Thanks

EBORG9

There are a few other things that seem strange.
I was listed, with this site, in DMOZ in less than 30 days, and the amount of indexed pages seems too high:

Search Engine Pages indexed

Google 3,660
MSN Search 21
Yahoo! 135
AlltheWeb 436
AltaVista 133
Total 4385

this blog, on this IP address is 1 month old.
Google bot has crawled 19,000 times this month.

WTF is going on, it's all too fast.

Is there something that i am not getting or did I overlook something and am now getting false readings?

EBORG9

Bump for some opinions. I know that these readings are abnormal for a new site? And the bandwidth is not matching the visits. I have tracked down everything that I can think of, all the way down to deactivating the page translator.

TheDoc

I think the 216.109.121.* range is yahoo's bot's.. they have many ip's

Some translator tools have loops, where they will go 2-3+ stage deep creating duplicate content and bot loops. This could count for the amount of visits and listings.

Awstats tracks raw page views and every image download / connection made as a hit/bw connection, ect.

__________________

EBORG9

So all this bandwidth is Yahoo continuously crawling and duplicating content over several IP's?
It's been going on now for a month.

TheDoc

Bots can crash your machines if you get them stuck in loops so making awstats look like it's burning a lot of bw is probably easy.

I'm not sure about duplicate content over several ip's - but you can duplicate trash translations on the same domain 100's of times and google/yahoo will give you an amazing amount of trash translated listings.

__________________

EBORG9

Well, I did have a universal page translator on it, that did the 13 languages, but I put that on a month ago.
I would think that , that would multiply the number of pages by....well 13, but it should all be crawled by now.
Does the translator hurt, or help me?

TheDoc

I really don't know if a translator helps or not in terms of seo. It does appear to produce another level of links & listings.. It does produce traffic from the search engines. For myself I couldn't use this traffic, so I cut it out. Someone that has trade scripts installed may benefit from the traffic.

With the amount of languages, I do feel that could have a negative impact. Translator software is hardly readable in most languages. Also, many designs break and fall apart due to the translations too.

Then, without a translator lets say you have 50 pages, google/yahoo may only find 40, then based off site structure and internal linking they can figure out the 5 pages are the most popular, sup results, ect.. So by using a translator you could confuse this, you could create a larger than needed linking structure, a cross linking of anchor text/title text that it can't understand and single out the keywords from.

Anyway, If you use a translator tool for real proper use it's better to open the page with the translation only and drop the menus and cross linking. Pretty much the same rules for print this pages.

__________________

spunkmaster

I have the same issue and it's proxy sites using your site/IP as a proxiy site.

The problem is you can't block by IP because there are thousands using the proxy !

__________________

TheDoc

That would some like some type of service is on that shouldn't be.. I'm not an admin, but it doesn't make since to me that someone other than me can use the IP, unless I'm allowing it.

__________________

Damian_Maxcash

I had a similar problem - I fixed it with robots.txt

I cant remember the details but might be worth a search.

TheDoc

Blah... lets redo that..

Visits from clients at proxyn.search.dcn.yahoo.net or proxyn.search.acd.yahoo.net are not crawler activity but proxies for browser page views using a 'translate this page' link from Yahoo! search results.

__________________

TheDoc

Which has something to do with the china slup... Hell hope it helps at least...


User-agent: Slurp China
Disallow: /

User-agent: Slurp
Crawl-delay: 3

__________________

EBORG9

OK OK I got all of that.
I have no problem with page break up , using the translator, and it actually holds together very well even with Chinese and Japanese which use big characters.

I am scared to change anything. Getting into DMOZ in 2 weeks, was a strong motivator to keep on pushing on, but I can't have these bandwidth problems.

"Visits from clients at proxyn.search.dcn.yahoo.net or proxyn.search.acd.yahoo.net are not crawler activity but proxies for browser page views using a 'translate this page' link from Yahoo! search results."

So does this mean the translator is actually helping the proxies? Could this be from link directories? Fuck I didn't even think of that!
I seriously like the translator, I see a good percentage of traffic coming in from translated pages, but i will have to rethink that now.

The Robots.txt...I will take care of that now, thanks.

Thanks everyone. If anyone else has any tips, as Ross Perot said, "I'm all ears."

EBORG9

One last question if anyones still out there.
The largest amount of bandwidth is coming from MY IP
12682 32520 332.13 MB

Does that mean that someone is piggy backing my system and using my site as a proxy?

Damian_Maxcash

I don't think so - I cant see why someone would do that to just your site.

I am always amazed at how much I burn when I am just checking my own sites - and as someone said AWStats counts every file that's pulled as a hit. If you have a page with 100 thumbs for example that's 101 hits at least every time you see it.

EBORG9

Oh crap, I have like 370 pages of content, and the first page has a lot of images on it, which reloads everytime I post or update, and I have been updating a lot.
Ok, that makes sense, thanks.