Quote:
|
Originally Posted by dcortez
Do you respect ROBOTS.TXT directives to ban your spider from sites not interested in being 'exposed' this way?
|
nope.
and trying to block our web crawlers isn't quite the answer for many reasons:
1) there are many, many other web crawlers out there so you'd be spending alot of time tracking them all down and we spider from many different IPs
2) our spiders are throttled to download at 56K modem speeds so that we don't drain a website any faster than a normal dial-up.. and we only retrieve html, so the amount retrieved is relatively small and we don't hit your website continually, so we do play nice.
3) the t3report is based on linking relationships, so websites that link to your domain would still be accessed and have links that connect to you. being able to block our spiders could mean that the external links that you have that connect to the target domain of the t3report could be blocked, and thus, your domain never shows up in the report.. but other web crawlers that do access your site could be harvested to build up the missing links that end up connecting you into the report.
4) anyone can go to alexa or google and type in link:domain.com where domain.com is your domain and be able to pull up links to your domain, so you can't really hide.. if you have a website that people will visit, web crawlers will also find you, robots.txt or not.
5) by having your website in the report, it could actually gain business for you if people see you have good traffic leading to you.. why chase down each and every linker to you? easier to just tap into your traffic by buying ad space on your site, link exchange, or entice to be an affiliate manger.
6) if you are getting bad traffic and passing it on, then having that revealed could be bad for you, but bad traffic is bad traffic, there is no defense for that.. and again to point #4, anyone could look you up via alexa or google, the major difference is we have alot more data that we present then either one of them who cap the results at 1,000 and are not focused on the adult space.
Trying to block out our spiders goes again to my point of chopping off your nose to spite your face.
Fight the Repeated Actions!