Quote:
Originally Posted by redwhiteandblue
Impossible to avoid every single last one but I block 99% of them. My favourites are the Bing bots, they are obvious because they have MSIE in the user agent and nobody uses that any more.
|
Bots are quite complex thing , it's not just search engine bots, but bots masked as regular traffic, and you cant block traffic which have almost zero difference from regular traffic. Only way to detect them is by behavior, for example if they click multiple links in short time, it's a bot.