Baning AI Bots from Crawling
 

What's your opinion on disabling AI bots crawling sites by robots.txt? Can it affect SEO efforts in the future considering planned Google AI changes

i dont think you can block all the bots , many of them wont bother reading robots.txt

https://github.com/mitchellkrogza/ap...ad-bot-blocker
https://github.com/mitchellkrogza/ng...ad-bot-blocker

1500+ records in robots.txt does that make site slower?

Why robots.txt? This script does not touch it.

Exactly what I use.

Adding thousands of rules in robots.txt won't slow down a website, since robots.txt is only fetched once, or once in a while.

By the way, that script is written for Apache. Basically a blacklist of rules and IPs.

I just quickly looked in the file and saw robots.txt. I plan on using this thanks for info.

Hi Femdom,

I looked at your site and I see a couple of problems with it, SEO wise. If your site is dropping in the rankings it is probably nothing to do with your robots.txt file, and more to do with the goofs in your code. You should get someone who really knows their SEO to help you out.

I havent been banned or blocked yet with my bots and proxy network. Sometimes a captcha throws me but it gets passed off to someone with real eyes.

bad bots don't care about your robots.txt file. maybe use a server-wide security solution like imunify360 or bitninja. they will block bad bots from scraping your sites. Cloudflare has a bot fight mode option as well.

The best bots just look like a regular guy browsing your sites :)