Thread: Tech Baning AI Bots from Crawling
View Single Post
Old 04-12-2024, 08:19 PM  
2intense
Too lazy to set a custom title
 
Industry Role:
Join Date: Dec 2009
Location: Google
Posts: 12,456
bad bots don't care about your robots.txt file. maybe use a server-wide security solution like imunify360 or bitninja. they will block bad bots from scraping your sites. Cloudflare has a bot fight mode option as well.
2intense is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote