|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
04-03-2024, 07:50 AM
|
#1 |
|
Confirmed User
Industry Role:
Join Date: Apr 2007
Posts: 5,107
|
Baning AI Bots from Crawling
What's your opinion on disabling AI bots crawling sites by robots.txt? Can it affect SEO efforts in the future considering planned Google AI changes
__________________
Femdom Destiny -------------------------------------------- ICQ: 463-630-426 email: webmaster(at)femdomdestiny.com |
|
|
|
04-03-2024, 10:05 AM
|
#2 |
|
Icq: 14420613
Industry Role:
Join Date: Mar 2001
Location: chicago
Posts: 15,407
|
i dont think you can block all the bots , many of them wont bother reading robots.txt
__________________
Need WebHosting ? Email me for some great deals [email protected] |
|
|
|
04-03-2024, 12:09 PM
|
#3 |
|
Confirmed User
Industry Role:
Join Date: Dec 2016
Posts: 961
|
|
|
|
|
|
04-05-2024, 09:40 PM
|
#4 | |
|
Confirmed User
Join Date: Jan 2003
Posts: 903
|
Quote:
1500+ records in robots.txt does that make site slower? |
|
|
|
|
|
04-06-2024, 12:16 AM
|
#5 | |
|
Confirmed User
Industry Role:
Join Date: Dec 2016
Posts: 961
|
Quote:
|
|
|
|
|
|
04-06-2024, 12:28 AM
|
#6 | |
|
Horsing Around
Industry Role:
Join Date: Sep 2002
Location: AU
Posts: 5,827
|
Quote:
|
|
|
|
|
|
04-06-2024, 08:15 AM
|
#7 | |
|
Biz Dev and SEO
Industry Role:
Join Date: Jun 2005
Posts: 15,126
|
Quote:
By the way, that script is written for Apache. Basically a blacklist of rules and IPs.
__________________
--- Busy ranking websites on Google... 
|
|
|
|
|
|
04-12-2024, 02:57 PM
|
#8 | |
|
Confirmed User
Join Date: Jan 2003
Posts: 903
|
Quote:
I just quickly looked in the file and saw robots.txt. I plan on using this thanks for info. |
|
|
|
|
|
04-12-2024, 06:40 PM
|
#9 | |
|
Videochat Solutions
Industry Role:
Join Date: Aug 2004
Location: Canada
Posts: 45,846
|
Quote:
I looked at your site and I see a couple of problems with it, SEO wise. If your site is dropping in the rankings it is probably nothing to do with your robots.txt file, and more to do with the goofs in your code. You should get someone who really knows their SEO to help you out.
__________________
|
|
|
|
|
|
04-12-2024, 06:47 PM
|
#10 |
|
Mark Osterholt Sucks Cock
Industry Role:
Join Date: Dec 2002
Location: In your AirBNB
Posts: 18,155
|
I havent been banned or blocked yet with my bots and proxy network. Sometimes a captcha throws me but it gets passed off to someone with real eyes.
|
|
|
|
|
04-12-2024, 08:19 PM
|
#11 |
|
Too lazy to set a custom title
Industry Role:
Join Date: Dec 2009
Location: Google
Posts: 12,357
|
bad bots don't care about your robots.txt file. maybe use a server-wide security solution like imunify360 or bitninja. they will block bad bots from scraping your sites. Cloudflare has a bot fight mode option as well.
__________________
Fast & Reliable Website Hosting |
|
|
|
|
04-13-2024, 02:50 PM
|
#12 | |
|
Mark Osterholt Sucks Cock
Industry Role:
Join Date: Dec 2002
Location: In your AirBNB
Posts: 18,155
|
Quote:
 |
|
|
|
|
