Baning AI Bots from Crawling

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • femdomdestiny
    Confirmed User
    • Apr 2007
    • 5185
    #1

    Tech Baning AI Bots from Crawling

    What's your opinion on disabling AI bots crawling sites by robots.txt? Can it affect SEO efforts in the future considering planned Google AI changes
    Femdom Destiny


    --------------------------------------------
    ICQ: 463-630-426
    email: webmaster(at)femdomdestiny.com
    Tags: None
    • sandman!
      Icq: 14420613
      • Mar 2001
      • 15431
      #2
      i dont think you can block all the bots , many of them wont bother reading robots.txt
      Need WebHosting ? Email me for some great deals [email protected]

        Comment

        • TurboB
          Confirmed User
          • Dec 2016
          • 1060
          #3
          https://github.com/mitchellkrogza/ap...ad-bot-blocker
          https://github.com/mitchellkrogza/ng...ad-bot-blocker

            Comment

            • money biz
              Confirmed User
              • Jan 2003
              • 1017
              #4
              Originally posted by TurboB
              https://github.com/mitchellkrogza/ap...ad-bot-blocker
              https://github.com/mitchellkrogza/ng...ad-bot-blocker

              1500+ records in robots.txt does that make site slower?

                Comment

                • TurboB
                  Confirmed User
                  • Dec 2016
                  • 1060
                  #5
                  Originally posted by money biz
                  1500+ records in robots.txt does that make site slower?
                  Why robots.txt? This script does not touch it.

                    Comment

                    • Darkhorse
                      Horsing Around
                      • Sep 2002
                      • 5879
                      #6
                      Originally posted by TurboB
                      https://github.com/mitchellkrogza/ap...ad-bot-blocker
                      https://github.com/mitchellkrogza/ng...ad-bot-blocker
                      Exactly what I use.

                        Comment

                        • Marshal
                          Biz Dev and SEO
                          • Jun 2005
                          • 15219
                          #7
                          Originally posted by money biz
                          1500+ records in robots.txt does that make site slower?
                          Adding thousands of rules in robots.txt won't slow down a website, since robots.txt is only fetched once, or once in a while.

                          By the way, that script is written for Apache. Basically a blacklist of rules and IPs.
                          ---
                          Busy ranking websites on Google...

                            Comment

                            • money biz
                              Confirmed User
                              • Jan 2003
                              • 1017
                              #8
                              Originally posted by Marshal
                              Adding thousands of rules in robots.txt won't slow down a website, since robots.txt is only fetched once, or once in a while.

                              By the way, that script is written for Apache. Basically a blacklist of rules and IPs.


                              I just quickly looked in the file and saw robots.txt. I plan on using this thanks for info.

                                Comment

                                • 2MuchMark
                                  Mark of 2Much.net
                                  • Aug 2004
                                  • 50991
                                  #9
                                  Originally posted by femdomdestiny
                                  What's your opinion on disabling AI bots crawling sites by robots.txt? Can it affect SEO efforts in the future considering planned Google AI changes
                                  Hi Femdom,

                                  I looked at your site and I see a couple of problems with it, SEO wise. If your site is dropping in the rankings it is probably nothing to do with your robots.txt file, and more to do with the goofs in your code. You should get someone who really knows their SEO to help you out.

                                    Comment

                                    • Mr Pheer
                                      So Fucking Banned
                                      • Dec 2002
                                      • 22083
                                      #10
                                      I havent been banned or blocked yet with my bots and proxy network. Sometimes a captcha throws me but it gets passed off to someone with real eyes.

                                        Comment

                                        • 2intense
                                          Too lazy to set a custom title
                                          • Dec 2009
                                          • 12495
                                          #11
                                          bad bots don't care about your robots.txt file. maybe use a server-wide security solution like imunify360 or bitninja. they will block bad bots from scraping your sites. Cloudflare has a bot fight mode option as well.
                                          Most Affordable Firewall & Malware Protection for Linux Servers

                                            Comment

                                            • Mr Pheer
                                              So Fucking Banned
                                              • Dec 2002
                                              • 22083
                                              #12
                                              Originally posted by 2intense
                                              bad bots don't care about your robots.txt file. maybe use a server-wide security solution like imunify360 or bitninja. they will block bad bots from scraping your sites. Cloudflare has a bot fight mode option as well.
                                              The best bots just look like a regular guy browsing your sites

                                                Comment

                                                Previous template Next
                                                Working...