Search engines and password protected directories?

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Mutt
    Too lazy to set a custom title
    • Sep 2002
    • 34431

    #1

    Search engines and password protected directories?

    Search engine robots can't get to content password protected by htaccess right?
    I moved my sites to Vacares Hosting. I've saved money, my hair is thicker, lost some weight too! Thanks Sly!
  • rowan
    Too lazy to set a custom title
    • Mar 2002
    • 17393

    #2
    A crawler is just like any other software requesting a page, so yeah, they should get back a 401 also.

    Some sites have set up their system so that googlebot gets access to protected content, so it is indexed.

    Comment

    • ottopottomouse
      She is ugly, bad luck.
      • Jan 2010
      • 13177

      #3
      if you do let googlebot in anybody with a user agent switcher is in too.
      ↑ see post ↑
      13101

      Comment

      • Emil
        Confirmed User
        • Feb 2007
        • 5658

        #4
        I have another question, is water good if I'm thirsty?

        Will searchengines index my lawn?

        Is the searchengine-bot really a spider?
        Free 🅑🅘🅣🅒🅞🅘🅝🅢 Every Hour (Yes, really. Free ₿itCoins.)
        (Signup with ONLY your Email and Password. You can also refer people and get even more.)

        Comment

        • anexsia
          Confirmed User
          • May 2010
          • 5735

          #5
          Originally posted by Emil
          I have another question, is water good if I'm thirsty?

          Will searchengines index my lawn?

          Is the searchengine-bot really a spider?
          yes

          Comment

          • rowan
            Too lazy to set a custom title
            • Mar 2002
            • 17393

            #6
            Originally posted by ottopottomouse
            if you do let googlebot in anybody with a user agent switcher is in too.
            Of course, that's why you restrict access with both the user-agent and IP range.

            Comment

            • u-Bob
              there's no $$$ in porn
              • Jul 2005
              • 33063

              #7
              Originally posted by Emil
              I have another question, is water good if I'm thirsty?
              http://gfy.com/showthread.php?t=1047...ht=dehydration

              Comment

              Working...