Robots.txt help. How do I do this?

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • wedouglas
    So Fucking Banned
    • Aug 2004
    • 5921
    #1

    Robots.txt help. How do I do this?

    I want to disallow the folder /sub/, but I want to allow everything in that folder. e.g. /sub/page1, /sub/page2

    Any quick way to allow all files/folders inside /sub/ but disallow /sub/ alone?
    Tags: None
    • After Shock Media
      It's coming look busy
      • Mar 2001
      • 35299
      #2
      yeah but cant remember, damn pain meds.

      but not all spiders follow those rules.

      Umm I think it is
      User-Agent: *
      Disallow: /dirofyourdesire/

      I take no responsibility if you loose all rankings, i am under the influence.

      [email protected] ICQ:135982156 AIM: Aftershockmed1a MSN: [email protected]

        Comment

        • spasmo
          Confirmed User
          • Dec 2005
          • 2678
          #3
          Originally posted by After Shock Media
          Umm I think it is
          User-Agent: *
          Disallow: /dirofyourdesire/
          That will disallow all the dirs under it too.

          Hope you feel better soon, man.

          Surfers: Go here for hot babes.

            Comment

            • boldy
              Macdaddy coder
              • Feb 2002
              • 2806
              #4
              Why dont you remove the index.html /.php from /sub/
              and add a .htaccess file in /sub/:
              Options -Indexes
              MacDaddy Coder.

                Comment

                • After Shock Media
                  It's coming look busy
                  • Mar 2001
                  • 35299
                  #5
                  Originally posted by spasmo
                  That will disallow all the dirs under it too.

                  Hope you feel better soon, man.

                  Thanks, i feel great actually. One of those things about pain meds.

                  Oh and thats also why i put in warning.

                  [email protected] ICQ:135982156 AIM: Aftershockmed1a MSN: [email protected]

                    Comment

                    • wedouglas
                      So Fucking Banned
                      • Aug 2004
                      • 5921
                      #6
                      Originally posted by boldy
                      Why dont you remove the index.html /.php from /sub/
                      and add a .htaccess file in /sub/:
                      Options -Indexes
                      There isn't one. This script uses a lot of mod rewrite. I can't htacces http://site.com/sub/ because it doesn't actually exist. apparently I can't redirect it either. My only option is to disallow it.

                        Comment

                        • borked
                          Totally Borked
                          • Feb 2005
                          • 6284
                          #7
                          Originally posted by wedouglas
                          I want to disallow the folder /sub/, but I want to allow everything in that folder. e.g. /sub/page1, /sub/page2

                          Any quick way to allow all files/folders inside /sub/ but disallow /sub/ alone?
                          Disallow /sub
                          Allow /sub/

                          -edit...but to play it safe, just name all the folders to allow

                          Allow /sub/folder1
                          etc
                          Last edited by borked; 09-16-2006, 02:20 PM.

                          For coding work - hit me up on andy // borkedcoder // com
                          (consider figuring out the email as test #1)


                          All models are wrong, but some are useful. George E.P. Box. p202

                            Comment

                            • borked
                              Totally Borked
                              • Feb 2005
                              • 6284
                              #8
                              http://www.google.com/support/webmas...48&ctx=sibling

                              For coding work - hit me up on andy // borkedcoder // com
                              (consider figuring out the email as test #1)


                              All models are wrong, but some are useful. George E.P. Box. p202

                                Comment

                                • wedouglas
                                  So Fucking Banned
                                  • Aug 2004
                                  • 5921
                                  #9
                                  Originally posted by borked
                                  http://www.google.com/support/webmas...48&ctx=sibling
                                  Yeah, I use there tool to test it. I ended up just allowing /sub/a-z* and disallowing /sub/

                                  Thanks

                                    Comment

                                    Previous template Next
                                    Working...