robots.txt

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • OzMan84
    Confirmed User
    • Nov 2003
    • 4740
    #1

    robots.txt

    Hi,

    Not sure if ive set this up incorrect but there are still pages getting indexed
    User-Agent: *
    Disallow: /*page/

    User-agent: *
    Disallow: /*?
    domain.com/455/report.php?id=455
    domain.com/316/page/popular_likes/0-20/

    are two examples of the type pages that are getting indexed

    any tips/suggestions would be great
    XferClick - 300,000 UV daily and growing quickly. Majority US, EU, CA and AU traffic. Contact me for more information
    ICQ: 304568058
    Tags: None
    • AdultKing
      Raise Your Weapon
      • Jun 2003
      • 15601
      #2
      Globbing and regular expressions are not supported by robots.txt generally

      therefore the * has no effect.

        Comment

        • OzMan84
          Confirmed User
          • Nov 2003
          • 4740
          #3
          what would be the wild card entry then? as i have a number of dynamic pages?
          XferClick - 300,000 UV daily and growing quickly. Majority US, EU, CA and AU traffic. Contact me for more information
          ICQ: 304568058

            Comment

            • AdultKing
              Raise Your Weapon
              • Jun 2003
              • 15601
              #4
              Try reading this page http://tools.seobook.com/robots-txt/

              * can be used however you are not using it correctly in your example.

                Comment

                • OzMan84
                  Confirmed User
                  • Nov 2003
                  • 4740
                  #5
                  ok cheers.

                  what about a 301 redirect

                  http://domain.com/1106/
                  to
                  http://domain.com/1106/word-word/


                  it seems to be indexing both versions of the site
                  XferClick - 300,000 UV daily and growing quickly. Majority US, EU, CA and AU traffic. Contact me for more information
                  ICQ: 304568058

                    Comment

                    • Barry-xlovecam
                      It's 42
                      • Jun 2010
                      • 18083
                      #6
                      .htaccess

                      Now this is not quite complete yet, since we need to specifically send a 301 status code. Without the status argument "Redirect" will send a temporary redirect status (302). So we simply take the above example and change it to:

                      Redirect 301 /foo http://foobar.com/foo

                      OR

                      Redirect permanent /foo http://foobar.com/foo

                      And there is also a specific 'RedirectPermanent' directive as well so you could just as easily say:

                      RedirectPermanent /foo http://foobar.com/foo

                        Comment

                        • V_RocKs
                          Damn Right I Kiss Ass!
                          • Nov 2003
                          • 32451
                          #7
                          Just turn your server off... Nothing gets indexed that way.


                          Seriously!



                          Nothing!

                            Comment

                            Previous template Next
                            Working...