Robots.txt question

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • PinPine
    Confirmed User
    • Mar 2007
    • 385

    #1

    Robots.txt question

    Guys,

    Quick question about robots.txt
    If this following is my robots.txt I can assume that google wont index /feed /trackback etc right ?

    I have placed robots.txt in my root directory and for some reason he is still indexing these pages e.g. www.teensqueezer.com/feed/ can be found on google. HOW ?

    User-agent: Googlebot
    Disallow: /*/feed/$
    Disallow: /*/feed/rss/$
    Disallow: /*/trackback/$
    User-agent: *
    Disallow: /wp-
    Disallow: /feed/
    Disallow: /trackback/
    Disallow: /rss/
    Disallow: /comments/feed/
    Disallow: /page/
    Disallow: /date/
    Disallow: /comments/
    Disallow: /linkex/
    ICQ: 490974172
    Looking for hardlink trades: www.solidteens.com/linkex/
  • CIVMatt
    Amateur Pimpin
    • Aug 2004
    • 13075

    #2
    you realize you may have to wait a while before the spider reads this new txt file right?
    Make easy money with Webcams

    Comment

    • PinPine
      Confirmed User
      • Mar 2007
      • 385

      #3
      I did not know that! i thought the very first time the spider would come along, it would read the list....maybe I've got to have some more patience.

      thnx
      ICQ: 490974172
      Looking for hardlink trades: www.solidteens.com/linkex/

      Comment

      • CIVMatt
        Amateur Pimpin
        • Aug 2004
        • 13075

        #4
        Yeah give it a little while, remember anytime you're dealing with SE or spiders, it's about patience, might take a while, esp if the site is newer.
        Make easy money with Webcams

        Comment

        • Subtle
          Confirmed User
          • Jun 2007
          • 2005

          #5
          Interesting!

          ----- Blog Themes for $49 -----
          ------ INSTANT DELIVERY ------

          Comment

          Working...