|
If you read the article it says that it's to stop text versions of articles being indexed. Just about every path listed there ends with /text
Here are the remaining exclusions
User-agent: *
Disallow: /cgi-bin
Disallow: /search
Disallow: /query.html
Disallow: /help
Disallow: /slideshow/gallery/iraq
User-agent: whsearch
Disallow: /cgi-bin
Disallow: /search
Disallow: /query.html
Disallow: /help
Disallow: /sitemap.html
Disallow: /privacy.html
Disallow: /accessibility.html
|