Originally Posted by psili

I think it's used more to define parts you don't want crawled. The meta tags should take care if you just want to be crawled.